Chapter 8: Robust Error Handling, Retries & Dynamic Resource Allocation

Chapter 8: Robust Error Handling, Retries & Dynamic Resource Allocation

On large HPC clusters and cloud instances, compute jobs inevitably fail due to transient out-of-memory (OOM) kills, node preemptions, or temporary network timeouts. Robust pipelines automatically catch errors, scale resource allocations dynamically, and retry without terminating the entire workflow.

1. Implementing Dynamic Retries in Nextflow

process BWA_MEM {
    tag "BWA on ${sample_id}"

    // Dynamic error handling
    errorStrategy { task.exitStatus in [137, 140, 143] ? 'retry' : 'finish' }
    maxRetries 3

    // Dynamic resource allocation: doubles memory with each attempt
    cpus   { 4 * task.attempt }
    memory { 8.GB * task.attempt }
    time   { 2.h  * task.attempt }

    input:
    tuple val(sample_id), path(reads)
    path index

    output:
    tuple val(sample_id), path("${sample_id}.bam")

    script:
    """
    bwa mem -t ${task.cpus} ref ${reads} | samtools sort -o ${sample_id}.bam
    """
}

2. Interpreting Common Linux & Cluster Exit Codes

# Exit Code Reference:
# -----------------------------------------------------------------------------
# Exit Code   Signal      Root Cause
# -----------------------------------------------------------------------------
# 1           General     Application syntax or runtime error
# 127         Command     Executable binary not found in PATH or container
# 137         SIGKILL     Task exceeded allocated memory limit (OOM Killer)
# 140         SIGXCPU     Task exceeded maximum wall-time limit
# 143         SIGTERM     Job terminated by cluster scheduler (preemption/eviction)
# -----------------------------------------------------------------------------

3. The Secrets of Nextflow Resume Caching

Nextflow computes a 128-bit SHA-256 hash for every task based on: process source code, script text, input file paths, file sizes, timestamps, and container image tags. If nothing changed, Nextflow outputs [cached] and completes the step instantaneously.