Chapter 9: HPC Cluster Integration: SLURM, SGE & LSF Executors

Chapter 9: HPC Cluster Integration: SLURM, SGE & LSF Executors

High-Performance Computing (HPC) clusters manage distributed compute nodes using batch schedulers like SLURM, PBS Pro, Sun Grid Engine (SGE), or IBM LSF. Nextflow's Executor abstraction automatically translates each pipeline process into a native scheduler job script (e.g. sbatch) without modifying pipeline code.

1. Configuring SLURM in nextflow.config

profiles {
    slurm {
        process {
            executor = 'slurm'
            queue    = 'bioinformatics'
            clusterOptions = '--account=lab_project --qos=standard'

            // Default resources for low-demand jobs
            cpus   = 2
            memory = '4 GB'
            time   = '1h'

            // Label-based resource overrides
            withLabel: 'process_high' {
                cpus   = 32
                memory = '128 GB'
                time   = '24h'
                queue  = 'high_mem'
            }
        }

        executor {
            queueSize = 50       // Maximum concurrent jobs submitted to SLURM queue
            submitRateLimit = '10 sec' // Avoid overwhelming the Slurm controller
        }
    }
}

2. Utilizing High-Speed Node-Local Scratch

Network filesystems (NFS, Lustre) suffer from severe I/O bottlenecks when hundreds of nodes read/write simultaneously. Use the scratch directive to instruct Nextflow to execute tasks on local node SSDs (e.g. /tmp or $TMPDIR), automatically transferring final outputs back upon completion.

process HEAVY_IO {
    scratch '/scratch/local/$USER'
    // ...
}