12: High-Performance Computing (HPC) & SLURM Job Scheduling

Chapter 12: High-Performance Computing (HPC) & SLURM Job Scheduling

Genome centers, hospitals, and universities maintain High-Performance Computing (HPC) clusters. Unlike a personal computer where you click to run an application, a supercomputer operates as a multi-user shared resource coordinated by a workload manager or job scheduler. The global standard scheduler in scientific computing is SLURM (Simple Linux Utility for Resource Management).

1. Authoring a Production SLURM Batch Script

Save this script as variant_calling.slurm:

#!/usr/bin/env bash
# ==============================================================================
# SLURM Submission Script for GATK4 Variant Calling
# ==============================================================================
#SBATCH --job-name=gatk_germline       # Job name in queue
#SBATCH --output=logs/gatk_%j.out      # Standard output log (%j inserts job ID)
#SBATCH --error=logs/gatk_%j.err       # Standard error log
#SBATCH --partition=compute            # Cluster partition / queue
#SBATCH --nodes=1                      # Number of nodes requested
#SBATCH --ntasks=1                     # Number of task instances
#SBATCH --cpus-per-task=16             # Multi-threading cores
#SBATCH --mem=64G                      # Total RAM requested
#SBATCH --time=12:00:00                # Time limit (HH:MM:SS)
#SBATCH --mail-type=END,FAIL           # Email notifications
#SBATCH --mail-user=scientist@omics.co.in

set -euo pipefail

echo "Job $SLURM_JOB_ID started on node $(hostname) at $(date)"

# Load required module or activate Conda environment
source ~/.bashrc
conda activate variant_caller

# Execute GATK HaplotypeCaller across 16 threads
gatk --java-options "-Xmx60g" HaplotypeCaller \
    -R /references/GRCh38.fa \
    -I data/patient_01.sorted.bam \
    -O results/patient_01.raw.vcf.gz \
    -ERC GVCF \
    --native-pair-hmm-threads 16

echo "Job $SLURM_JOB_ID finished successfully at $(date)"

2. Submitting & Monitoring Cluster Jobs

# Submit the job to the SLURM scheduler
sbatch variant_calling.slurm
# Output: Submitted batch job 984521

# Monitor your active jobs in the queue
squeue -u $USER

# Inspect all compute nodes and partition states across the cluster
sinfo

# Cancel a running or pending job
scancel 984521

# Launch an interactive shell on a compute node for debugging (NOT the login node!)
srun --partition=interactive --cpus-per-task=4 --mem=16G --time=02:00:00 --pty bash

3. High-Throughput Cohort Parallelization with SLURM Job Arrays

If you have 200 patient exomes to analyze, submitting 200 individual scripts manually is inefficient. A SLURM Job Array submits a single script that executes 200 independent worker tasks simultaneously:

#!/usr/bin/env bash
#SBATCH --job-name=cohort_align
#SBATCH --array=1-200%20               # Run tasks 1 through 200, max 20 at a time
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=04:00:00
#SBATCH --output=logs/align_%A_%a.out  # %A = Array Job ID, %a = Task Index

# Read sample name from a text file corresponding to the current Task ID ($SLURM_ARRAY_TASK_ID)
SAMPLE_NAME=$(sed -n "${SLURM_ARRAY_TASK_ID}p" sample_list.txt)

echo "Task $SLURM_ARRAY_TASK_ID processing sample: $SAMPLE_NAME"
bwa-mem2 mem -t 8 /ref/GRCh38.fa "raw/${SAMPLE_NAME}_R1.fq.gz" "raw/${SAMPLE_NAME}_R2.fq.gz" | \
    samtools sort -o "bams/${SAMPLE_NAME}.bam" -

4. Cluster Efficiency & Resource Profiling

Over-requesting resources (e.g. requesting 128 GB RAM for a tool that only uses 4 GB) blocks other lab members from accessing compute nodes. Use sacct and seff to inspect actual usage:

# Inspect past completed job metrics (CPU time, MaxRSS peak memory, Exit Code)
sacct -j 984521 --format=JobID,JobName,Partition,Elapsed,MaxRSS,State,ExitCode

# Check CPU and Memory utilization efficiency:
seff 984521
# Example output:
# CPU Efficiency: 92.4% of 16 cores
# Memory Efficiency: 88.5% of 64.00 GB (Peak: 56.64 GB)