8: I/O Redirection, Unix Pipes & Stream Chaining

Chapter 8: I/O Redirection, Unix Pipes & Stream Chaining

In high-throughput genomics, disk I/O is the primary computational bottleneck. Writing an intermediate 60 GB uncompressed SAM file from an aligner to a physical hard drive, only to immediately read it with SAMtools to sort it, wastes minutes of compute time and fills disk quotas. Unix Input/Output (I/O) redirection and pipes (|) stream data directly through computer memory (RAM) between tools without creating intermediate files on disk.

1. The Redirection Operators

# Overwrite stdout to a file (>):
grep -v "^#" input.vcf > clean_variants.vcf

# Append stdout to an existing file without overwriting (>>):
echo "Sample_01 completed successfully" >> pipeline.log

# Read stdin from a file (<):
wc -l < sequences.fasta

# Redirect stderr (error messages) to a log file (2>):
samtools sort input.bam -o sorted.bam 2> sort_errors.log

# Combine stdout and stderr into the same unified log file (&> or 2>&1):
bwa mem ref.fa read1.fq read2.fq > align.sam 2> align.log
# Or both combined:
run_pipeline.sh &> full_run.log

2. Zero-Disk Processing with Unix Pipes (|)

The pipe operator (|) connects the stdout of the left-hand command directly into the stdin of the right-hand command in active memory:

# Decompress, filter, and count in a single zero-disk memory stream:
zcat reads.fastq.gz | grep "^@" | wc -l

# Extract genes, sort alphabetically, and count occurrences:
cut -f 1 significant_genes.tsv | sort | uniq -c | sort -nr

3. Simultaneous Monitoring and Logging with tee

The tee command duplicates an output stream, sending one copy to a physical file and displaying the second copy live on your terminal screen:

# Run quality control while simultaneously viewing progress and saving the log:
fastp -i in1.fq -I in2.fq -o out1.fq -O out2.fq 2>&1 | tee fastp_run.log

4. End-to-End Production Alignment Stream

A production example chaining trimming, reference alignment, and coordinate sorting into a single continuous stream:

# Stream reads directly through BWA-MEM2 into SAMtools coordinate-sorted BAM:
bwa-mem2 mem -t 16 -R '@RG\tID:S1\tSM:Sample1\tPL:ILLUMINA' \
    /references/GRCh38.fa \
    sample1_R1.fastq.gz sample1_R2.fastq.gz | \
samtools sort -@ 4 -m 2G -o sample1.sorted.bam -

# Notice the trailing hyphen (-): Tells samtools sort to read from stdin!
# Index the resulting BAM file:
samtools index sample1.sorted.bam