Module 2: Key Long-Read Data Formats: FAST5, POD5, FASTQ & SAM/BAM Specifications
Tested Stacks:
GATK 4.6.1
Samtools 1.21
Nextflow 24.10
Seurat v5.1
Python 3.12
Prerequisites: Basic Linux CLI & Sequencing Formats
📚 Chapter 2 of 12
0% Course Completed
1. Evolution of Raw Signal Containers: FAST5 to POD5
Unlike Illumina sequencers which output raw pixel intensities that are discarded after real-time basecalling, Oxford Nanopore sequencers preserve the raw physical measurements so users can re-basecall data as newer neural network architectures emerge.
| Attribute | Legacy FAST5 Format | Modern POD5 Format |
|---|---|---|
| Underlying Technology | Hierarchical Data Format 5 (HDF5). Highly flexible but suffers from filesystem locking and poor multithreaded I/O. | Apache Arrow columnar IPC format with custom dictionary encoding and zstandard compression. |
| File System Footprint | Historically written as single-read files (causing millions of inodes) or multiread batches ($4000$ reads/file). Slow random access. | Monolithic, thread-safe batches ($100,000+$ reads per file). Highly compressed, eliminating inode exhaustion. |
| Basecalling Throughput | Basecallers spend up to $30-40\%$ of GPU run time waiting for HDF5 decompression and file read operations. | Zero-copy memory mapping allows NVIDIA GPUs to saturate tensor cores, increasing basecalling speeds by $3-5\times$. |
| Tooling & Python APIs | h5py, ont_fast5_api | pod5 Python library, pod5-file-validator |
2. The Anatomy of a POD5 File
Each read inside a POD5 file contains discrete hardware and signal metadata:
# Inspecting raw signal traces directly from POD5 using Python
import pod5
with pod5.Reader("sample_reads.pod5") as reader:
for read in reader.reads():
print(f"Read ID: {read.read_id}")
print(f"Channel: {read.channel} | Well: {read.well}")
print(f"Sample Rate: {read.sample_rate} Hz")
print(f"Signal Length: {len(read.signal)} raw pA measurements")
# Raw picoampere calculation from 16-bit integer ADC values:
# pA = (raw_adc + offset) * range / digitisation
signal_pA = read.signal_pa
print(f"Mean current: {signal_pA.mean():.2f} pA, StdDev: {signal_pA.std():.2f} pA")
break3. SAM/BAM Tags Unique to Long Reads
When long reads are aligned or basecalled with modifications, specific SAM tags convey epigenetic and structural information:
MM:Z:(Base Modifications): Encodes the positions of modified bases (e.g.C+m?for 5mC) relative to canonical bases.ML:B:C(Modification Probabilities): Array of unsigned 8-bit integers ($0 - 255$) expressing the Phred-scaled probability that each indicated base is modified ($P = \text{val} / 256$).SA:Z:(Supplementary Alignment): Crucial for structural variant split-read analysis. Lists chimeric alignment segments:rname,pos,strand,CIGAR,mapQ,NM;.HP:i:andPS:i:(Haplotype & Phase Set): Generated by phasing tools like WhatsHap or LongPhase to assign individual reads to maternal ($1$) or paternal ($2$) chromosomes within a phase set block.
⚠️
Common Pitfalls & Troubleshooting Checklist
- Java Heap Exhaustion (GATK / Picard): If processes terminate with
java.lang.OutOfMemoryError, set your environment variable explicitly:export _JAVA_OPTIONS="-Xmx16g -Djava.io.tmpdir=/tmp". - Index Timestamp Mismatch: If tools complain that a BAM index is missing or out of date, regenerate with
samtools index -@ 4 sample.bam. - Conda Channel Conflicts: Avoid ABI incompatibilities between bioconda and conda-forge by setting strict priority:
conda config --set channel_priority strict. - Chromosome Naming Contamination: Always verify whether reference FASTA headers use
chr1or1; divergent naming causes silent alignment drops.
Was this guide helpful to your research?
Help us maintain 100% accurate, peer-reviewed bioinformatics standards.