omics.co.in logo omics.co.in

Module 2: Key Long-Read Data Formats: FAST5, POD5, FASTQ & SAM/BAM Specifications

Tested Stacks: GATK 4.6.1 Samtools 1.21 Nextflow 24.10 Seurat v5.1 Python 3.12
Prerequisites: Basic Linux CLI & Sequencing Formats
📚 Chapter 2 of 12 0% Course Completed

1. Evolution of Raw Signal Containers: FAST5 to POD5

Unlike Illumina sequencers which output raw pixel intensities that are discarded after real-time basecalling, Oxford Nanopore sequencers preserve the raw physical measurements so users can re-basecall data as newer neural network architectures emerge.

AttributeLegacy FAST5 FormatModern POD5 Format
Underlying TechnologyHierarchical Data Format 5 (HDF5). Highly flexible but suffers from filesystem locking and poor multithreaded I/O.Apache Arrow columnar IPC format with custom dictionary encoding and zstandard compression.
File System FootprintHistorically written as single-read files (causing millions of inodes) or multiread batches ($4000$ reads/file). Slow random access.Monolithic, thread-safe batches ($100,000+$ reads per file). Highly compressed, eliminating inode exhaustion.
Basecalling ThroughputBasecallers spend up to $30-40\%$ of GPU run time waiting for HDF5 decompression and file read operations.Zero-copy memory mapping allows NVIDIA GPUs to saturate tensor cores, increasing basecalling speeds by $3-5\times$.
Tooling & Python APIsh5py, ont_fast5_apipod5 Python library, pod5-file-validator

2. The Anatomy of a POD5 File

Each read inside a POD5 file contains discrete hardware and signal metadata:

# Inspecting raw signal traces directly from POD5 using Python
import pod5

with pod5.Reader("sample_reads.pod5") as reader:
    for read in reader.reads():
        print(f"Read ID: {read.read_id}")
        print(f"Channel: {read.channel} | Well: {read.well}")
        print(f"Sample Rate: {read.sample_rate} Hz")
        print(f"Signal Length: {len(read.signal)} raw pA measurements")
        
        # Raw picoampere calculation from 16-bit integer ADC values:
        # pA = (raw_adc + offset) * range / digitisation
        signal_pA = read.signal_pa
        print(f"Mean current: {signal_pA.mean():.2f} pA, StdDev: {signal_pA.std():.2f} pA")
        break

3. SAM/BAM Tags Unique to Long Reads

When long reads are aligned or basecalled with modifications, specific SAM tags convey epigenetic and structural information:

  • MM:Z: (Base Modifications): Encodes the positions of modified bases (e.g. C+m? for 5mC) relative to canonical bases.
  • ML:B:C (Modification Probabilities): Array of unsigned 8-bit integers ($0 - 255$) expressing the Phred-scaled probability that each indicated base is modified ($P = \text{val} / 256$).
  • SA:Z: (Supplementary Alignment): Crucial for structural variant split-read analysis. Lists chimeric alignment segments: rname,pos,strand,CIGAR,mapQ,NM;.
  • HP:i: and PS:i: (Haplotype & Phase Set): Generated by phasing tools like WhatsHap or LongPhase to assign individual reads to maternal ($1$) or paternal ($2$) chromosomes within a phase set block.
⚠️

Common Pitfalls & Troubleshooting Checklist

  • Java Heap Exhaustion (GATK / Picard): If processes terminate with java.lang.OutOfMemoryError, set your environment variable explicitly: export _JAVA_OPTIONS="-Xmx16g -Djava.io.tmpdir=/tmp".
  • Index Timestamp Mismatch: If tools complain that a BAM index is missing or out of date, regenerate with samtools index -@ 4 sample.bam.
  • Conda Channel Conflicts: Avoid ABI incompatibilities between bioconda and conda-forge by setting strict priority: conda config --set channel_priority strict.
  • Chromosome Naming Contamination: Always verify whether reference FASTA headers use chr1 or 1; divergent naming causes silent alignment drops.
Was this guide helpful to your research?
Help us maintain 100% accurate, peer-reviewed bioinformatics standards.
📝 Propose Edit on GitHub ↗