Module 3: Navigating Public Long-Read Repositories: NCBI SRA & GIAB Consortium

1. Benchmark Standards: The Genome in a Bottle (GIAB) Consortium

Rigorous validation of long-read alignment, variant calling, and de novo assembly requires reference standard cell lines sequenced to exhaustive depths. The National Institute of Standards and Technology (NIST) GIAB consortium provides deeply characterized reference benchmark samples:

• HG001 (NA12878 / CEPH): The foundational pilot genome with hundreds of orthogonal datasets.
• HG002 (NA24385 / Ashkenazi Son): The global benchmark standard for small variant and structural variant evaluations, with high-confidence truth sets across chromosomes 1–22 and X.
• HG005 (NA24631 / Chinese Son): Key Asian trio reference for broad population validation.

2. Direct Cloud Retrieval from AWS Open Data

Downloading terabytes of long-read FASTQ and POD5 files through the NCBI web portal is slow and prone to interruption. Both the GIAB consortium and the Human Pangenome Reference Consortium (HPRC) host public AWS S3 buckets allowing direct, multithreaded downloads using the AWS CLI without an AWS account:

# List available Oxford Nanopore R10.4.2 PromethION datasets for HG002
aws s3 ls --no-sign-request s3://human-pangenomics/submissions/ | grep -i hg002

# Download high-depth PacBio HiFi BAM files for benchmark validation
aws s3 cp --no-sign-request \
    s3://human-pangenomics/submissions/3A2CCCB4-E14F-4C82-899A-40D0E700E261--PACBIO_HIFI/HG002/ \
    ./local_hg002_hifi/ \
    --recursive \
    --exclude "*" \
    --include "*.bam"

3. Programmatic SRA Ingestion for Long Reads

When downloading from NCBI SRA, long-read sequencing runs require specific parameters to ensure single-molecule reads are preserved without artificial splitting or base quality degradation:

# Prefetch the SRA archive for an ONT run (e.g. SRR24804364)
prefetch SRR24804364 --max-size 100G

# Convert SRA to unclipped long-read FASTQ with original quality scores
fasterq-dump SRR24804364 \
    --threads 8 \
    --split-files \
    --include-technical \
    --outdir ./raw_fastq/

# Compress immediately to save disk space
pigz -p 8 ./raw_fastq/SRR24804364.fastq