Chapter 2: End-to-End Pipeline Architecture & Workflow Flowchart
Tested Stacks:
GATK 4.6.1
Samtools 1.21
Nextflow 24.10
Seurat v5.1
Python 3.12
Prerequisites: Basic Linux CLI & Sequencing Formats
📚 Chapter 2 of 10
0% Course Completed
Chapter 2: End-to-End Pipeline Architecture & Workflow Flowchart
An industrial-grade clinical NGS pipeline must execute deterministically across every biological sample. From the moment the physical sequencer completes fluorescence or electrical imaging through the final clinical PDF report, every tool must pass standardized data formats and maintain rigorous checksum audits.
The Unbroken Execution Architecture
Below is the complete, end-to-end NGS pipeline architecture detailed in our masterclass specification. It bridges raw bioinformatics processing (FASTQ to VCF) directly into clinical variant curation, database cross-referencing, and ACMG automated interpretation.
Figure 2.1: Clean End-to-End NGS Pipeline Architecture & Clinical Interpretation Bridge
Phase-by-Phase Technical Transition Matrix
| Step | Input Format | Benchmark Tools | Output Format | Clinical Objective |
|---|---|---|---|---|
| 1. QC & Preprocessing | Raw FASTQ (.fastq.gz) | FastQC, FastP, Trimmomatic | Cleaned FASTQ | Remove synthetic adapters, eliminate Phred < 20 bases, prevent false-positive variant calls. |
| 2. Read Alignment | Cleaned FASTQ + GRCh38 FASTA | BWA-MEM, STAR, Bowtie2 | SAM / BAM | Determine exact genomic coordinates for billions of sequencing reads. |
| 3. Post-Alignment | Unsorted BAM | Samtools, Picard, GATK BQSR | Sorted, Recalibrated BAM/CRAM | Sort coordinates, mark optical/PCR duplicates, adjust Phred scores for systematic errors. |
| 4. Variant Calling | Recalibrated BAM/CRAM | HaplotypeCaller, DeepVariant, MuTect2, FACETS | Raw VCF (.vcf.gz) | Detect non-reference mutations: constitutional germline (0/0, 0/1, 1/1) or subclonal somatic (>1% VAF). |
| 5. Annotation Bridge | Raw VCF | VEP, ANNOVAR, SnpEff, CADD, AlphaMissense | Annotated VCF / TSV | Map coordinates to MANE Select transcripts, compute protein impact and in silico damage scores. |
| 6. Repositories Query | Annotated VCF | gnomAD, ClinVar, COSMIC, OMIM | Clinical Evidence Matrix | Determine population control rarity, past lab clinical classifications, and disease inheritance. |
| 7. ACMG Interpretation | Clinical Evidence Matrix | ACMG/AMP Master Rules Engine | Signed Clinical Report | Apply formal criteria combination rules to yield 5-tier classification and actionable therapeutics. |
⚠️
Common Pitfalls & Troubleshooting Checklist
- Java Heap Exhaustion (GATK / Picard): If processes terminate with
java.lang.OutOfMemoryError, set your environment variable explicitly:export _JAVA_OPTIONS="-Xmx16g -Djava.io.tmpdir=/tmp". - Index Timestamp Mismatch: If tools complain that a BAM index is missing or out of date, regenerate with
samtools index -@ 4 sample.bam. - Conda Channel Conflicts: Avoid ABI incompatibilities between bioconda and conda-forge by setting strict priority:
conda config --set channel_priority strict. - Chromosome Naming Contamination: Always verify whether reference FASTA headers use
chr1or1; divergent naming causes silent alignment drops.
Was this guide helpful to your research?
Help us maintain 100% accurate, peer-reviewed bioinformatics standards.