Chapter 1
Foundations
1: Introduction to Linux, Terminals & Shell Architecture
- • Why Linux dominates bioinformatics and supercomputing clusters
- • Anatomy of Linux: Kernel, shell, system calls & user space
- • Connecting to remote HPC clusters via OpenSSH and key-based auth
- • Command syntax: Executables, short/long flags and exit codes ($?)
- • Core environment commands: whoami, hostname, uname -a & which
Chapter 2
Filesystem
2: Filesystem Hierarchy & Directory Operations
- • The Linux Filesystem Hierarchy: /, /home, /tmp, /opt & HPC /scratch
- • Absolute vs relative paths, parent (..), current (.), and home (~)
- • Navigating directories: pwd, cd, ls -lah & tree visualization
- • Creating and managing project trees: mkdir -p, cp -r, mv & rm
- • Shell globbing: Wildcards (*, ?) and brace expansion for paired reads
Chapter 3
Security
3: File Permissions, Ownership & Security
- • The Unix security model: User (u), Group (g), and Others (o)
- • Permission anatomy: Read (r=4), Write (w=2), Execute (x=1)
- • Modifying permissions with chmod: Symbolic vs Octal notation (755, 644)
- • Managing ownership and group collaboration with chown & chgrp
- • Securing patient genomes: umask 027 and multi-tenant cluster safety
Chapter 4
Data Streaming
4: Biological File Inspection & Stream Viewing
- • Non-destructive file viewing: Peeking multi-gigabyte records with head & tail
- • Monitoring pipeline executions in real time with tail -f log.txt
- • Interactive paging without line wrapping: less -S, search & navigation
- • High-speed line and read counting with wc -l (FASTQ lines / 4)
- • Compressed stream viewing with zcat, zless, bgzip and tabix indexing
Chapter 5
Regex & Search
5: Pattern Matching & Regular Expressions with grep
- • Searching plain text and sequence motifs with grep flags (-i, -n, -c)
- • Regular expression anchors (^, $) and biological character classes ([ATCG])
- • Extracting sequence headers from FASTA files (grep '^>' genome.fa)
- • Filtering VCF variant calls: Inverting matches to strip metadata (grep -v)
- • Contextual extraction: Capturing 4-line FASTQ records with -B 1 -A 2
Chapter 6
Stream Editing
6: Stream Editing & Bulk Text Transformation with sed
- • The sed execution model: Pattern space, cycles, and address matching
- • Global substitution syntax: sed 's/pattern/replacement/g'
- • Biological transformations: Remapping Ensembl (1) to UCSC (chr1) notation
- • Safe in-place editing with automatic backups (sed -i.bak)
- • Deleting lines by line number or regex match & linearizing FASTA files
Chapter 7
Data Wrangling
7: Columnar Data Wrangling & Genomic Arithmetic with awk
- • The awk processing model: BEGIN blocks, per-record filters, and END summaries
- • Delimiters & field splitting: Parsing VCF, BED, and GTF files with awk -F'\t'
- • Genomic coordinate arithmetic: Feature length ($3 - $2) and coverage depth
- • Filtering variant records: Quality thresholds ($6 >= 30) and PASS filters
- • Associative arrays: Aggregating gene read counts & formatted printf reports
Chapter 8
Pipes & Streams
8: I/O Redirection, Unix Pipes & Stream Chaining
- • Standard streams: stdin (0), stdout (1), and stderr (2) explained
- • Stream redirection: >, >>, <, and capturing error logs with 2>&1
- • RAM-based zero-disk processing using continuous Unix pipes (|)
- • Simultaneous file logging and screen display with tee
- • End-to-end stream: zcat -> fastp -> bwa-mem2 -> samtools sort
Chapter 9
Concurrency
9: Process Management, Multi-Core Processing & GNU Parallel
- • Process monitoring: Tracking CPU & RAM utilization with top, htop, and ps
- • Managing execution states: Foreground, background (&), Ctrl+Z, and bg/fg
- • Persistent terminal multiplexing: Keeping pipelines alive with tmux
- • Terminating unresponsive processes safely: kill, killall & signal types
- • Multi-core batch acceleration using xargs -P and GNU Parallel
Chapter 10
Package Management
10: Environment Isolation with Conda, Mamba & Bioconda
- • Why rootless package management is mandatory in multi-user bioinformatics
- • The Conda ecosystem vs the high-speed C++ solver Mamba / Micromamba
- • Configuring channels: conda-forge & bioconda strict channel priorities
- • Creating isolated pipeline environments: mamba create -n rnaseq
- • Exporting and sharing reproducible recipes with environment.yml
Chapter 11
Automation
11: Production Bash Scripting & Defensive Programming
- • Script structure: The shebang (#!/usr/bin/env bash) and execution bits
- • Defensive programming: set -euo pipefail prevents silent data corruption
- • Variables, arguments ($1, $@), and dynamic command substitution
- • Input validation, sample loops, and automated directory creation
- • Error trapping (trap 'cleanup' EXIT) and timestamped logging routines
Chapter 12
HPC & Slurm
12: High-Performance Computing (HPC) & SLURM Job Scheduling
- • HPC cluster topology: Head login nodes vs dedicated worker compute nodes
- • Writing submission headers: #SBATCH cpus-per-task, memory, and time
- • Job lifecycle management: sbatch, squeue, sinfo, and scancel
- • Processing 500 patient cohorts in parallel using SLURM Job Arrays
- • Profiling CPU and memory efficiency with sacct and seff
Chapter 13
Reproducibility
13: Reproducible Containers (Singularity) & Scientific Workflows
- • Computational reproducibility crisis in genomics and software drift
- • Docker vs Singularity / Apptainer: Why HPC clusters forbid Docker
- • Pulling pre-compiled BioContainers directly from Docker Hub / Quay.io
- • Filesystem mounting and container execution with --bind /scratch
- • Orchestrating scalable workflows: Nextflow & Snakemake introductory pipelines
Peer-Reviewed Foundations
Seminal peer-reviewed benchmarks and methodology literature
📚 Foundational Literature & Methodology Benchmarks
Unix Computing in Biology
- Buffalo V. (2015). Bioinformatics Data Skills: Reproducible and Robust Research with Open Source Tools. O'Reilly Media. doi:10.5555/2815647
- Tange O. (2018). GNU Parallel 2018. Zenodo. doi:10.5281/zenodo.1146014
- Quinlan AR, Hall IM. (2010). BEDTools: a flexible suite of utilities for comparing genomic features. Bioinformatics 26(6):841–842. doi:10.1093/bioinformatics/btq033
HPC Clusters & Job Schedulers
- Yoo AB, Jette MA, Grondona M. (2003). SLURM: Simple Linux Utility for Resource Management. Job Schedul Strateg Parallel Process :44–60. doi:10.1007/10968980_3
- Gentleman R, Carey VJ. (2012). High-performance computing for genomic data analysis. Nat Rev Genet 13:670–674. doi:10.1038/nrg3329
- Langmead B, Nellore A. (2018). Cloud computing for genomic data analysis. Nat Rev Genet 19:65–70. doi:10.1038/nrg.2017.85
Environments & Scientific Containers
- Grüning B, Dale R, Sjödin A, et al. (2018). Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods 15(7):475–476. doi:10.1038/s41592-018-0046-7
- Kurtzer GM, Sochat V, Bauer MW. (2017). Singularity: scientific containers for mobility of compute. PLOS ONE 12(5):e0177459. doi:10.1371/journal.pone.0177459
- Baker M. (2016). 1,500 scientists lift the lid on reproducibility. Nature 533(7604):452–454. doi:10.1038/533452a