4: Biological File Inspection & Stream Viewing

Chapter 4: Biological File Inspection & Stream Viewing

Bioinformatics data files are massive: whole-genome sequencing (WGS) FASTQ files often exceed 50 GB, reference genome assemblies contain billions of base pairs, and BAM alignment files frequently reach hundreds of gigabytes. Attempting to open these files in traditional graphical editors (like Notepad or Word) will immediately exhaust system RAM and crash your computer. Linux offers dedicated command-line stream utilities that inspect and parse multi-gigabyte genomic files in milliseconds without loading the entire file into memory.

1. Fast File Peeking: head & tail

Examine the first or last lines of a file instantaneously:

# View the first 20 lines of a FASTQ sequencing file
head -n 20 sample_R1.fastq

# View the first 4 lines (exactly 1 FASTQ read record)
head -n 4 sample_R1.fastq

# View the last 50 lines of an alignment output
tail -n 50 alignment.sam

# Real-time pipeline monitoring: Follow a running log file as new lines are appended
tail -f gatk_variant_calling.log
tail -n 100 -f nextflow_run.log  # Show last 100 lines and keep following

2. Interactive Paging with less -S

The less utility allows non-destructive scrolling through files of arbitrary size without memory consumption:

# Open a VCF file without word wrapping (-S chops long lines horizontally)
less -S cohort_variants.vcf

# Open compressed files interactively
zless -S raw_reads.fastq.gz

Essential less Keybindings:

• Space / f : Scroll forward one entire page.
• b : Scroll backward one page.
• → / ← : Scroll horizontally left and right (vital for wide genomic tables).
• /pattern : Search forward for a text string or gene symbol (e.g., /chr17).
• n : Jump to the next search match; N jumps to previous.
• G : Jump directly to the end of the file; 1G jumps to the beginning.
• q : Quit less and return to shell prompt.

3. Counting Lines and Reads with wc -l

The wc (word count) command calculates lines, words, and byte counts:

# Count lines in a plain text file
wc -l annotation.gtf

# Calculate total sequencing reads in a FASTQ file (FASTQ format has 4 lines per read):
# Total Lines / 4 = Total Reads
wc -l sample_R1.fastq

# Stream count on compressed files without decompressing to disk
zcat sample_R1.fastq.gz | wc -l

4. Streaming Compressed Files with zcat & tabix

Modern genomics formats use gzip or BGZF block-compression:

# View first 8 lines (2 reads) of a gzip-compressed FASTQ file
zcat sample_R1.fastq.gz | head -n 8
# On macOS BSD systems use: gzcat or gzip -dc

# Indexed random access with Tabix (requires block-gzipped .gz and .tbi index)
# Query chromosome 7 between position 55,000,000 and 55,200,000 (EGFR locus)
tabix clinvar.vcf.gz chr7:55000000-55200000

# View header metadata using tabix -H
tabix -H clinvar.vcf.gz | tail -n 10