6: Stream Editing & Bulk Text Transformation with sed

Chapter 6: Stream Editing & Bulk Text Transformation with sed

The Stream Editor (sed) is a non-interactive, stream-oriented text transformation tool. Unlike interactive editors where you open a file, modify text manually, and save, sed reads input line by line, applies programmatic editing rules to its internal pattern space, and streams modified lines directly to standard output. In genomics, sed is standard for bulk chromosome name remapping, header cleaning, and file reformatting.

1. The Core sed Substitution Syntax

# Basic syntax: sed 's/find_pattern/replacement_text/flags' filename
# 's' denotes substitute; 'g' denotes global (replace all occurrences per line)
sed 's/old_text/new_text/g' sample.txt

# Using alternative delimiters when dealing with file paths (avoids leaning toothpicks):
sed 's#/scratch/old_dir#/data/new_dir#g' config.txt

2. Re-annotating Chromosomes: Ensembl to UCSC

# Prepend 'chr' to the beginning of every line in a VCF or BED file:
sed 's/^/chr/' ensembl_coordinates.bed > ucsc_coordinates.bed

# Standardize mitochondrial chromosome notation from chrMT to chrM:
sed 's/chrMT/chrM/g' input.vcf > standardized.vcf

# Strip 'chr' prefix to convert UCSC coordinates to Ensembl format:
sed 's/^chr//' ucsc_regions.bed > ensembl_regions.bed

3. Safe In-Place Editing with Backup: sed -i.bak

By default, sed streams output to the terminal without touching the source file. The -i flag enables direct in-place editing on disk. Always specify a backup extension to prevent accidental data loss:

# Modifies pipeline.sh directly on disk, preserving pipeline.sh.bak original copy
sed -i.bak 's/THREADS=8/THREADS=16/g' pipeline.sh

# If verified correct, remove the backup:
rm pipeline.sh.bak

4. Deleting Lines by Index or Pattern Match

# Delete the 1st line (e.g. remove table header)
sed '1d' expression_counts.tsv > counts_no_header.tsv

# Delete lines 1 through 10
sed '1,10d' input.txt

# Delete lines matching a specific pattern (e.g. remove SAM header lines starting with @)
sed '/^@/d' alignment.sam > alignments_only.sam

# Delete trailing whitespace from all lines
sed 's/[[:space:]]*$//' dirty_metadata.csv > clean_metadata.csv

5. Linearizing Multi-Line FASTA Records

FASTA sequence files frequently wrap DNA sequences across multiple 60- or 80-character lines. Linearizing sequences onto a single continuous line simplifies length calculations and sequence searches:

# Linearize multi-line FASTA using sed:
sed -e 's/\(^>.*$\)/#
#/' genome.fa | tr -d "
" | tr -d "
" | tr "#" "
" | sed -e '/^$/d' > linearized_genome.fa