omics.co.in logo omics.co.in

File Input and Output

Tested Stacks: GATK 4.6.1 Samtools 1.21 Nextflow 24.10 Seurat v5.1 Python 3.12
Prerequisites: Basic Linux CLI & Sequencing Formats
📚 Chapter 5 of 13 0% Course Completed
🐍 Downloadable Lesson Python Script 3.6 KB

ch05_fasta_csv_io.py

Memory-efficient buffered generator for multi-line FASTA records and CSV/TSV expression matrix parser calculating per-gene mean and variance.

$ python ch05_fasta_csv_io.py --fasta sample.fasta

File Input and Output in Bioinformatics

Biological data formats such as FASTA, FASTQ, SAM/BAM, BED, GFF3, and VCF are primarily text or compressed text files. Effective reading, streaming, and writing of files prevents memory overflows during large-scale analysis.

# Reading large TSV gene expression files line by line
with open("expression_matrix.tsv", "r") as infile, open("significant_genes.tsv", "w") as outfile:
    header = infile.readline()
    outfile.write(header)
    
    for line in infile:
        parts = line.strip().split("\t")
        gene_id = parts[0]
        log2_fc = float(parts[1])
        padj = float(parts[2])
        
        # Filter for significantly upregulated genes
        if log2_fc >= 2.0 and padj <= 0.01:
            outfile.write(line)
⚠️

Common Pitfalls & Troubleshooting Checklist

  • Java Heap Exhaustion (GATK / Picard): If processes terminate with java.lang.OutOfMemoryError, set your environment variable explicitly: export _JAVA_OPTIONS="-Xmx16g -Djava.io.tmpdir=/tmp".
  • Index Timestamp Mismatch: If tools complain that a BAM index is missing or out of date, regenerate with samtools index -@ 4 sample.bam.
  • Conda Channel Conflicts: Avoid ABI incompatibilities between bioconda and conda-forge by setting strict priority: conda config --set channel_priority strict.
  • Chromosome Naming Contamination: Always verify whether reference FASTA headers use chr1 or 1; divergent naming causes silent alignment drops.
Was this guide helpful to your research?
Help us maintain 100% accurate, peer-reviewed bioinformatics standards.
📝 Propose Edit on GitHub ↗