Chapter 14: Unsupervised Machine Learning: Hierarchical Clustering & Consensus Subtyping

1. Discovering Novel Disease Subtypes Without Ground-Truth Labels

Biomedical phenotypes (such as breast cancer, sepsis, or autoimmune diseases) are frequently heterogeneous clinical entities composed of distinct molecular sub-endotypes. Unsupervised machine learning models cluster patients solely on their transcriptomic profiles to identify clinically distinct disease classes with divergent survival outcomes and therapeutic vulnerabilities.

2. Consensus Clustering Mechanics (ConsensusClusterPlus)

Standard \(K\)-means or hierarchical clustering can produce unstable clusters that shift drastically with minor changes in sample composition. Consensus Clustering introduces algorithmic rigor through perturbation resampling: it repeatedly sub-samples 80% of items and features over 1,000 iterations, constructing a Consensus Matrix of co-clustering frequency. The optimal cluster number \(k\) is determined by evaluating:

1. The consensus matrix heatmap sharpness.
2. The empirical Cumulative Distribution Function (CDF) progression.
3. The delta area change under the CDF curve.

3. Consensus Subtyping & Survival Association Script

if (!requireNamespace("ConsensusClusterPlus", quietly = TRUE)) {
    BiocManager::install("ConsensusClusterPlus")
}
library(ConsensusClusterPlus)
library(survival)
library(survminer)

# Input: Median-centered VST expression matrix
vst_mat <- assay(vsd)[select[1:200], ]
vst_centered <- sweep(vst_mat, 1, apply(vst_mat, 1, median))

# Execute Consensus Clustering across k = 2 to k = 5
results_ccp <- ConsensusClusterPlus(
    vst_centered,
    maxK = 5,
    reps = 100,
    pItem = 0.8,
    pFeature = 1.0,
    title = tempfile(),
    clusterAlg = "hc",
    distance = "pearson",
    seed = 12621,
    plot = NULL
)

# Extract cluster assignments for optimal k = 3
cluster_assignments <- results_ccp[[3]]$consensusClass
clinical_clean$Molecular_Subtype <- factor(paste0("Subtype_", cluster_assignments))

# Kaplan-Meier Survival Analysis across Identified Subtypes
# Simulated survival data for demonstration
clinical_clean$OS_Months <- rpois(nrow(clinical_clean), lambda = 36)
clinical_clean$Status <- sample(c(0, 1), nrow(clinical_clean), replace = TRUE, prob = c(0.4, 0.6))

surv_fit <- survfit(Surv(OS_Months, Status) ~ Molecular_Subtype, data = clinical_clean)
ggsurvplot(
    surv_fit,
    data = clinical_clean,
    pval = TRUE,
    risk.table = TRUE,
    palette = c("#06b6d4", "#f43f5e", "#10b981"),
    title = "Kaplan-Meier Survival Stratification by Transcriptomic Consensus Subtypes"
)