Terminal
Terminal/Skill
Advises on biomarker discovery and validation for researchers: intent, feature selection, cross validation, leakage, and external replication before clinical use.
By Amazon Web Services
Works with
npx skills add awslabs/hcls-agent-skills --skill biomarker-discoveryThis skill teaches the agent how to think about biomarker discovery and validation, not how to run a specific pipeline. A biomarker is only useful if three things line up: (1) the intent is clearly defined (prognostic, predictive, or diagnostic), (2) the discovery statistics are honest (no leakage, appropriate penalization, nested resampling), and (3) the validation plan reaches external replication and demonstrates a clinical decision change.
Use this skill to interrogate proposals, spot methodological weaknesses, and guide the user toward the right design before code is written.
Invoke this skill when the user:
The agent should respond by:
The first question is always: what decision is this biomarker supposed to inform?
| Intent | Question answered | Minimal design | Key estimand |
|---|---|---|---|
| Prognostic | What is this patient's outcome regardless of treatment? | Single-arm / observational cohort with outcome follow-up | Hazard ratio, C-index, calibration |
| Predictive | Who benefits more from treatment A vs B? | RCT or well-emulated target trial with both arms | Treatment × biomarker interaction |
| Diagnostic | Does this patient have the disease now? | Case–control with full disease spectrum | Sensitivity, specificity, PPV/NPV |
Consequences of the classification:
Pick based on the shape of the problem, not familiarity.
| Method | Strength | When it fails |
|---|---|---|
| LASSO (L1) | Sparse solutions, interpretable, fast | Picks one arbitrary feature from correlated groups |
| Elastic net (L1 + L2) | Handles correlated features, groups them | Two hyperparameters to tune |
| Random (survival) forest importance | Captures non-linearities and interactions | Biased toward high-cardinality features; importance ≠ causation |
| Stability selection | Reports features selected in >60% of subsamples; controls false discoveries | Computationally heavier; needs a base selector (usually LASSO) |
| Univariate filtering | Simple | Ignores multivariate structure; almost always wrong as a sole method in high-d |
Start
│
├── Is n < p (more features than samples)?
│ ├── Yes → Use penalization (LASSO / elastic net) OR stability selection.
│ │ Never rely on univariate filtering alone.
│ └── No → Continue.
│
├── Are features strongly correlated (e.g., gene modules, radiomic clusters)?
│ ├── Yes → Elastic net (groups correlated features) or
│ │ group LASSO if groupings are known a priori.
│ └── No → LASSO is fine.
│
├── Do you suspect non-linear effects or interactions?
│ ├── Yes → Random forest / gradient-boosted trees for importance,
│ │ confirm with permutation importance (not impurity-based).
│ └── No → Stay with linear penalized models.
│
└── Need reproducibility / publication-grade selection?
└── Wrap whatever base method above in stability selection
(subsample → refit → keep features chosen in ≥60% of runs).
Biomarker discovery must use nested CV:
Rules:
Leakage invalidates everything downstream. Check for:
Choosing the wrong metric hides or inflates effects.
Always report confidence intervals (bootstrap or analytic) alongside point estimates.
A biomarker graduates through tiers. Do not skip.
A biomarker is not validated until step 3 at minimum, and the model + threshold were locked before seeing the external data. Re-tuning on external data = discovery, not validation.
Start
│
├── Only one cohort available?
│ └── Internal validation only (nested CV + bootstrap optimism correction).
│ Label result as "discovery", not "validated".
│
├── Multi-site data within a single study?
│ └── Add internal-external (leave-one-site-out).
│ Report performance per site, not just pooled.
│
├── Truly independent cohort available?
│ └── Lock model + threshold → evaluate once → report with CIs.
│ Pre-register the analysis plan if possible.
│
└── Intended for clinical deployment / regulatory submission?
└── Prospective validation on the target population,
under intended-use conditions, with the locked model.
For Cox and logistic models: aim for 10–20 events per candidate variable considered during modeling (not just the final set). Counting only the selected variables after LASSO is a mistake — the selection itself used the full candidate pool.
In high-dimensional settings (omics, radiomics) events-per-variable is unattainable. Compensate with:
Event counts, not sample size, are the binding constraint for time-to-event outcomes. 1000 patients with 30 events ≈ a 30-event study.
When combining transcriptomics + proteomics + imaging + clinical:
Default with small-to-moderate sample sizes: late integration. Escalate to intermediate or early only if n supports it and shared-factor interpretation is a goal.
Missing modalities are the norm, not the exception — design for it from the start.
A biomarker with excellent AUC that does not change any decision is not useful. For every candidate, the agent should force the user to answer:
If the user cannot answer these, clinical utility is undefined — regardless of AUC.
Wrong: Claiming a biomarker is "predictive" from a single-arm study Right: Require a comparator arm (RCT or emulated target trial) to estimate a treatment-by-biomarker interaction Why: Without a counterfactual arm, you cannot distinguish prognostic from predictive — the biomarker may simply mark prognosis regardless of treatment
Wrong: Performing feature selection on the full dataset, then running CV on the reduced feature set Right: Always perform feature selection inside the CV inner loop so each fold selects features independently Why: Selected features already "know" the test labels, inflating performance estimates and producing unreproducible models
Wrong: Splitting by row when a patient contributes multiple samples Right: Split at the patient level so all samples from one patient stay in the same fold Why: Repeated measures, bilateral lesions, and longitudinal labs leak patient identity across folds, inflating apparent performance
Wrong: Tuning the decision threshold on the test set Right: Lock thresholds on training data or pre-specify them based on clinical cost-benefit reasoning Why: Optimizing the threshold on test data overfits to the evaluation set and produces unreliable operating characteristics
Wrong: Reporting PPV/NPV from a 50/50 case-control sample and implying it generalizes Right: Re-estimate PPV and NPV at the deployment prevalence using Bayes' theorem Why: PPV and NPV are prevalence-dependent — values from enriched samples are misleading at real-world disease rates
Wrong: Ignoring calibration when reporting model performance Right: Always report calibration (calibration-in-the-large, calibration slope, calibration plots) alongside discrimination Why: Good discrimination with poor calibration produces clinically dangerous probability estimates that misinform decisions
Wrong: Counting events-per-variable only after feature selection Right: Count EPV against all candidate variables considered during modeling, not just the final selected set Why: The selection process itself used the full candidate pool — the effective degrees of freedom reflect the original dimensionality
Wrong: Declaring a biomarker "validated" based on cross-validation on the discovery cohort Right: Require external replication on an independent cohort with a locked model and pre-specified threshold Why: Cross-validation is internal validation only — it estimates optimism but does not prove generalizability
Wrong: Confounding the biomarker signal with site, batch, platform, or scanner effects Right: Always check whether site/batch predicts outcome as well as the biomarker does; include site as a covariate or use leave-one-site-out validation Why: If all cases come from one site and controls from another, the model learns site identity, not biology
Wrong: Using future information as a feature (e.g., a lab ordered because of clinical suspicion) Right: Define a clean time origin and only use data available before it Why: A lab ordered because of suspicion encodes the suspicion itself — this is temporal leakage, not predictive signal
Wrong: Early integration of multi-omics with small sample sizes Right: Use late integration (fit one model per omic, combine predictions via stacking) when n is small to moderate Why: Concatenating thousands of features across layers with a few hundred patients overfits to batch effects, not biology
Wrong: Reporting only point estimates without confidence intervals Right: Always include bootstrap or analytic confidence intervals for all performance metrics Why: Single numbers hide instability — a C-index of 0.78 with a 95% CI of [0.55, 0.92] tells a very different story
Wrong: Optimizing AUC while ignoring clinical utility Right: Perform decision curve analysis to demonstrate net benefit over treat-all and treat-none strategies Why: A 0.85 AUC that changes no clinical decision is worse than a 0.70 AUC that reliably changes one
Wrong: Retraining or tuning the model on external validation data "just a little" Right: Lock the model completely before seeing external data — any tuning collapses validation back into discovery Why: Even minor adjustments on external data eliminate its independence, making the "validation" circular
Wrong: Skipping decision curve analysis when reporting biomarker performance Right: Always include net-benefit curves across the plausible threshold range Why: Without net-benefit reasoning, there is no evidence the biomarker helps patients — only that it correlates with outcome