Terminal
Terminal/Skill
Helps plan machine learning studies on healthcare and life sciences data, covering splits, metrics, calibration, fairness and regulatory readiness.
By Amazon Web Services
Works with
npx skills add awslabs/hcls-agent-skills --skill ml-researcherThis skill teaches the agent how to think about machine learning experiments on healthcare and life sciences data. HCLS ML differs from general ML: samples are scarce, labels are noisy, distributions shift across sites and time, subgroup harms are real, and deployment is governed by reporting standards and regulators. The dominant failure mode is not model choice — it is leakage, miscalibration, and evaluation that does not mirror deployment.
Use this skill to structure a study before any code is written, to critique an existing pipeline, or to decide whether a model is ready to advance.
Invoke this skill when the user:
Outputs should be prescriptive: name the design, splits, metrics, and reporting checklist. Challenge assumptions; ask for the index time, the cohort definition, the decision point, and the site structure before recommending models.
Walk the chain in order. A gap at any link invalidates downstream choices:
If evaluation does not mirror deployment, the model is not evaluated.
Common in omics, rare-disease imaging, biomarker discovery.
Default stance: do nothing to the data.
Decision tree:
Is the positive class rare (e.g., <10%)?
├── No → standard training, report AUPRC alongside AUROC.
└── Yes → Is the model's ranking adequate (AUPRC vs. prevalence baseline)?
├── Yes → shift the decision threshold to match clinical costs.
│ Use decision curve analysis to pick the operating point.
└── No → try class weights or focal loss BEFORE resampling.
If you must resample:
- Resample only the training fold.
- Recalibrate probabilities afterward (Platt/isotonic on held-out fold).
- Never resample the validation or test fold.
Resampling distorts the prior and breaks calibration. Most "imbalance problems" are really threshold problems.
Patients are longitudinal. Random splits leak the future into the past.
For every feature, ask: Was this value known at index time? If not, it cannot be used.
Required construction:
Default choices. Deviate only with justification.
| Modality | First choice | Strong alternative | Notes |
|---|---|---|---|
| Tabular clinical (structured EHR, labs, vitals) | Logistic regression + elastic net | XGBoost / LightGBM | Start linear for interpretability and calibration. Tree GBMs usually win accuracy with modest gain. |
| Medical imaging (2D, 3D, WSI) | Transfer learning from ImageNet/RadImageNet/foundation models | nnU-Net for segmentation | Fine-tune with aggressive augmentation; freeze early layers on small data. |
| Temporal EHR (sequences of events) | GRU or Transformer with time embeddings | Tabularize (aggregate over windows) + GBM | Tabularized GBM is a hard baseline; beat it before claiming a sequence model helps. |
| Genomics (expression, methylation) | Elastic net on pathway-aggregated features | GNN on protein–protein-interaction graph | Pathway aggregation reduces dimension and increases biological interpretability. |
| Graphs / small molecules | GNN (GIN, D-MPNN) with scaffold splits | Fingerprint + GBM | Random splits on molecules overstate performance 2–5×. |
| Clinical text | Domain-adapted encoders (ClinicalBERT, BioClinicalBERT, Gatortron) | Instruction-tuned clinical LLM with retrieval | General-domain BERT underperforms; adapt to clinical vocabulary. |
What is the input?
├── Structured rows (one row per patient) → Tabular branch
│ ├── n small or interpretability required → logistic + elastic net
│ └── n moderate/large, accuracy priority → GBM (XGBoost/LightGBM)
├── Images → Imaging branch
│ ├── Classification → transfer learning (foundation model preferred if in-domain)
│ └── Segmentation → nnU-Net
├── Longitudinal events → Temporal branch
│ ├── Fixed feature set works → tabularize + GBM (baseline, often wins)
│ └── Irregular, long sequences, rich history → Transformer with time encoding
├── Sequences with biology (DNA/RNA/protein) → specialized encoder or pathway features
├── Molecules → GNN with scaffold split (or fingerprint + GBM baseline)
└── Free text → domain-adapted encoder; RAG if generative
Always run a trivial baseline (majority class, logistic regression, or existing clinical score). If the deep model cannot beat it by a clinically meaningful margin, stop.
The split defines the claim. Match the split to the deployment claim.
What claim is being made?
├── "Works on new patients at this hospital" → patient-level random split
├── "Works at other hospitals" → site-level holdout (leave-site-out CV)
├── "Works in the future" → temporal holdout + forward-chaining CV
├── "Works on new chemical series" → scaffold split
└── "Works on a new population/demographic" → demographic holdout + subgroup eval
Report all three. Always.
Is the task classification with a clinical decision attached?
├── Yes → discrimination + calibration + DCA (all three).
│ Report subgroup performance for prespecified groups.
└── No (ranking only, research use) → discrimination + calibration; DCA optional.
Is prevalence < ~10%?
├── Yes → lead with AUPRC, include AUROC; show PR curve.
└── No → AUROC primary, AUPRC secondary.
Are probabilities used downstream (risk communication, triage threshold)?
├── Yes → calibration is mandatory. Recalibrate on a held-out fold if off.
└── No → calibration still reported for transparency.
Bias in HCLS data is rarely random. Labels themselves encode historical care disparities.
Match the standard to the artifact:
Require the checklist before writing. Missing items are almost always missing experiments, not missing sentences.
For models intended as Software as a Medical Device (SaMD):
Each of these invalidates a study. Treat finding any of them as a stop-the-line event.
Wrong: Including features derived from post-outcome data (e.g., post-event labs, discharge codes) in the training set Right: Audit every feature against the index time — only data available at the decision point may be used Why: Outcome-containing features yield near-perfect AUROC in development and zero real-world performance
Wrong: Randomly splitting longitudinal patient data or molecular datasets without respecting grouping Right: Use patient-level grouping for clinical data and scaffold splits for molecules — no entity in both train and test Why: Random splits leak future information or structural similarity, producing inflated performance estimates (2–5× overstatement for molecules)
Wrong: Reporting only AUROC on imbalanced data (prevalence < 10%) Right: Always report AUPRC with prevalence baseline and decision curve analysis alongside AUROC Why: AUROC can be 0.95 while the model is clinically useless — AUPRC and DCA reveal whether the model helps at realistic operating points
Wrong: Using resampling (SMOTE, oversampling) as the primary fix for class imbalance Right: Adjust the decision threshold first; if resampling is truly needed, apply only within training folds and recalibrate afterward Why: Resampling distorts the prior and breaks probability calibration — most "imbalance problems" are actually threshold problems
Wrong: Claiming generalizability from a single-site study Right: Require site-level holdout or external validation from a different hospital/scanner/EHR vendor Why: One site's data reflects its specific population, workflows, and coding practices — this does not generalize
Wrong: Evaluating a foundation model on data that overlaps with its pretraining corpus Right: Demand a clean, post-cutoff, or curated held-out benchmark with no pretraining overlap Why: Any overlap between pretraining data and evaluation set inflates performance and masks true generalization ability
Wrong: Computing normalization statistics, imputation parameters, or feature selection on the full dataset before splitting Right: Refit all preprocessing steps inside each training fold — treat them as part of the model Why: Full-dataset preprocessing leaks test-set information into training, producing optimistic and unreproducible results
Wrong: Tuning hyperparameters using the test set (single-loop CV) Right: Use nested CV — outer loop for performance estimation, inner loop for hyperparameter tuning Why: Test-set tuning reports a tuned score, not a generalization estimate — the test set is no longer independent
Wrong: Skipping subgroup analysis because subgroup sample sizes are small Right: Report subgroup performance with wide confidence intervals rather than hiding the group Why: Small subgroups are often the most vulnerable to model harm — silence about them is not safety
Wrong: Deploying a model with high AUROC without checking calibration Right: Always assess calibration (slope, intercept, reliability diagram) — recalibrate if probabilities do not match observed rates Why: Probabilities drive thresholds, triage, and patient communication — an uncalibrated model with high AUROC is clinically dangerous
If any box is unchecked, the answer to "is it ready?" is no.