Terminal
Terminal/Skill
Runs and interprets GPU cluster checks on Azure HPC systems: NCCL bandwidth, GPU and thermal tests, InfiniBand, and draining bad nodes.
By Microsoft
Works with
npx skills add azure/ai-infrastructure-on-azure --skill ai-infraOperational knowledge for Azure HPC GPU clusters running on CycleCloud Workspace for Slurm or AKS, with NVIDIA GB300 (NDv6) and H100 (NDv5) GPUs.
This single skill is the entry point for all cluster validation, diagnosis, and remediation questions. It contains a routing index that points to detailed reference files in references/. Read the relevant reference files before giving commands — do not invent thresholds or procedures.
Standard_ND128isr_GB300_v6 or Standard_ND96isr_H100_v5), nodelist, cluster name, orchestrator (Slurm or AKS), failing job context.If the user invokes this skill with /ai-infra <request>, treat that as an explicit invocation and proceed with the same workflow.
All reference files live in skills/ai-infra/references/.
| File | Covers |
|---|---|
sku_baselines.md | Expected NCCL busbw, GPU GFlops, thermal limits, IB ports, rack sizes for GB300 and H100. Warn and GHR thresholds. |
rack_topology.md | MNNVL domain discovery via nvidia-smi, ClusterUUID, expected rack sizes per SKU, FabricManager troubleshooting. |
ib_validation.md | InfiniBand port state (operstate, ibstat), partition keys, error counters, link flap detection, soft fixes. |
nccl_diagnosis.md | Bisection algorithm, intra-rack vs inter-rack scoping, GPU vs network root cause, bandwidth pattern interpretation. |
outlier_detection.md | Statistical methods (absolute threshold, z-score, MAD) for fleet-wide GPU and NCCL analysis. |
ghr.md | Azure Guest Health Reporting (GHR) — used for reporting a node failure to Azure. Covers full impact category reference, IMDS/KVP data collection, REST API format, insight polling. |
| File | Covers |
|---|---|
nccl_test.md | Run all_reduce_perf via the Slurm launcher. Per-SKU env vars (MNNVL/SHARP/GDR), output columns, quick vs full sweep. AKS path: see infrastructure_validations/aks/NCCL/. |
gpu_validation.md | Run ubergemm GEMM benchmark via Slurm. Parse CSV output, identify underperforming GPUs. |
thermal_test.md | Run dcgmproftester thermal stress via Slurm. Interpret pass/fail, throttle reasons, DCGMI levels. AKS path: see infrastructure_validations/aks/. |
| File | Covers |
|---|---|
node_drain_and_replace.md | Slurm drain/undrain/reboot, decision tree for drain vs reboot vs GHR, post-replacement validation. |
Read: sku_baselines.md, rack_topology.md, nccl_test.md, gpu_validation.md, thermal_test.md
Read: nccl_diagnosis.md, sku_baselines.md, ib_validation.md, rack_topology.md (when topology correlation is needed)
Read: nccl_test.md, nccl_diagnosis.md, ib_validation.md, outlier_detection.md (fleet-wide analysis)
Read: gpu_validation.md, outlier_detection.md, sku_baselines.md
Read: thermal_test.md, sku_baselines.md, gpu_validation.md (if thermal impact on GEMM is suspected)
Read: ib_validation.md, nccl_diagnosis.md, rack_topology.md (when rack-locality matters)
Read: outlier_detection.md, sku_baselines.md, plus gpu_validation.md and/or nccl_test.md depending on the metric source
Triggers: "file a GHR", "guest health report", "report a bad node to Azure", "report a node failure to Azure", "open a hardware issue with Azure", "health impact category".
Read: node_drain_and_replace.md, ghr.md
These apply to every operational answer:
ghr.md for exact commands.sku_baselines.md. Do not invent warn/GHR cutoffs.#wrong in NCCL output, dcgmproftester non-zero exit, XID errors in dmesg are all hard signals.ghr.md. Use HpcGenericFailure only when nothing else fits.infrastructure_validations/ in this repo. The references describe how to run them and how to interpret output; the runnable artifacts are the scripts themselves.infrastructure_validations/slurm/NCCL/ — NCCL all_reduce_perf launcher with per-SKU configsinfrastructure_validations/slurm/gpu_test/ — GPU GEMM benchmark (ubergemm)infrastructure_validations/slurm/thermal_test/ — Thermal stress test (dcgmproftester)infrastructure_validations/slurm/NHC/ — Node Health Checkinfrastructure_validations/aks/NCCL/ — NCCL on AKS (via MPI Operator)infrastructure_validations/aks/NHC/ — Node Health Check on AKSinfrastructure_validations/aks/fio/ — Storage I/O test on AKSFor operational responses, follow this structure:
sku_baselines.md or the relevant test reference.