Terminal
Terminal/Skill
Helps operators fix GPU and model serving problems on AKS: pods stuck Pending, CUDA driver mismatches, OOM on model load, and idle GPU cost.
By Microsoft
Works with
npx skills add azure/aks-skills --skill aks-gpu-inferenceOperate and troubleshoot GPU and model-serving workloads on AKS after they exist. This is the failure → check → fix loop, not a setup guide — for enabling GPUs, KAITO, or a model runway, use azure-skills airunway-aks-setup.
Most "my GPU workload won't run" incidents are one of these, and they stack — clearing one exposes the next:
az vm list-usage --location <region> -o table and look for the family row (e.g. Standard NC Family vCPUs, Standard NCADSA100v4 Family vCPUs). Error signature: "Insufficient vCPU quota for Standard NCASv3_T4 Family…".az vm list-skus --location <region> --resource-type virtualMachines -o table.sku=gpu:NoSchedule. A GPU pod must carry the matching toleration (key=sku, value=gpu, effect=NoSchedule), request nvidia.com/gpu: 1, and (by convention) select accelerator: nvidia. A missing toleration is the most common Pending cause after quota.nvidia.com/gpu advertised on the node → a driver / device-plugin model mismatch. See the driver decision matrix in references/gpu-scheduling.md.| Symptom | Reference |
|---|---|
Pod Pending "Insufficient nvidia.com/gpu"; taints/quota/scheduling; driver models; --gpu-driver | references/gpu-scheduling.md |
| KAITO Workspace never becomes ready; conditions; the delete-doesn't-delete-the-pool cost trap | references/kaito-workspaces.md |
| Idle GPU cost, scale-to-zero, spot eviction, SKU-to-model right-sizing, KEDA on GPU | references/gpu-cost-and-scaling.md |
| GPU utilization/VRAM metrics, DCGM, OOMKilled on weight load | references/gpu-observability.md |
# Are GPUs schedulable? (capacity/allocatable nvidia.com/gpu + the accelerator label)
kubectl describe node <gpu-node> | grep -EA6 "Labels:|Capacity:|Allocatable:"
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu'
# Why is a GPU pod Pending?
kubectl describe pod <pod> | grep -A15 Events
# Is the driver/device-plugin stack present?
kubectl get pods -A -o wide | grep -Ei "nvidia|device-plugin|dcgm"
az aks nodepool show -g <rg> --cluster-name <cluster> -n <pool> --query gpuProfile
# GPU vCPU quota by family (the exact "Insufficient vCPU quota" check)
az vm list-usage --location "<region>" -o table | grep -Ei "Total Regional|NC|ND"
curl localhost:9400/metrics against a managed pool returns nothing.DCGM_FI_DEV_FB_USED/FB_FREE) already shows the trend. Size the SKU to the model up front.