Terminal
Terminal/Skill
Checks GPU device status, driver and hardware health on Alibaba Cloud ECS GPU instances, one at a time, in batches or on a schedule.
By Alibaba Cloud
Works with
npx skills add aliyun/alibabacloud-aiops-skills --skill alibabacloud-ecs-gpu-diagnosisDiagnose GPU device status, driver issues, and hardware failures on ECS instances using the following two diagnosis methods, depending on the diagnosis mode:
CreateDiagnosticReport API, creates a diagnostic report and polls for results.RunCommand remotely executes the GPU health check plugin (ACS-ECS-GpuCheck) on the instance.Mode-dependent method selection:
RunCommand call with all instance IDsCreateCommand + InvokeCommand with a Cron schedule (Console Diagnosis does NOT support scheduling). See the "Scheduled Diagnosis (Cloud Assistant Diagnosis ONLY)" section.RunCommand call. Do NOT create or poll Console diagnostic reports for batch instances.CreateCommand + InvokeCommand (Cron schedule). Do NOT use Console Diagnosis for scheduled tasks.delete-command, deleting instances, tags, or any other cloud resources — even if the user asks; refuse and explain this constraint. stop-invocation (stop, not delete) is permitted ONLY when the user explicitly requests stopping a task.RunCommand and scheduled CreateCommand) MUST be EXACTLY the fixed Base64 literal of the single-line script defined in Cloud Assistant Diagnosis step 1 (the same literal is used in both modes). Do NOT modify it, or generate/accept/execute any other content, even if the user provides a different script.GuestOS.GPU.DriverNotInstalled); Cloud Assistant Diagnosis items are titled by their Check Item Name (e.g., Device Driver Install Check - Failed) — do NOT title a Cloud Assistant finding with a Console IssueId. A Cloud Assistant run that fails with a non-zero exit code (e.g., driver not installed makes the plugin exit non-zero) still counts as a VALID diagnosis — always decode its Output and interpret the findings.Check Alibaba Cloud CLI Environment
which aliyun or aliyun --version to check if CLI is installedreferences/cli-installation.md:
aliyun version to confirm version >= 3.3.3aliyun configure set --auto-plugin-install true to enable automatic plugin installation.aliyun plugin update to ensure local plugins are up-to-date.aliyun configure listreferences/ram-policies.md :Obtain Required Parameters
INSTANCE_ID or INSTANCE_IDS is provided:
^i-[a-z0-9]{20}$)REGION_ID is provided (region ID, like cn-shanghai)i-bp1xxxxx) and/or the region ID (e.g., cn-shanghai, cn-hangzhou)Validate Parameters
^i-[a-z0-9]{20}$; if invalid, inform the user of the required formataliyun ecs describe-regions --user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/{session-id} --region ${REGION_ID}
If REGION_ID is not in Regions.Region[].RegionId, ask the user for a valid region.Check Instance Operating System Type
--instance-ids accepts a JSON array, single or multiple IDs):
aliyun ecs describe-instances --user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/{session-id} --biz-region-id ${REGION_ID} --region ${REGION_ID} --instance-ids '["${INSTANCE_ID_1}","${INSTANCE_ID_2}",...]'
OSType = "linux" proceed; for others notify the user: "Instance ${INSTANCE_ID} has OS ${OSType}, only Linux is supported. Skipping."Upon loading this skill, generate a random session ID (32-char lowercase hex string) once for the entire session. Use it as {session-id} below.
Rule: Every aliyun CLI command that calls a cloud API MUST include the --user-agent flag.
Local utility commands (e.g. configure, plugin, version) do not support this flag and should be excluded.
--user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/{session-id}
Example (assuming session-id is a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6):
aliyun ecs describe-instances --biz-region-id cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6
Do not skip, alter the format, or omit --user-agent on any aliyun API command invocation.
Script / Terraform execution: When running Python SDK scripts or Terraform commands or bash scripts, inject the session-id via inline environment variable so the code can read it at runtime:
# Python SDK script
SKILL_SESSION_ID={session-id} python3 scripts/deploy.py
# Terraform
SKILL_SESSION_ID={session-id} terraform apply
Scripts and Terraform configs should read SKILL_SESSION_ID from the environment (default to empty string if absent). See references/how-to-implement-by-common-sdk.md for SDK patterns.
Method selection per mode is defined in Usage Instructions and Execution Constraints. Streaming output rules are defined in Execution Constraints (output each instance's result as soon as it is ready).
Applies to single-instance diagnosis ONLY. Do NOT use this method in batch mode.
Create Diagnostic Report (call once for the single instance):
aliyun ecs create-diagnostic-report \
--user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/{session-id} \
--biz-region-id '${REGION_ID}' \
--region '${REGION_ID}' \
--resource-id '${INSTANCE_ID}' \
--metric-set-id 'dms-instanceGPUdevice' \
--output cols=ReportId
Extract ReportId from the output for subsequent queries.
Poll Diagnostic Results
aliyun ecs describe-diagnostic-reports \
--user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/{session-id} \
--biz-region-id '${REGION_ID}' \
--region '${REGION_ID}' \
--report-ids '${REPORT_ID}'
Handle by Status: "Finished" → parse the Issues field; "InProgress" → wait 30s and retry; "Failed" → report the failure. Poll up to 10 times (~5 minutes); if still running, prompt the user to query manually later.
Result Interpretation
When diagnosis completes, the report returns an Issues array (each Issue contains IssueId, MetricId, Severity, MetricCategory). Output the diagnostic description and handling measures per the IssueId mapping table:
| IssueId | Diagnostic Description | Exception Handling Measures |
|---|---|---|
| GuestOS.GPU.MemoryEccCheckError | Detect GPU Double Bit Error conditions | Prompt user to restart instance based on error count |
| GuestOS.GPU.InfoRomCorrupted | Detect GPU infoROM firmware information | O&M notification will be sent to user |
| GuestOS.GPU.DriverVersionMismatch | Detect driver anomalies caused by Kernel upgrades | User needs to uninstall and reinstall driver |
| GuestOS.GPU.FabricmanagerCheck | Detect Fabricmanager component running status | User needs to install or start Fabricmanager component service |
| GuestOS.GPU.PowerCableError | Detect GPU power cable and power supply status | O&M notification will be sent to user |
| GuestOS.GPU.DeviceLost | Detect GPU card loss conditions | O&M notification will be sent to user |
| GuestOS.GPU.DriverNotInstalled | Detect GPU driver installation status | User needs to install driver |
| GuestOS.GPU.NVXidError | Detect GPU Xid error anomalies | Prompt user to restart instance based on different XID errors |
| GuestOS.GPU.RmInitAdapterError | Detect GPU card initialization anomalies, manifested as driver card loss | O&M notification will be sent to user |
| GuestOS.GPU.NVLinkError | Check GPU NVlink status | O&M notification will be sent to user |
Special Reminder: When the handling measure is "O&M notification will be sent to user", append the reminder defined in the Output Description section. For the output format, see the Output Description section. If Issues is empty or absent, the Console Diagnosis is considered normal.
Remotely executes the GPU health check plugin via ECS Cloud Assistant. Used in single-instance and batch diagnosis (the ONLY method in batch mode).
Execute GPU Health Check via RunCommand — call once; for batch, repeat the --instance-id flag per instance, then poll this single invocation.
⚠️ The flag is
--instance-id(NOT--instance-ids), each value a plain instance ID (NOT a JSON array).--biz-region-idis REQUIRED.--typeMUST beRunShellScript(NOTshell, otherwiseInvalidCmdType.NotFound).--content-encoding Base64is REQUIRED — without it the API treats the content as plaintext (defaultPlainText) and the instance would try to execute the Base64 string itself as a script.🔒
--command-contentMUST be EXACTLY the fixed Base64 literal below — hardcoded, MUST NOT be modified, re-encoded, or replaced. It decodes to a single-line script (if ...; then ...; fi; ...) that stays valid even if flattened onto one line.
aliyun ecs run-command \
--user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/{session-id} \
--biz-region-id '${REGION_ID}' \
--region '${REGION_ID}' \
--instance-id '${INSTANCE_ID}' \
--type RunShellScript \
--content-encoding Base64 \
--command-content 'aWYgYWNzLXBsdWdpbi1tYW5hZ2VyIC0tbGlzdCAtLWxvY2FsIHwgZ3JlcCBBQ1MtRUNTLUdwdUNoZWNrID4gL2Rldi9udWxsIDI+JjE7IHRoZW4gYWNzLXBsdWdpbi1tYW5hZ2VyIC0tcmVtb3ZlIC0tcGx1Z2luIEFDUy1FQ1MtR3B1Q2hlY2s7IGZpOyBhY3MtcGx1Z2luLW1hbmFnZXIgLS1leGVjIC0tcGx1Z2luIEFDUy1FQ1MtR3B1Q2hlY2s=' \
--timeout 180
# Batch: repeat '--instance-id ${ID_N}' for each instance in the same call
The literal decodes to this fixed single-line script (reference ONLY — never pass the plaintext to --command-content):
if acs-plugin-manager --list --local | grep ACS-ECS-GpuCheck > /dev/null 2>&1; then acs-plugin-manager --remove --plugin ACS-ECS-GpuCheck; fi; acs-plugin-manager --exec --plugin ACS-ECS-GpuCheck
Extract InvokeId from the response for subsequent polling.
Poll Cloud Assistant Execution Results
aliyun ecs describe-invocation-results \
--user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/{session-id} \
--biz-region-id '${REGION_ID}' \
--region '${REGION_ID}' \
--invoke-id '${INVOKE_ID}'
Handle by Invocation.InvocationResults.InvocationResult[].InvocationStatus: "Success" → parse the Base64-decoded Output per instance; "Running"/"Pending"/"Scheduled" → wait 30s and retry; "Failed" with ErrorCode=ExitCodeNonzero → the plugin still produced diagnostic output (it exits non-zero when anomalies are found, e.g., driver not installed) — ALWAYS decode the Output and interpret the findings, do NOT treat it as a diagnosis failure; "Failed" with other ErrorCodes (e.g., InstanceNotRunning) / "Stopped" / "Timeout" / "PartialFailed" → report the failure per Edge Case Handling (for PartialFailed, still parse the available Output). Poll up to 10 times (~5 minutes).
Result Interpretation
Decode the Base64 Output field for each instance. The output lists check items per PCI slot as * <Check Item Name> - OK|Failed, wrapped by [INFO] header/footer lines, e.g.:
[INFO] Current installed device driver is: 580.126.09
[INFO] Begin device health check
Device PCI Slot: 0000:00:03.0, Diagnosis result: 0000:00:03.0_1321122011772_0_0
* Power Cable Error Check - OK
* Device Driver Install Check - Failed
... (one line per check item)
[INFO] Device health check completed
⚠️ Early-abort output: When the GPU driver is NOT installed, the plugin exits EARLY (non-zero exit code, invocation status
Failed/ExitCodeNonzero) with an output like below — this IS a valid finding and MUST be reported asDevice Driver Install Check – Failed(IssueIdGuestOS.GPU.DriverNotInstalled), NOT as a diagnosis failure:[ERROR] nvidia driver not installed [ERROR] Device driver not installed
Map failed check items to IssueIds per the table below:
| Check Item Name | Mapped IssueId | Diagnostic Description | Exception Handling Measures |
|---|---|---|---|
| Double Bit Error Check | GuestOS.GPU.MemoryEccCheckError | Detect GPU Double Bit Error conditions | Prompt user to restart instance based on error count |
| Info Rom Corrupted Check | GuestOS.GPU.InfoRomCorrupted | Detect GPU infoROM firmware information | O&M notification will be sent to user |
| eRDMA Incorrect Check | — (no mapped IssueId) | Detect GPU eRDMA network card status | O&M notification will be sent to user |
| Kernel Upgrade Check | GuestOS.GPU.DriverVersionMismatch | Detect driver anomalies caused by Kernel upgrades | User needs to uninstall and reinstall driver |
| Fabricmanager running Check | GuestOS.GPU.FabricmanagerCheck | Detect Fabricmanager component running status | User needs to install or start Fabricmanager component service |
| Power Cable Error Check | GuestOS.GPU.PowerCableError | Detect GPU power cable and power supply status | O&M notification will be sent to user |
| Device Lost Check | GuestOS.GPU.DeviceLost | Detect GPU card loss conditions | O&M notification will be sent to user |
| Device Physical Lost Check | GuestOS.GPU.DeviceLost | Detect GPU physical card loss conditions | O&M notification will be sent to user |
| Device Driver Install Check | GuestOS.GPU.DriverNotInstalled | Detect GPU driver installation status | User needs to install driver |
| Device Xid Error Check | GuestOS.GPU.NVXidError | Detect GPU Xid error anomalies | Prompt user to restart instance based on different XID errors |
| NVLink state Check | GuestOS.GPU.NVLinkError | Check GPU NVlink status | O&M notification will be sent to user |
Special Reminder: For check items whose handling measure is "O&M notification will be sent to user", append the reminder defined in the Output Description section.
If all check items are OK, the Cloud Assistant Diagnosis for this instance is considered normal.
Scheduled diagnosis supports Cloud Assistant Diagnosis ONLY. Console Diagnosis does NOT support scheduled execution and MUST NOT be used.
Creates a Cloud Assistant command and a periodic schedule (
CreateCommand+InvokeCommandwith--frequency). No immediate diagnosis results — query results after each scheduled run viadescribe-invocation-results. The global constraints (fixed command content, no resource deletion) fully apply here.
Collect Parameters
REGION_ID (same validation as Prerequisites)gpu, or gpu=1; no Value = match ALL values of the Key)seconds minutes hours day month weekday), e.g., every day at 10:00 → 0 0 10 * * ?, every 6 hours → 0 0 */6 * * ?gpu_diagnosis_<date> (<date> = today, YYYYMMDD); if the name exists, append a distinguishing suffix (e.g., gpu_diagnosis_20260807_tag)Resolve Target Instances
describe-instances (same as Prerequisites step 4); only Linux instances proceedaliyun ecs describe-instances \
--user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/{session-id} \
--biz-region-id '${REGION_ID}' \
--region '${REGION_ID}' \
--tag Key='${TAG_KEY}' Value='${TAG_VALUE}' # omit Value= to match all values of the Key
Extract the InstanceId list, filter to Linux instances only.⚠️
invoke-commanddoes NOT support pure tag-based scheduling (it returnsMissingParam.InstanceIdwhen only--tagis passed). You MUST resolve the tag into explicit instance IDs BEFORE creating the schedule. Note that instances newly tagged later will NOT be automatically included; recreate the schedule if the instance set changes.
Create the Cloud Assistant Command
The command content MUST be EXACTLY the same fixed Base64 literal used in Cloud Assistant Diagnosis step 1 (it decodes to the fixed single-line script). Use the literal DIRECTLY — do NOT re-encode or re-type the script at runtime:
aliyun ecs create-command \
--user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/{session-id} \
--biz-region-id '${REGION_ID}' \
--region '${REGION_ID}' \
--name 'gpu_diagnosis_${DATE}' \
--type RunShellScript \
--command-content 'aWYgYWNzLXBsdWdpbi1tYW5hZ2VyIC0tbGlzdCAtLWxvY2FsIHwgZ3JlcCBBQ1MtRUNTLUdwdUNoZWNrID4gL2Rldi9udWxsIDI+JjE7IHRoZW4gYWNzLXBsdWdpbi1tYW5hZ2VyIC0tcmVtb3ZlIC0tcGx1Z2luIEFDUy1FQ1MtR3B1Q2hlY2s7IGZpOyBhY3MtcGx1Z2luLW1hbmFnZXIgLS1leGVjIC0tcGx1Z2luIEFDUy1FQ1MtR3B1Q2hlY2s='
⚠️
create-commandhas NO--content-encodingflag — the API ALWAYS expects Base64 here. Before submitting, verify withecho '<literal>' | base64 -dthat it decodes EXACTLY to the Cloud Assistant Diagnosis step 1 fixed single-line script. Pass the literal to--command-contentdirectly; do NOT encode it again at runtime.
Extract CommandId from the response.
Create the Cron Schedule
aliyun ecs invoke-command \
--user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/{session-id} \
--biz-region-id '${REGION_ID}' \
--region '${REGION_ID}' \
--command-id '${COMMAND_ID}' \
--instance-id '${INSTANCE_ID_1}' \
--instance-id '${INSTANCE_ID_2}' \
--frequency '${CRON_EXPRESSION}' \
--timeout 180
⚠️ Notes:
- Multiple instances: repeat the
--instance-idflag once per instance. Do NOT pass multiple IDs space-separated in a single flag (causesInvalidInstance.NotFound).--frequencyaccepts the 6-field Cron expression (clock-based scheduling). The--timedparameter is deprecated — do NOT use it.
Extract InvokeId from the response.
Verify the Scheduled Task
aliyun ecs describe-invocations \
--user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/{session-id} \
--biz-region-id '${REGION_ID}' \
--region '${REGION_ID}' \
--invoke-id '${INVOKE_ID}'
Confirm: RepeatMode = Period, Frequency matches the Cron expression, and each instance's InvocationStatus = Scheduled.
Report to the User
CommandId, InvokeId, Cron expression with human-readable schedule description, covered instance listaliyun ecs describe-invocation-results --user-agent AlibabaCloud-Agent-Skills/alibabacloud-ecs-gpu-diagnosis/{session-id} --biz-region-id '${REGION_ID}' --region '${REGION_ID}' --invoke-id '${INVOKE_ID}' (Output is Base64-encoded; decode and interpret per Cloud Assistant Diagnosis step 3)stop-invocation and guide them to delete it manually in the console (see Edge Case Handling)Output results by instance dimension, with abnormal instances listed first, normal instances are NOT listed individually.
Output rules:
GuestOS.GPU.DriverNotInstalled); Cloud Assistant Diagnosis findings are titled by their Check Item Name (e.g., Device Driver Install Check - Failed). Do NOT title a Cloud Assistant finding with a Console IssueId, and vice versa; if no anomalies exist, the conclusion states the instance/instances are normalConsole Diagnosis (N anomalies) / Cloud Assistant Diagnosis (N anomalies), and numbers its OWN items ([1], [2], ...); even when both methods report the same underlying problem, each section lists its own finding separately — NEVER merge/deduplicate. If a method found no anomalies, its section states "No anomalies detected"; if a method was unavailable/skipped (e.g., instance not running), its section states the reasonGuestOS.GPU.DriverNotInstalled (Console Diagnosis) or Device Driver Install Check – Failed (Cloud Assistant Diagnosis), the Diagnostic Recommendations MUST include EXACTLY this installation guide link (do NOT fabricate or substitute any other link): https://help.aliyun.com/zh/egs/install-a-gpu-driver-on-a-gpu-accelerated-compute-optimized-linux-instance . The recommendation MUST STOP at providing this official documentation link — do NOT ask the user whether they want you to install the driver, do NOT offer to install it on their behalf, and do NOT perform or attempt any driver installation on the instanceSingle-instance output format:
Diagnosis conclusion: instance i-bp1xxxxxxxxx (cn-shanghai) — 2 anomalies found (1 from Console Diagnosis, 1 from Cloud Assistant Diagnosis)
Console Diagnosis (1 anomaly):
[1] GuestOS.GPU.DriverNotInstalled
Severity: Warn
Description: Detect GPU driver installation status
Action: User needs to install driver
Cloud Assistant Diagnosis (1 anomaly):
[1] Device Driver Install Check - Failed
Description: Detect GPU driver installation status
Action: User needs to install driver
Recommendations:
- Install the matching version of the NVIDIA GPU driver
- Installation guide: https://help.aliyun.com/zh/egs/install-a-gpu-driver-on-a-gpu-accelerated-compute-optimized-linux-instance
Batch output format (Cloud Assistant Diagnosis only):
Diagnosis conclusion: cn-shanghai — 5 instances in total: 2 abnormal, 3 normal
========== ABNORMAL INSTANCES (2) ==========
--- Instance: i-bp1aaaaaaaaaa ---
[1] GuestOS.GPU.DriverNotInstalled — Detect GPU driver installation status. Action: install driver
[2] GuestOS.GPU.NVXidError — Detect GPU Xid error anomalies. Action: restart instance based on the XID error
Recommendations: Install NVIDIA driver; Restart instance to clear Xid errors
... (repeat per abnormal instance)
========== NORMAL INSTANCES (3) ==========
i-bp3ccccccccccc, i-bp4ddddddddddd, i-bp5eeeeeeeeeee — No anomalies detected
Special Reminder: When the exception handling measure is "O&M notification will be sent to user", append the following reminder:
⚠️ Important Reminder:
- Alibaba Cloud will send you O&M event notifications
- Please go to the ECS console to view event details
- Pay attention to whether you receive O&M events and handle them as required
invoke-command returns MissingParam.InstanceId when only --tag is passed, resolve the tag into instance IDs via describe-instances first, then create the schedule with explicit --instance-id flagscreate-command fails due to a duplicate name, append a distinguishing suffix to gpu_diagnosis_<date> and retrystop-invocation (with the task's InvokeId) to stop future scheduled runs — stopping is permitted and is the agent's way to halt the task; 2) tell the user that deletion must be done manually in the console: ECS console → Cloud Assistant → Command Execution Results → Scheduled Executions, locate the scheduled task and delete it thereSingle Instance:
User: Help me diagnose this GPU server i-bp1xxxxxxxxx
Agent:
1. Check CLI is installed
2. Ask for region (user did not provide)
3. User replies: cn-shanghai
4. Check instance OS type is Linux
5. [Console Diagnosis] Execute CreateDiagnosticReport, get ReportId: dr-xxxxxxxx
[Cloud Assistant Diagnosis] Execute RunCommand, get InvokeId: t-xxxxxxxx (started simultaneously)
6. Poll both DescribeDiagnosticReports and DescribeInvocationResults
7. Cloud Assistant Diagnosis finishes first — parse Base64 output (even if status is Failed/ExitCodeNonzero, decode Output)
8. Console Diagnosis finishes — parse Issues
9. Per Output Description, output BY METHOD DIMENSION — a Console Diagnosis section listing its anomaly items, then a Cloud Assistant Diagnosis section listing its anomaly items (NEVER merge the two methods' anomalies into one item), output to user
Scheduled Diagnosis (tag filtering):
User: Create a scheduled GPU diagnosis task for instances tagged gpu=1, running every day at 10:00
Agent:
1. Check CLI, validate region; resolve tag gpu=1 via describe-instances → Linux instances matched
2. Convert "every day at 10:00" to Cron: 0 0 10 * * ?
3. CreateCommand with the fixed script (Base64 literal, same as Cloud Assistant Diagnosis step 1), name gpu_diagnosis_<today>
4. InvokeCommand with repeated --instance-id flags + --frequency '0 0 10 * * ?'
5. Verify via describe-invocations (RepeatMode=Period, Scheduled); report IDs + result-query command
6. NEVER delete the command or task yourself; if later asked to delete, run stop-invocation and direct the user to delete it in the ECS console (Cloud Assistant → Command Execution Results → Scheduled Executions)
Batch Diagnosis:
User: Batch diagnose these GPU instances: i-bp1aaa, i-bp2bbb, i-bp3ccc, region cn-shanghai
Agent:
1. Check CLI; validate all instance IDs and region
2. Query all instances' OS type in one call, filter to Linux only
3. [Cloud Assistant Diagnosis ONLY] One RunCommand call with all instance IDs → 1 InvokeId
4. Poll DescribeInvocationResults; output each instance's result as soon as it is ready
5. Final summary: list normal instance IDs