Terminal
Terminal/Skill
Read-only review of SageMaker AI endpoints, training jobs, notebooks and domains against AWS best practices, including production readiness checks.
By Amazon Web Services
Works with
npx skills add aws/tools-for-devops-agent --skill sagemaker-ai-ops-reviewRun the Amazon SageMaker AI operational review checks against a customer's Amazon SageMaker AI
resources and produce an Amazon SageMaker AI Operational Review report. It evaluates
8 pillars, 20 checks using native AWS APIs (via use_aws). The Best Practices pillar's
recommendations are grounded in the public AWS Well-Architected lenses — see pillar-checks.md.
This is a strict READ-ONLY review: data is collected through native AWS List* /
Describe* control-plane APIs, CloudWatch metric reads, servicequotas:GetServiceQuota,
health:DescribeEvents, and savingsplans:DescribeSavingsPlans. It performs no model
invocations, launches no jobs, and reads no inference payloads.
Activate this skill when the user asks to review, audit, or assess an Amazon SageMaker AI workload, check SageMaker AI best-practices posture, or run an Operational Readiness Review (ORR) for SageMaker AI — for one pillar, a subset of checks, or the full set.
The ORR use case is the primary one: run this review before a team deploys a SageMaker AI workload to production, as the readiness gate. Because every finding is severity-ranked and carries a concrete remediation, the report doubles as the pre-production punch list — clear the High and Medium findings, then launch. It is equally suited to a recurring cadence afterwards (weekly or monthly posture review) and to an ad-hoc audit of a newly inherited account.
Run checks grouped by pillar in the order below. Load references/pillar-checks.md for
each check's APIs, logic, thresholds, and output fields.
| Pillar | Checks |
|---|---|
| Security | Check Encryption · SageMaker VPC Check · VPC Configuration Check |
| Performance | SageMaker Endpoint Inference Type · SageMaker Endpoint Latency |
| Cost Optimization | SageMaker Resource Tagging Check · Trainium and Inferentia Usage · Autoscaling Endpoint Check · Sagemaker Savings Plan · Sagemaker Lifecycle Configurations · Sagemaker Inference Recommender Jobs Check · Sagemaker Stale Endpoints Check |
| Service Quotas | Service Quotas Check |
| Resiliency | SageMaker Endpoint Instances · SageMaker Lifecycle Events |
| Operational Excellence | Sagemaker Project Check · Sagemaker Pipeline Check · SageMaker Endpoint Datacapture Enabled Check |
| Sustainability | Domain Region Check |
| Best Practices | Well-Architected Recommendations (SageMaker AI) |
Confirm with the user:
sts:GetCallerIdentity). If regions are unspecified, discover active regions with ce:GetCostAndUsage (SERVICE = "Amazon SageMaker", grouped by REGION); Cost Explorer is payer-scoped, so if it returns nothing, fall back to sweeping a default region set with sagemaker.list-endpoints/list-domains/list-notebook-instances. Conclude "no activity" only after both come back empty.references/pillar-checks.md is authoritative on each).For each in-scope check, call the APIs listed in references/pillar-checks.md via use_aws
and build the check's result rows. Follow this behavior:
List* then Describe*; paginate every call that returns a token.{ error } row — a failed
check never aborts the review.us-east-1, never inside the per-region loop:
health (describe-events, describe-affected-entities), ce (get-cost-and-usage), and
savingsplans (describe-savings-plans). They have no regional endpoints. Looping them per
region fails everywhere but us-east-1, and the failure mimics the checks' legitimate
degradation paths — a Health error looks like "no Business/Enterprise Support plan", a Savings
Plans error looks like "permission not granted" — so the report states a plausible wrong reason
instead of surfacing a bug. Health returns events for all regions; filter to the in-scope
regions client-side.ModelLatency / OverheadLatency are published in microseconds — label the
column and also give the millisecond conversion. An unlabelled six-figure latency reads as
milliseconds and manufactures a false performance escalation.AIDevOpsAgentAccessPolicy on the DevOps Agent role. The one exception —
savingsplans:DescribeSavingsPlans (Savings Plan check) — is an optional add-on. The AWS
Health APIs used by the Lifecycle Events check are covered by the managed policy but
additionally require a Business/Enterprise Support plan. On AccessDenied for a check, report it
as "not evaluated — permission not granted" and continue; never emit a false "none found"
from an access error.references/pillar-checks.md:
High, Medium, Low, or Informational (inventory checks with no pass/fail signal).
Checks with a compliance signal set severity as defined there — e.g. Studio domain not VpcOnly
→ High; no autoscaling / an Inference Component endpoint whose host instance fleet is fixed while its
components autoscale / idle endpoint at least 90 days old / notebook with no customer-managed
KMS key / no VPC config / Savings Plan expired or within 30 days of expiry / AWS Health event with
actionability = ACTION_REQUIRED → Medium; missing tags / data capture disabled → Low. The Service
Quotas Check derives its tier from utilization (≥ 90% High, ≥ 75% Medium, else Low; Unknown if no
usage data).(check, region, resource).
Do not aggregate resources into a single finding — three notebooks with no customer-managed
key are three Medium findings, not one. Aggregation breaks the severity counts and makes runs incomparable.Produce a single Markdown report titled "Amazon SageMaker AI Operational Review", with the structure below.
# Amazon SageMaker AI Operational Review
**Account IDs:** <comma-separated account IDs>
**Regions:** <comma-separated regions>
**Date Range:** <range or "Not specified">
> **AI Disclaimer:** The AI-generated insights in this report are provided for informational purposes only. They should be reviewed and validated by qualified personnel before taking any action. AWS is not responsible for any decisions made based on AI-generated content.
## Executive Summary
<severity-ranked roll-up of findings across all pillars: count by severity (High / Medium /
Low), then the High and Medium findings listed most-severe first, each with its one-line
recommendation. Omit only if there are no High/Medium/Low findings at all.>
## <Pillar Name>
### <Check Name>
**Guidance**
<what the check evaluates and the relevant SageMaker best practice>
**AI Insights**
<optional per-check analysis of the gathered data; prefix with a note that it is AI-generated and must be verified. Omit if not generated.>
**Data**
<a Markdown table of the check's result rows (fields per references/pillar-checks.md, including a `severity` column for checks that define one), or "No data available for this check.">
**Recommendations**
<one concrete SageMaker-specific recommendation per High or Medium finding in this check, each labelled with its severity. Omit this block entirely if the check has no High/Medium findings.>
Rules:
2026-09-18 (point-in-time)). Do not inline every check's window into it;
per-check windows are fixed by the checks and belong in each check's own section.## section per in-scope pillar, in the table order above; one ### sub-section per
check in that pillar. Include every in-scope check even when it found nothing (render its
empty-state row).references/pillar-checks.md,
including the severity field for checks that define one.references/pillar-checks.md.These bound what the review can honestly conclude. The skill's README is not packaged into the uploaded skill, so these are restated here where the runtime can actually read them. Where a limitation applies to a check you ran, say so in that check's Guidance rather than letting the reader assume wider coverage.
Check Encryption covers notebook instances only. Training jobs, processing jobs, endpoint
configs, S3 model artifacts, and Feature Store stores are not assessed for encryption. Never
present the Security pillar as a complete encryption audit — name the gap. Note also that a
notebook without a KmsKeyId is still encrypted (system-managed key); the finding is the absence
of a customer-managed key, never "not encrypted". The remediation is re-creation, not an
update — UpdateNotebookInstance has no KmsKeyId parameter, so never name it; the key is settable
only at creation.Native AWS APIs only: sagemaker, cloudwatch (get-metric-statistics, get-metric-data,
list-metrics), application-autoscaling (describe-scalable-targets,
describe-scaling-policies), servicequotas (get-service-quota), ce (get-cost-and-usage
for region discovery), health (describe-events, describe-affected-entities), plus the one
optional add-on savingsplans (describe-savings-plans). All but that add-on are covered by
the AWS-managed AIDevOpsAgentAccessPolicy. health, ce, and savingsplans are global —
call each once against us-east-1, outside the per-region loop. No data-plane calls and no
non-AWS tooling — the skill is self-contained on the DevOps Agent's cloud-source IAM role.