Terminal
Terminal/Skill
Checks whether AgentCore agents send traces, metrics, and logs to CloudWatch, and lists exact fixes for any gaps.
By Amazon Web Services
Works with
npx skills add aws/tools-for-devops-agent --skill agentcore-observability-setupValidate and bootstrap observability for Amazon Bedrock AgentCore workloads, aligned with the AgentCore observability documentation. Produce a readiness report that states, per check, what is configured vs. what should be, and generate the exact remediation for every gap.
Verify where reachable; prescribe everywhere else. Every check is exactly one of:
This skill is read-only. It validates and generates configuration/commands. The customer decides whether to apply them. Never mutate IAM, resources, or configuration.
Activate when the user asks to:
This skill degrades gracefully based on which read-only permissions the DevOps Agent role holds. Detect the tier by attempting calls and noting failures; run the checks the tier allows and report what to add to unlock the rest. Never attempt to modify IAM.
| Tier | Permissions | Unlocks |
|---|---|---|
| 1 — Standard (default DA IAM) | logs:DescribeLogGroups, logs:FilterLogEvents, logs:GetLogEvents, logs:StartQuery, logs:GetQueryResults, cloudwatch:GetMetricData, cloudwatch:ListMetrics, cloudwatch:DescribeAlarms | Telemetry-arrival verification for any host |
| 2 — Runtime config | bedrock-agentcore:GetAgentRuntime, bedrock-agentcore:ListAgentRuntimes, xray:GetTraceSegmentDestination, logs:DescribeDeliveries, logs:DescribeDeliverySources, logs:DescribeDeliveryDestinations, logs:DescribeResourcePolicies | Runtime tracing/env, Transaction Search state, Memory/Gateway delivery, X-Ray resource policy |
| 3 — Non-runtime host | lambda:GetFunctionConfiguration; ecs:DescribeTaskDefinition, ecs:DescribeServices, ecs:ListTasks; eks:DescribeCluster | Host-side config verification for Lambda / ECS / EKS |
If Tier 2/3 permissions are absent, tell the user the check is prescriptive-only here and give the
exact scoped inline policy to attach to the DevOps Agent role (see references/iam-tiers.md) so a
re-run can verify it. The shared managed policy is cross-tenant and must not be modified per-skill.
Host detection:
bedrock-agentcore:GetAgentRuntime / bedrock-agentcore:ListAgentRuntimes (Tier 2). If absent, ask the user.Surfaces to assess (ask which apply, or discover via Tier 2 List*):
Runtime agents · Memory resources · Gateway resources · Built-in tools (code interpreter, browser) · non-runtime host.
Outcome matrix:
| Host / Surface | Expected outcome |
|---|---|
| Runtime agent | Verify (Tier 1+2) + Prescribe |
| Memory resource | Verify delivery + tracing (Tier 2) + Prescribe |
| Gateway resource | Verify delivery + tracing (Tier 2) + Prescribe |
| Built-in tools | Prescribe; verify telemetry-arrival where present |
| Lambda | Verify host config (Tier 3) + Prescribe |
| ECS | Verify host config (Tier 3) + Prescribe |
| EKS | Verify telemetry arrival (Tier 1) + Prescribe config (pod env is K8s-RBAC-gated, not IAM-verifiable) |
| On-prem / multi-cloud | Prescribe only |
Assign each finding a severity (CRITICAL / HIGH / MEDIUM / LOW / INFO) and a type (VERIFY / PRESCRIBE).
The complete check catalog with APIs, log-group patterns, and pass/fail logic is in
references/checks-catalog.md. Summary below.
xray:GetTraceSegmentDestination → destination must be CloudWatchLogs and status ACTIVE. If XRay/inactive, Transaction Search is not enabled → CRITICAL (spans will not be delivered to CloudWatch Logs; this is the single most common silent root cause).references/remediation-runtime.md).logs:DescribeLogGroups prefix /aws/bedrock-agentcore/runtimes/. Missing → HIGH.logs:FilterLogEvents on the runtime log stream in the last 24h. No recent events after invocations → HIGH.spans log stream in the agent's log group (unified destination) or the shared aws/spans log group. No spans despite invocations → CRITICAL (points to Transaction Search disabled or missing instrumentation).cloudwatch:ListMetrics namespace bedrock-agentcore. Absent → MEDIUM.bedrock-agentcore:GetAgentRuntime → inspect env for UNIFIED_TRACES_DESTINATION_ENABLED and DISABLE_ADOT_OBSERVABILITY. Starting 2026-07-20, newly created agents in supported AWS Regions default to the unified span destination (the agent's own log group) — UNIFIED_TRACES_DESTINATION_ENABLED=false is now the opt-out. Agents created before that date remain on shared aws/spans unless opted in. If ADOT observability disabled unintentionally → HIGH.logs:DescribeResourcePolicies — must allow xray.amazonaws.com to logs:PutLogEvents on the agent's log group. Applies by default for agents created after 2026-07-20 in supported Regions, since unified span destination is the new default (per the AgentCore release notes). Mark N/A only when the agent was created before 2026-07-20 and still delivers to shared aws/spans, or the customer has explicitly opted out via UNIFIED_TRACES_DESTINATION_ENABLED=false. Missing → HIGH.aws-opentelemetry-distro>=0.10.0 (≥0.18.0 for unified span destination) + boto3 in requirements.txt; launch with opentelemetry-instrument python main.py (container CMD ["opentelemetry-instrument","python","main.py"]); framework tracing enabled (e.g. Strands tracer, opentelemetry-instrumentation-langchain); session id via X-Amzn-Bedrock-AgentCore-Runtime-Session-Id.logs:DescribeDeliveries / DescribeDeliverySources / DescribeDeliveryDestinations for an APPLICATION_LOGS source on the resource ARN and a TRACES→XRAY delivery. Default log group /aws/vendedlogs/bedrock-agentcore/{memory|gateway}/APPLICATION_LOGS/{resource-id}. No delivery → HIGH (no logs), MEDIUM (no traces).put_delivery_source/put_delivery_destination/create_delivery SDK sequence (see references/remediation-memory-gateway.md).X-Amzn-Trace-Id, traceparent) on the tool APIs.lambda:GetFunctionConfiguration → OTEL Layer present and AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument; agent log group env vars set. Gaps → HIGH. Note: Lambda uses the Layer, not the aws-opentelemetry-distro package.ecs:DescribeTaskDefinition → container env carries the OTEL variable set (see 3.6). Gaps → HIGH.Required variables (per docs): AGENT_OBSERVABILITY_ENABLED=true, OTEL_PYTHON_DISTRO=aws_distro,
OTEL_PYTHON_CONFIGURATOR=aws_configurator, OTEL_RESOURCE_ATTRIBUTES (service.name, aws.log.group.names,
cloud.resource_id), OTEL_EXPORTER_OTLP_LOGS_HEADERS, OTEL_EXPORTER_OTLP_TRACES_HEADERS (optional
unified spans; needs ADOT ≥0.18.0 + X-Ray log-group resource policy), OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf,
OTEL_TRACES_EXPORTER=otlp. Full block with Lambda-specific vars in references/remediation-non-runtime.md.
ADOT Collector is not supported — use the ADOT SDK or the Lambda Layer only.
Produce a shareable artifact named agentcore-observability-review-<target>-<YYYY-MM-DD>.md
(<target> = agent/runtime id or a user-supplied label).
Sections:
# | Check | Type (VERIFY/PRESCRIBE) | Severity | Current state | Expected | Remediation.references/ file).| Severity | Definition |
|---|---|
| CRITICAL | Observability is broken — telemetry cannot flow (e.g. Transaction Search disabled, no spans despite invocations). |
| HIGH | Significant gap — a required piece is missing (log group, delivery, resource policy, host env). |
| MEDIUM | Notable gap — metrics/traces partially configured or best-practice not met. |
| LOW | Minor hardening or optimization. |
| INFO | Observation, no action required. |
references/checks-catalog.md — full per-check catalog (API, logic, severity, VERIFY/PRESCRIBE).references/iam-tiers.md — the three read-only tiers + ready-to-attach scoped inline policy JSON.references/remediation-runtime.md — Transaction Search, runtime tracing, ADOT, X-Ray resource policy.references/remediation-memory-gateway.md — log delivery + tracing (console + SDK).references/remediation-non-runtime.md — Lambda / ECS / EKS / on-prem OTEL env and setup.