Terminal
Terminal/Skill
Adds Agent Observability SDK instrumentation to your own LLM app, then checks with gcx that generations actually arrive in Grafana Cloud.
By Grafana Labs
Works with
npx skills add grafana/gcx --skill agento11y-instrumentHelp a developer wire their own LLM app or agent to Grafana Agent Observability, from zero or from a broken/partial state, and keep going until data actually lands in the stack. The value this skill adds over the static instrumentation prompt is two things a prompt can't do:
gcx: after each change, run the app and check the
gcx agento11y agents / conversations / generations commands to confirm generations arrived.
Diagnose the next gap from what's missing, not from guesswork.The SDK reference (env vars, provider snippets, field lists, framework adapters, workflow steps)
lives in agento11y's llms.txt "Path B". Fetch it rather than restating it here; this file holds
the flow and the decision logic. A minimal fallback lives in
references/instrumentation.md for when the fetch is unavailable.
https://raw.githubusercontent.com/grafana/agento11y/main/llms.txt (Path B). Only inline decision
logic here. If the fetch fails, fall back to references/instrumentation.md.AGENTO11Y_ENDPOINT,
AGENTO11Y_PROTOCOL, AGENTO11Y_AUTH_MODE, AGENTO11Y_AUTH_TENANT_ID, AGENTO11Y_AUTH_TOKEN,
OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS) or ask the developer. Never fabricate a
URL or mint a token.AGENTO11Y_* env vars, never SIGIL_*. SIGIL_* is a deprecated legacy fallback. Do
this even if sibling apps or existing .env files in the repo use SIGIL_* — matching a stale
local convention perpetuates it. If the app already reads SIGIL_*, add the AGENTO11Y_* names
(the SDK still honors both) and note the old ones are deprecated. Do not "match the siblings."langchain-anthropic) to make the run succeed. If the developer separately says
they want a different model, that is an app change they own — tell them to make it and re-invoke
this skill; do not fold it into the instrumentation diff. Swapping the model silently changes what
the app does and what gets observed, which defeats the point. The provider API key
(ANTHROPIC_API_KEY, OPENAI_API_KEY, …) is the app's own concern, not the instrumentation's —
it authenticates the LLM call, not the telemetry export, and the app already has it if it runs at
all. So don't ask for it, configure it, or rewire it; if the live verify-run fails on a missing
provider key, skip the run and report verified-by-construction (see Step 5). Just don't conflate the
two 401s: a 401 on generation ingest is observability auth and is yours to fix (usually a
missing AGENTO11Y_PROTOCOL/AGENTO11Y_AUTH_MODE); an auth error from the model provider is
not — surface it and let the developer handle their own key.cache_write_input_tokens, NOT cache_creation_input_tokens. agent_version
maps to the gen_ai.agent.version label and is required for per-version Performance charts.
MessageRole (Python SDK) has only USER, ASSISTANT, TOOL — there is no SYSTEM (or
DEVELOPER) member; MessageRole.SYSTEM raises AttributeError. Fold the system prompt into the
USER message (or a text_part), and prefer the user_text_message() / assistant_text_message()
/ tool_result_message() helpers over hand-building Message(role=...). Always confirm enum members
and helper names against the installed package before running — do not assume from llms.txt.agento11y-test-starter; tenant eval rules + guards on
real traffic → agento11y-prod-setup. Coding-agent telemetry plugins (Claude Code, Cursor, …) →
llms.txt "Path A". Any control-plane write.The app needs, in its environment before the SDK starts, seven vars — not five; the two mode vars are the ones most often forgotten, and their absence is a silent 401:
AGENTO11Y_ENDPOINT, AGENTO11Y_PROTOCOL=http, AGENTO11Y_AUTH_MODE=basic,
AGENTO11Y_AUTH_TENANT_ID, AGENTO11Y_AUTH_TOKEN.OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS.AGENTO11Y_PROTOCOL=http and AGENTO11Y_AUTH_MODE=basic are required for Cloud, not optional —
the SDK defaults (grpc / no-auth) return a silent 401 against the Cloud HTTP ingest endpoint. (They
scope the ingest channel only; the OTel channel's transport/auth is set entirely by the OTEL_*
vars — see references/instrumentation.md.) The only var that is sometimes omittable is
OTEL_EXPORTER_OTLP_HEADERS: required when sending directly to the Cloud OTLP gateway (the common
case, gateway enforces Basic auth), omittable only when OTEL_EXPORTER_OTLP_ENDPOINT points at a
local Alloy / OTel Collector that already holds the Cloud credentials. First check what's already set
(including any existing .env) — if all are present, skip to Step 1. Watch for the near-miss where
the endpoint is set under the wrong name (e.g. AGENTO11Y_API_ENDPOINT — the SDK reads
AGENTO11Y_ENDPOINT, so the wrong name is silently ignored and ingest falls back to a default host).
When any value is missing, do NOT just list the variable names and ask — hand the developer the exact place to get each one (link + clicks), every time. The concrete sources are in point 3 below; surface them proactively. The most common failure of this skill is naming
OTEL_EXPORTER_OTLP_ENDPOINT/OTEL_EXPORTER_OTLP_HEADERSand leaving the developer to guess — the answer is the stack OTLP tile + "Generate now", which precomputes both. Give that first.
What gcx does for you (run these):
gcx config current-context — is there a working context? If not, just ask the developer to log
in to the stack they want the instrumentation to connect to — e.g. "run gcx login against your
stack." Do not fabricate the login command yourself (don't guess the host or flags); let them run
their own login (the Agent Observability setup screen gives them the exact command, or they use
gcx login). Instrumentation itself needs no gcx login — only Step 5 verification does, so this
never blocks writing the code.gcx cloud stacks list, then gcx cloud stacks get <stack-slug> — identify the target stack and
its URLs. This gives you the stack to point the developer at, and confirms which tenant the Step 5
verification will read from.What still needs the Connection page (gcx cannot do these today):
gcx does not generate the Agent Observability OTLP gateway URL, and does not mint the ingest / access-policy token. When you ask the developer for a value, always tell them exactly where to get it — a link and the clicks — never just name the variable and wait. The two channels come from two different places:
OTEL_* (traces/metrics) — easiest, let Cloud build them. Send the developer to the stack's
OTLP tile: https://grafana.com/orgs/<org-slug>/stacks/<stack-id>/otlp-info. It already shows
OTEL_EXPORTER_OTLP_ENDPOINT and the Instance ID; under Password / API Token → "Generate now"
it mints a token and then fills an Environment Variables block with all OTEL_* vars — the
base64 OTEL_EXPORTER_OTLP_HEADERS is precomputed, ready to copy. No manual base64. (In Python,
the value uses Basic%20… — keep it as given.)
AGENTO11Y_* (generation ingest) — the plugin Connection page. AGENTO11Y_ENDPOINT and the
token come from https://<stack>.grafana.net/plugins/grafana-agento11y-app → Connection tab. When the
developer creates the token via "Create a token in Cloud Access Policies", tell them the scopes:
sigil:write, metrics:write, traces:write, logs:write. UI heads-up: sigil is not in
the default resource list — add it via "Add scope" (then tick Write); the scope is still
sigil:* (the Cloud resource keeps the old name). The same glc_… token works for both channels
if it has all four scopes. Also set AGENTO11Y_PROTOCOL=http and AGENTO11Y_AUTH_MODE=basic (see
references/instrumentation.md — the SDK defaults grpc/none give a 401).
Ask the developer to put the values in a gitignored .env (or export them) themselves — do
not ask them to paste a secret token into the chat (it is captured in the transcript). Instrument
the code to read from the environment and have them supply the values out-of-band. Never invent a
URL or mint a token.
Two different tokens — don't confuse them. gcx logs in with its own OAuth token (
gat_) and refreshes it automatically; that is what authenticates thegcxcommands here. It is not the app's ingest token. The app needs a separate access-policy token (glc_…) inAGENTO11Y_AUTH_TOKEN/OTEL_EXPORTER_OTLP_HEADERS, and gcx does not create that one.
Once gcx has a working context, the Step 5 verification commands (under the gcx agento11y group)
work against the developer's tenant even before the app's own credentials are fully wired.
The Connection page is the only manual step. If a future gcx release can create the access-policy token and surface the OTLP endpoint, this step collapses to gcx-only — but do not assume it can today; check
gcx cloud --helprather than promising it.
Find and read, recording file:line for each:
TracerProvider / MeterProvider would be created.agento11y / @grafana/agento11y / the Go
agento11y package — or the legacy sigil_sdk / @grafana/sigil-sdk-js / Go sigil names in
older code) or AGENTO11Y_* usage.Detect:
pyproject.toml/.py, package.json/.ts,
go.mod/.go, gradle/.java, .csproj/.cs).langgraph, langchain, openai-agents, llamaindex, google-adk,
strands, pydantic-ai, litellm, claude-agent-sdk, vercel-ai-sdk, crewai, or a custom
orchestrator.Before recommending a provider wrapper or framework adapter, confirm it exists for this language (fetch llms.txt "SDK API surface" — the matrix is not symmetric across languages).
State the classification explicitly, with the file:line evidence that led to it:
none (greenfield) — no SDK import anywhere. Full setup from scratch.partial — an SDK client is constructed and some generations are wrapped, but coverage is
incomplete (no OTel providers, no agent_version, no workflow steps for an agentic pipeline, no
parent links). Run the full checklist; recommend + apply only the gaps.broken — the SDK is present but wrong: metrics silently lost (no MeterProvider), export
misconfigured, legacy SIGIL_* vars, stream/non-stream mode mismatch, set_result/SetResult
never called, or rec.err()/Err() never checked. Fix first, then gap-check.All three paths converge on the same checklist (Step 3); they differ only in how much is already done.
Walk each item against the code. Record PRESENT / MISSING / WRONG with file:line. This mechanical
audit is the skill's core value. Items 0, 1, 2, 5, 6 fail silently (no error) — always check them.
Items 3, 7, 8 mean data lands but analysis is degraded. For the fix, read the named section of
the fetched llms.txt (locate it by its heading — do not trust line numbers, they drift).
| # | Check | Silent-failure symptom | llms.txt section |
|---|---|---|---|
| 0 | The .env actually takes effect. Confirm the app loads its own .env by an explicit path (not a bare load_dotenv() resolved by CWD) and that it wins over vars already in the environment. Verify by printing os.environ["AGENTO11Y_ENDPOINT"] / OTEL_EXPORTER_OTLP_ENDPOINT after all imports, not before | import litellm (and some other libs) inject localhost OTLP/ingest defaults into os.environ at import time (OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4318, AGENTO11Y_ENDPOINT=localhost:8080). A plain load_dotenv() does not override already-set vars → the Cloud endpoints in .env never apply and everything ships to localhost, returning 200 OK if a local stack is up. Zero error signal, and gcx against the Cloud tenant shows nothing. Fix: load_dotenv(<path-relative-to-__file__>, override=True) before constructing providers/client. A bare load_dotenv() also resolves the wrong .env by CWD | "Environment" |
| 1 | OTel TracerProvider and MeterProvider created before the SDK client (verify by construction + Performance view / OTLP POSTs — not via gcx, which can't see OTel; see Step 5) | spans/metrics go to no-op → all latency/token/cost metrics lost. The #1 failure. | "OTel setup (required)" |
| 2 | Providers shut down after shutdown() | last batch of spans/metrics dropped on exit | "OTel setup (required)" |
| 3 | agent_name + agent_version set on generations / handlers | per-version Performance charts break (join on gen_ai.agent.version) | "Agent Observability architecture and ingest model", "Telemetry fields to prioritize" |
| 4 | set_result/SetResult includes response_id, response_model, finish/stop reason, full token usage (incl. cache_read_input_tokens, cache_write_input_tokens, reasoning_tokens), and input/output populated with Message objects (system+user prompt in input, model reply in output) | charts/cost blank; wrong cache_creation_input_tokens name silently ignored; empty input/output → the conversation thread shows "No messages in this turn" — tokens land but there is no visible prompt/response | "Implementation rules", "Telemetry fields to prioritize" |
| 5 | rec.err()/Err() checked after the recorder closes | SDK validation/enqueue errors are silent → generations never arrive, no signal | "Implementation rules" |
| 6 | SYNC (non-stream) vs STREAM (stream) set correctly | streaming metrics (TTFT) corrupted | "Agent Observability architecture and ingest model", "Implementation rules" |
| 6b | operation_name is a recognized value — generateText (SYNC default), streamText (STREAM default), embeddings, execute_tool, framework_chain, framework_retriever. Best: omit it and take the SDK default. Do not invent one like "chat" | the span reaches Tempo but the UI classifies gen_ai.operation.name as unknown → the conversation renders a synthetic generation node with no attached span → the trace does not show in the conversation and the "T" (trace) icon is absent, even though trace_id/span_id are set. Silent, like #1 | "Agent Observability architecture and ingest model", "Implementation rules" |
| 7 | parent_generation_ids set on multi-agent / fan-in generations | no dependency DAG; upstream eval failures don't propagate | "Multi-agent dependency tracking" |
| 8 | Workflow steps emitted for agentic pipelines with non-LLM nodes | execution graph invisible; node input/output state lost. Use the adapter if one exists, else enqueue_workflow_step; never both for one node (duplicates) | "Workflow step instrumentation (agentic pipelines)" |
| 9 | Env vars are AGENTO11Y_* (not legacy SIGIL_*); client built config-free when env present | drift; duplicated config | "Environment" |
| 10 | Content-capture mode intentional (SDK default no_tool_content); no secrets in tags/metadata/user_id | PII/secrets leak into exports | "Content capture", "Tags, metadata, and user id" |
| 11 | Client tags low-cardinality; end-user identity via user_id, not a tag | high-cardinality tags blow up metric labels | "Tags, metadata, and user id", "Implementation rules" |
Emit the report using llms.txt's output contract (its "Output contract" section): top opportunities
first, and per opportunity — exact file:line, why it matters, a concrete diff proposal, a test
plan, and any risk. Rank by severity: .env not taking effect (#0 — nothing lands at all) first,
then missing OTel provider (metrics data loss), then broken export, then missing agent_version,
then coverage gaps. Every recommendation cites a file:line. Then stop and ask
before applying anything.
Only after the developer confirms a diff. Bounded to ~3–4 iterations.
Apply the confirmed diff (Edit/Write). Order of preference: provider wrapper → framework
adapter → hand-instrument the core SDK — only what exists for the language. Add/update a focused
test for the changed instrumentation. Preserve flush/shutdown lifecycle. Never touch app logic.
Pull exact usage from llms.txt / the per-language README / examples/getting-started/*.
Run the app for one turn to generate traffic (ask the developer to run it, or run it if there's a clean entrypoint and they approve). If the run can't happen — missing provider API key, no clean entrypoint, needs a full runtime — do not work around it by changing the model or adding a provider. Stop the loop, report the wiring as verified-by-construction (imports resolve, providers build, client + handler construct), and tell the developer the one thing left is to run one turn themselves with their key. A verified-by-construction result is a fine outcome.
Verify — two independent channels, don't conflate them. Instrumentation sends data on two separate paths, and confirming one says nothing about the other.
First, confirm gcx reads the same tenant the app writes to. A verification against the wrong
tenant is worse than no verification — an empty agents list gets misread as "data isn't landing"
when it is, just elsewhere. Before drawing any conclusion from a gcx query: read the app's
AGENTO11Y_ENDPOINT + AGENTO11Y_AUTH_TENANT_ID from its .env, then check gcx config current-context and that the active context points at that same stack/tenant. If it doesn't (e.g.
context is local but the app writes to a Cloud stack), switch context or ask the developer to
gcx login to the right stack — do not guess the login command. (If the gcx token is merely
expired, that blocks Step 5 verification only, not writing the code — say so and continue.)
Channel A — generations (the SDK ingest client → /api/v1/generations:export). Carries the
prompt, response, tokens, cost, model, finish_reason. This is what gcx can read.
gcx agento11y agents list (does the agent appear?);
gcx agento11y agents get <agent-name> (is generation_count climbing?). To find the run's
conversation, either gcx agento11y conversations list --limit <n> (most-recent first, no
filters — the quickest post-run check) or gcx agento11y conversations search --filters 'agent = "<agent-name>"' (--filters alone is enough; --from/--to, RFC3339, are optional
and only needed to narrow a busy tenant). Then gcx agento11y conversations get <conversation-id>
or gcx agento11y generations get <generation-id> — check tokens, finish reason, cost, and (for
a multi-agent pipeline) that parent_generation_ids reproduce the DAG. This proves generation
ingest + set_result are wired. It does NOT prove OTel is wired.Channel B — OTel spans/metrics (the TracerProvider/MeterProvider → OTLP exporter →
/v1/traces, /v1/metrics). Carries latency/token/cost metrics. This is checklist #1, the
#1 silent failure — and gcx cannot see it (traces/metrics land in the stack's Tempo/Prometheus,
not the Agent Observability ingest API). So verify Channel B separately, by the strongest signal
available, in this order:
POST …/v1/traces → 2xx and
POST …/v1/metrics → 2xx. The SDK emits spans automatically from start_generation/end (one
traces POST per generation) and metrics on an interval — a clean shutdown flush surfaces both.
Traces and metrics are separate exports: seeing only /v1/metrics does NOT mean traces work,
and vice-versa. Never claim "traces/metrics verified" from a probe that only exercised one of
them — that is the exact trap that reports Channel B as done when half of it was never sent. If
you write a throwaway verification script, it must build the TracerProvider and MeterProvider
and record a real generation, or just instrument the app and read its debug output.service.name). If conversations appear
(Channel A) but Performance is empty, the MeterProvider is missing or no-op → back to checklist
#1. Do not report OTel as wired on the strength of generations get alone, nor on metrics alone.gen_ai.operation.name is an unrecognized value (e.g. "chat") → the UI classifies it as
unknown and can't attach it to the conversation node. This is checklist #6b — fix
operation_name to a recognized value (or omit it for the default) and re-run.If a signal is missing, diagnose the next gap from what the checks showed, propose the fix, and
loop back to step 1. After ~3–4 iterations without full signal, stop and report exactly what
lands, what doesn't, and what to check next (app stderr for agento11y: warnings, credentials).
Once generations land and metrics populate, instrumentation is done — that's the prerequisite for
everything else. Point the developer at the next step: agento11y-test-starter to build an
offline test suite for the agent (useful before shipping and for regression-testing new
versions once it's live), and agento11y-prod-setup to set up online eval rules + guards on
real traffic once it's deployed. The split is offline test suite vs online rules/guards — not
before-traffic vs after-traffic.
The SDK reference (env vars, provider snippets, field lists, workflow-step schema, adapter matrix) is
intentionally not duplicated here — it lives in agento11y's llms.txt "Path B" and the
per-language READMEs, which are the shipped source of truth. This skill holds only decision logic
(state classification + gap checklist + the gcx verification loop). When a user-facing semantic
changes (new SDK field, renamed env var, new framework adapter), update llms.txt (and its onboarding
wizard copy); this skill points at llms.txt by section heading, not line number (line numbers
drift as llms.txt is edited), so no re-pointing is needed unless a heading itself is renamed. If you
find yourself pasting a provider snippet into this file, stop — fetch llms.txt instead. The
references/instrumentation.md fallback is deliberately minimal for the same reason.