Terminal
Terminal/Skill
Builds an offline starter test suite for an AI agent before it ships, from its code, with a runner stub and suggested evaluators.
By Grafana Labs
Works with
npx skills add grafana/gcx --skill agento11y-test-starterHelp a developer test an AI agent before it ships — while there is no real traffic yet. The hard part isn't running the test — it's having cases to test against and knowing how to score them, grounded in the agent's actual code.
Scope: this is the pre-production, offline skill — it writes test cases and a local runner, and never touches the tenant. Once the agent is deployed and has real traffic, setting up online eval rules + guards on that traffic is a different skill,
agento11y-prod-setup.
Always produce:
Then, depending on how runnable the agent is (Step 1):
run_experiment.py) that wires the suite
to the SDK with two holes to fill — and optionally run it (Step 6), only with permission.
For an agent that needs a harness or full runtime, point to the existing eval infra instead
of a runner that can't actually call it.This skill is language-agnostic — the reading, recommending, and YAML it produces do not
depend on the agent's language. What differs is how runnable the agent is: recommendations +
YAML always apply, but the runner (Step 4) and the optional run (Step 6) adapt to whether the
agent has a clean function seam or needs a harness / full stack. For deeper run-side patterns
(binding existing generations, cross-process verifiers) point to the per-language run skill
(Python: agento11y-experiments).
Note:
agento11y-experimentscurrently ships in the grafana/agento11y repo (python/skills/agento11y-experiments/), not in this gcx bundle yet — install it from there for now. Consolidating it into the gcx bundle is pending.
The generated runner imports the Agent Observability SDK. Install it in the agent's environment before running (Step 6):
pip install agento11y python-dotenv (the experiments API lives in
agento11y.experiments; the runner uses python-dotenv to load the agent's .env, and it is
not a dependency of agento11y)github.com/grafana/agento11y/goOnly needed to run the suite (Step 6). Reading the agent, recommending evaluators, and writing the suite YAML (Steps 1–5) need nothing installed.
AGENTO11Y_ENDPOINT and AGENTO11Y_AUTH_TOKEN; if the
endpoint isn't set, ask for it, do not invent one..env or an exported env var they supply
themselves) — do not ask them to paste a secret token into the chat (it is captured in the
transcript), and do not create one.llm_judge uses the LLM provider the agent already uses — don't add a new one. If the
agent calls OpenAI, the judge calls OpenAI; if Anthropic, Anthropic. Do NOT default the judge to
litellm (or any other provider SDK) when the app doesn't already depend on it — that adds a
dependency and a second provider just for scoring. Reuse the agent's existing client/SDK.AGENTO11Y_ENDPOINT and
AGENTO11Y_AUTH_TOKEN (the ingestion key from the Connection page) — the SDK raises before
making any request if the token is empty, so both are always required. Read the endpoint from an
existing AGENTO11Y_ENDPOINT / .env / sibling app; never invent one or mint a token.Find and read these in the target repo, and record the file path and line range of each:
from openai import OpenAI → OpenAI; from anthropic import Anthropic → Anthropic.gpt-4o-mini, claude-sonnet-5) → the MODEL_NAME the runner uses.
State both explicitly, and have the judge use the same provider as the agent (never a
different one). If you can't tell from the code, ask; don't default to a provider the app doesn't use..env (load_dotenv()), a secret manager, an explicit client(api_key=...). The
runner should reuse the same mechanism (it already calls load_dotenv(), and provider SDKs
read their standard env var themselves). Don't hardcode a key, and don't tell the developer to
"set OPENAI_API_KEY" as if that's the only way — if the run can't find the key, ask them how
their app loads it and mirror that.Every recommendation must cite one of these locations.
Also assess runnability — how hard it is to invoke this agent for one input and get its output — because it decides what Step 4 produces. Classify it as one of:
run_agent.Signals that push toward in-process/full-stack: the agent isn't Python; tools hit real
APIs/datasources; there is already a dedicated eval harness, Docker stack, or
scenario/ground-truth files in the repo. If a repo already has eval scenarios with expected
outputs, note them — they are better test cases than anything generated, and later steps
should point to or reuse them.
Separately, note the agent's language, because the Agent Observability experiments SDK exists only in Python and Go (not JS/TS, Java, or .NET yet). This is independent of runnability — a TS agent can be trivially runnable yet have no experiments SDK in its language. If the agent is Python or Go, the runner is native. If it is any other language, say so plainly: the runner must be Python or Go (calling the agent across a process boundary), or the developer waits for experiments support in their language. Do not imply a native JS/Java/.NET experiments API exists.
The SDK models an evaluator as an id plus a kind. There are two kinds:
llm_judge — a model grades the output against a rubric (relevance, helpfulness,
groundedness, tone, task completion, format, safety, and similar judgment calls).deterministic — code decides pass/fail (exact/substring match, JSON validity, schema
or regex shape, length bounds, "not empty", a required field is present).Pick 3–6 (not more). Map each to what you read in Step 1, and give a one-line why per pick
that cites a file:line. Common mappings:
| If the agent… | Evaluator | Kind |
|---|---|---|
| answers open-ended user requests | relevance, helpfulness | llm_judge |
| retrieves / cites sources / does RAG | groundedness | llm_judge |
| must stay on-topic / in-scope | task_adherence | llm_judge |
| must emit JSON or a fixed shape | json_valid / schema_match | deterministic |
| must follow a specific output format | format_adherence | llm_judge |
| calls tools | tool_call_correct | llm_judge (or deterministic if the correct call is checkable in code) |
| handles user data that could be echoed | pii_leak | llm_judge |
| produces public-facing text | toxicity | llm_judge |
| must always return something | response_not_empty | deterministic |
Evaluator ids are yours to choose — pick clear, stable ids. Be concrete about what "defining"
each one means, because it is not an Agent Observability control-plane action here: in the offline SDK flow
an evaluator is code the developer writes in the runner (Step 4). An llm_judge is a
function that calls a model and returns a score; a deterministic one is a plain code check.
agento11y.Evaluator(evaluator_id=..., kind=...) is only the label attached to the score, not the
logic. (Forking a predefined template is a separate online-eval path, not needed here.)
This skill targets the offline phase: run these evaluators as offline experiments against the suite from Step 3, before there is traffic. Do not recommend live/online evaluation to an agent with no traffic.
Once the agent ships and has traffic, the same evaluation criteria can also be applied online — as Agent Observability Rules over ingested conversations, or as SDK guard hooks on the request path — but those are separate Agent Observability surfaces, configured elsewhere, and out of scope for this skill. Mention this only as a one-line "next, once you have traffic" note; do not instruct on it here.
Write a suite file in the target repo (suggest evals/<agent>-starter.yaml). It must load
with the SDK's TestSuite.from_yaml(...), so match this schema exactly:
suite_id (required), plus optional name, version, description, tags,
changelog, and cases (a list). version defaults to 1.0.0.id (required), plus optional name, description, tags, category,
input, expected, weight, metadata. input and expected are free-form (a string
or a mapping).Derive cases from the agent's real task. Produce at least 6, weighted toward edge and
adversarial (generated cases skew easy). Use category values happy, edge,
adversarial. Keep the header comment verbatim.
Every case must actually reach the agent — test the agent, not its harness. From Step 1, note where the entrypoint parses/validates input before the model runs (e.g. a JSON parse, a CLI arg check). Do not generate cases that fail in that pre-agent layer (malformed JSON, wrong CLI flags) as if they tested the agent; if such a boundary is worth covering, label it clearly as a harness case in its notes, or leave it out.
# STARTER DRAFT — review before use. Generated from your agent code (<file:line refs>).
# NOT validated. Add your own real cases; the edge/adversarial cases need your judgment on
# expected behavior. Loads with agento11y.TestSuite.from_yaml(...).
suite_id: <agent>-starter
name: <Agent> starter suite
version: 1.0.0
cases:
- id: happy-basic-request
category: happy
tags: [smoke]
input:
prompt: "<a real request this agent is built to answer>"
expected: "<what a good answer looks like, or a rubric note>"
- id: edge-underspecified
category: edge
input:
prompt: "<a vague / multi-part / boundary request>"
expected: "<how the agent should handle ambiguity>"
- id: adversarial-prompt-injection
category: adversarial
input:
prompt: "<an injection / out-of-scope / data-extraction attempt>"
expected: "<agent should refuse / stay in scope / not leak>"
The suite YAML alone does not run anything. The Agent Observability SDK stores and aggregates scores, but it does not run the agent or compute the evaluators — the developer writes both. Left at just the YAML, a developer new to offline eval is still blocked ("now what?"). So generate a minimal bootstrap runner — just enough to get one experiment running.
This is deliberately the simplest path, not the full run-side API. The agento11y-experiments
skill is the reference for everything beyond bootstrap — binding an already-instrumented
agent's real generations/conversations, auditable LLM-judge grading, cross-process verifiers
(TrialRef), pass@k/pass^k. Don't reproduce those patterns here; generate the minimal runner
and point to agento11y-experiments for depth.
What you generate depends on the runnability you assessed in Step 1:
evals/run_experiment.py below. One hole to fill
(run_agent).run_agent that can't actually call the agent (e.g.
a Go agent). Generate the same experiment wiring, but make run_agent shell out to a small
harness in the agent's language (or write that harness), and say plainly the seam is the
injectable LLM client. Point to any existing test that already invokes the agent in-process
as the template.Also branch on language (the experiments SDK is Python/Go only):
run_experiment.py, or the Go agento11y package).node your-agent.js and reads its output), or the developer waits
for experiments support in their language. Offer the subprocess bridge only as a labeled
option with its cost (serializing input to the CLI, parsing output), not as a clean default.For an easy Python/Go agent, write evals/run_experiment.py. It must:
TestSuite.from_yaml(...).agento11y.experiment(...)) and one trial per case.run_agent(case) — the first of the
two holes the developer fills; wire it to the real entrypoint you found in Step 1.llm_judge — a real model
call that returns a JSON {score, passed, explanation}), so they see the shape and can copy
it for the others. Reference the rest by name in a comment; do not stub all of them. The
judge's model call must use the provider the agent already uses (reuse its OpenAI/Anthropic
client or SDK) — do not pull in litellm or another provider just for the judge. The one model
call inside the judge is the second hole — leave it as an explicit NotImplementedError so a
developer who only fills run_agent gets a clear "fill the judge call" error, not a NameError
on an undefined helper.trial.record_io(...)) and emit trial.final_score(...) with the evaluator.Keep the header verbatim, and be honest in it about what still needs doing:
#!/usr/bin/env python3
"""STARTER RUNNER — generated by agento11y-test-starter, review before use.
Runs <agent> over evals/<agent>-starter.yaml as an Agent Observability experiment and publishes scores.
You still need to: (1) fill run_agent(case) to call YOUR agent (first hole); (2) fill the model
call inside judge_<evaluator> using the agent's own provider client, then tune it (second hole);
(3) set real credentials — `AGENTO11Y_ENDPOINT` and `AGENTO11Y_AUTH_TOKEN` (your Grafana Cloud
ingestion key) and the agent's provider key. The SDK stores scores; it does not run the agent or
the judge.
Set AGENTO11Y_INGEST_ACTOR to a stable value: the run and its trials must share one actor, or
trial creation fails with "401: experiment is owned by another actor".
AGENTO11Y_ENDPOINT=... AGENTO11Y_AUTH_TOKEN=... AGENTO11Y_INGEST_ACTOR=ingest:sdk/python \
python evals/run_experiment.py
"""
import json, os, time
from pathlib import Path
from dotenv import load_dotenv
from agento11y import experiments as agento11y
load_dotenv()
SUITE = Path(__file__).parent / "<agent>-starter.yaml"
def run_agent(case: agento11y.TestCase) -> str:
"""FIRST HOLE YOU FILL — call your agent for this case, return its output text."""
raise NotImplementedError("wire this to your agent entrypoint (see Step 1 refs)")
def judge_<evaluator>(case_input, output) -> tuple[float, bool, str]:
"""Sketched llm_judge — a model call returning (score 0-1, passed, explanation). Tune it.
Uses the SAME provider client the agent uses (imported below from the agent module) — do NOT
swap in litellm or another provider. Adapt the call to your provider's API.
IMPORTANT for grounding/relevance judges: the judge must SEE what the agent saw. If the agent
retrieves context (RAG), re-run its retriever here and put the passages in the prompt —
otherwise the judge marks correct, cited answers as "unverifiable" and scores them low. Import
the agent's own retriever (e.g. `from agent import retrieve`) and include its output.
"""
prompt = (
'Grade <what this evaluator checks>. Reply with ONLY a JSON object, no prose, no markdown '
'fences, explanation LAST and short: {"score": <0-1 float>, "passed": <bool>, "explanation": "<one sentence>"}.\n\n'
# For a grounding judge, prepend the retrieved context here:
# f"CONTEXT:\n{retrieved_passages}\n\n"
f"Input:\n{case_input}\n\nOutput:\n{output}"
)
model = os.getenv("GRADER_MODEL") or os.getenv("MODEL_NAME") # a LIVE model id; no default (dead ids 404)
# SECOND HOLE YOU FILL — call the model using the SAME client the agent uses (do NOT add a new
# provider). Import it from the agent module and make ONE call that returns the reply text into
# `text`. Give it enough max_tokens (~600) that the JSON verdict is never truncated mid-object.
# Anthropic: from agent import client
# text = client.messages.create(model=model, max_tokens=600,
# messages=[{"role": "user", "content": prompt}]).content[0].text
# OpenAI: from agent import client
# text = client.chat.completions.create(model=model, max_tokens=600,
# messages=[{"role": "user", "content": prompt}]).choices[0].message.content
raise NotImplementedError("call the agent's provider client here and assign the reply to `text`")
d = _parse_judge_json(text)
score = max(0.0, min(1.0, float(d.get("score", 0.0))))
return score, bool(d.get("passed", score >= 0.6)), str(d.get("explanation", ""))
def _parse_judge_json(text: str) -> dict:
"""Robustly pull the JSON verdict out of a model reply (handles prose / ```json fences)."""
if not text:
return {}
t = text.strip()
if "```" in t: # strip a ```json ... ``` fence if present
t = t.split("```")[1].removeprefix("json").strip() if t.count("```") >= 2 else t
try:
return json.loads(t)
except json.JSONDecodeError:
pass
# Fallback: scan for the first balanced {...} object rather than first-brace/last-brace.
depth = 0; start = -1
for i, ch in enumerate(t):
if ch == "{":
if depth == 0:
start = i
depth += 1
elif ch == "}":
depth -= 1
if depth == 0 and start >= 0:
try:
return json.loads(t[start:i + 1])
except json.JSONDecodeError:
start = -1
return {} # unparseable → caller treats as score 0; investigate the raw reply
def main() -> None:
suite = agento11y.TestSuite.from_yaml(str(SUITE))
verifier = agento11y.Evaluator(evaluator_id="<evaluator>", version="draft-0", kind="llm_judge")
candidate = {
"agent_name": "<agent>",
# Always send a declared agent_version. Without it Agent Observability auto-derives a version from
# the system-prompt hash, so you can't reliably attribute scores to a version or
# compare versions. Replace "v1" with your real version (git tag, prompt version,
# semver...) — the "v1" fallback is a placeholder, not a version worth comparing.
"agent_version": os.getenv("AGENT_VERSION", "v1"),
"git_sha": os.getenv("GIT_SHA", ""),
"model_name": os.getenv("MODEL_NAME", ""),
}
with agento11y.experiment(name="<agent> starter", experiment_id=f"<agent>-starter-{int(time.time())}",
suite=suite, candidate=candidate, tags=["starter"],
actor=os.getenv("AGENTO11Y_INGEST_ACTOR", "ingest:sdk/python")) as exp:
for case in suite.test_cases:
with exp.trial(case) as trial:
out = run_agent(case)
trial.record_io(input=json.dumps(case.input), output=out,
model_provider="<provider>", model_name=os.getenv("MODEL_NAME", ""))
score, passed, why = judge_<evaluator>(case.input, out)
trial.final_score(score, passed=passed, explanation=why, evaluator=verifier)
print(f" {case.test_case_id}: score={score:.2f} passed={passed}")
print(f"\nExperiment: {exp.url}")
if __name__ == "__main__":
main()
Use a fresh experiment_id per run (a timestamp works) — reusing an id created by a
different auth actor fails with 401: experiment is owned by another actor.
Output, in this order:
why (with file:line).
Add a one-line "once you have traffic, these criteria can also run online (Agent Observability Rules or
guard hooks) — separate surfaces, not set up here."evals/<agent>-starter.yaml and
evals/run_experiment.py), and a one-line reminder to review the edge/adversarial cases
and add real ones.run_agent(case) (first hole); fill the judge's
model call using the agent's own provider client, then tune it and add the other recommended
evaluators the same way (second hole); and set credentials (AGENTO11Y_ENDPOINT +
AGENTO11Y_AUTH_TOKEN + the provider key).
State the boundary explicitly: this skill only
bootstraps the first run; for anything past that — binding an already-instrumented agent's
real generations, auditable LLM-judge grading, cross-process verifiers, repeated-sampling
metrics — the agento11y-experiments skill is the reference.Only offer this for an easy agent (clean function seam) in Python or Go (the languages
with an experiments SDK). For in-process/full-stack agents, or agents in a language with no
experiments SDK (TS/Java/.NET), don't offer to run — point to the existing harness/infra (or
the subprocess-bridge option) and stop; a real run there is out of scope for this skill.
After the summary, offer to run the starter experiment for them — do not run automatically. Ask: "Want me to try running this now?" Only proceed if they say yes.
If they accept:
run_agent(case) — wire it to the real entrypoint from Step 1, so the runner
actually calls their agent.https://<your-stack>.grafana.net/plugins/grafana-agento11y-app:
AGENTO11Y_ENDPOINT = the API URL on that page. If unset, ask — never invent one.AGENTO11Y_AUTH_TENANT_ID = the Instance ID on that page.AGENTO11Y_AUTH_TOKEN — always required (the SDK raises before any request if it is empty).
The developer creates it via "Create a token in Cloud Access Policies" on the Connection page. Tell them
the exact scopes: sigil:write, metrics:write, traces:write, logs:write. Heads-up on
the UI: sigil is not in the default resource list — they must add it via "Add scope"
(then tick Write); metrics/traces/logs are already listed (tick Write). The scope is still
sigil:* (the Cloud resource keeps the old name). Never mint or fabricate a token — the
developer creates and supplies it; you only read it from the environment or ask.OPENAI_API_KEY or ANTHROPIC_API_KEY) — name the
exact env var, since both the agent and the judge need it. Ask for it; never mint it.MODEL_NAME is a live model (dead model ids fail with a 404 not_found).AGENTO11Y_INGEST_ACTOR so run and trials share one actor (else 401: owned by another actor).AGENT_VERSION (in the candidate). Without it Agent Observability auto-derives a version
from the system-prompt hash, and the developer can't attribute scores to a version or
compare versions — which is the whole point of the agent's Quality view. Confirm a real
value (git tag / prompt version / semver), don't leave it defaulted.exp.url, and note this published one experiment's
scores (no tenant evaluators/rules/guards were created).llm_judge forgives — e.g. an exact-string check vs a paraphrase), and (b) adding
a case that actually stresses the agent's likely real failure mode.If they decline, stop after the summary.