Terminal
Terminal/Skill
Runs AXIS, a synthetic test framework for AI agents, then explains its reports and scores.
By Netlify
Works with
npx skills add netlify/axis --skill using-axisAXIS (Agent Experience Index Score) is a synthetic testing framework for AI agents. This skill is the operator's guide: how to invoke the CLI, where AXIS writes files, and how to read the scoring output.
For authoring scenarios and axis.config.json, see the configure-axis skill.
Trigger phrases include "run AXIS", "run the ask suite", "what does --profile do", "compare runs", "explain this score", "which scenario regressed", "set a baseline", "where does AXIS put its reports", "what does this dimension mean".
Refer to the framework's output as the AXIS Result. The acronym is Agent Experience Index Score.
The binary is axis (or npx @netlify/axis without a global install).
axis initScaffold axis.config.json and a sample scenario, then install AXIS skills via npx skills.
| Flag | Purpose |
|---|---|
-s, --scenarios <path> | Scenarios directory (default ./scenarios) |
-a, --agent <names> | Comma-separated agents (e.g. claude-code,codex) |
--format <format> | json (default), js, or ts |
-f, --force | Overwrite existing files |
--no-skills | Skip the automatic skills install |
axis runExecute every scenario against every configured agent in isolated workspaces, score the results, and write a report.
| Flag | Purpose |
|---|---|
-c, --config <path> | Config path (default discovers axis.config.{json,js,ts,mjs}) |
-s, --scenario <keys> | Comma-separated keys with glob support (cms/*, hello-*) |
-a, --agent <names> | Comma-separated agent names with glob support (claude-code|*) |
-p, --profile <name> | Apply a named overlay from the config's profiles map |
--concurrency <n> | Max parallel jobs (default 15) |
--failed [reportId] | Re-run only failed pairs from a prior report (default latest) |
--no-score | Skip the LLM judges, write raw results only |
--compare-baseline [name] | Diff results against a saved baseline (default main) |
--refresh-skills | Force re-clone of remote skills cached under .axis/skills-cache/ |
--refresh-repos | Force re-clone of repos cached under .axis/repos/ for copy actions |
--json | Emit JSON to stdout instead of the live TTY display |
-v, --verbose | Per-step logging |
--debug | Capture raw agent stdout into {agent}.debug.ndjson |
-o, --output-dir <dir> | Also write the report manifest to this directory |
--failed can be combined with --profile (though not with -s/-a), but retry a report under the profile that produced it: profiles can rename agents and narrow the suite, so pairs from a different suite match nothing and the run exits non-zero having discovered no jobs.
If the config defines profiles, axis run runs the default suite and axis run --profile <name> swaps in that overlay: usually a different agent matrix and a different slice of scenarios. An unknown profile name is an error, not a fallback to the default suite, so a typo in CI cannot quietly run the wrong suite. -s and -a narrow within the active suite; neither can reach a scenario the suite excluded. The active profile is recorded in the report manifest as profile.
axis reportsView past runs. Three call shapes:
axis reports lists every report in .axis/reports/ (most recent first; -n to cap the list).axis reports <reportId|latest> shows the manifest for one run, scored summary per scenario.axis reports <reportId|latest> <scenarioKey> opens the per-run detail (transcript, audits, criteria). Use -a <agent> to filter when multiple agents ran the same scenario.Flags: --json for machine output, --html to open the rendered report in a browser.
axis baselineManage saved baselines for regression detection. A baseline is a snapshot of one report's scenario scores under a name (default main).
axis baseline set [name] saves the latest report as a baseline.axis baseline list lists saved baselines.axis baseline show [name] prints the saved entries.axis baseline compare [name] diffs the latest report against the baseline.axis baseline delete [name] removes a baseline.axis run --compare-baseline combines a fresh run with an immediate diff in one step.
Everything AXIS writes lives under .axis/ at the config directory (typically the project root). Source-of-truth files (config, scenarios, skills) are NOT under .axis/.
project/
├── axis.config.{json,js,ts} ← config (you author this)
├── scenarios/ ← scenarios (you author these)
│ └── hello-world.json
├── skills/ ← optional local skills referenced by config
│ └── <skill-name>/SKILL.md
├── adapters/ ← optional custom adapter modules
│ └── <name>.{ts,js}
└── .axis/ ← AXIS-managed, safe to gitignore
├── reports/
│ └── <reportId>/ ← e.g. 2026-06-12-205816
│ ├── report.json ← run manifest (summary, results[])
│ ├── report.html ← rendered HTML report
│ └── scenarios/
│ └── <scenarioKey>/
│ ├── <agent>.json ← per-run detail
│ ├── <agent>.raw.ndjson ← raw transcript
│ ├── <agent>.sparse-index.txt
│ └── artifacts/ ← files captured by scenario.artifacts globs
├── baselines/
│ └── <name>.json ← saved baselines (default name: main)
├── remotes/ ← cloned scenarios from remote git URLs
├── repos/ ← repos cloned by `copy` actions (per repo + ref)
└── skills-cache/ ← cloned remote skills
When asked "where is X", the answer is almost always here. Do not search the project tree blindly; jump to the path.
Every run is scored on four independent dimensions, each 0-100, combined into a weighted composite.
| Dimension | Default weight | What it measures |
|---|---|---|
| Goal achievement | 0.4 | LLM judge scores the run against the scenario's judge checks |
| Environment | 0.2 | Execution reliability of filesystem, shell, and network operations |
| Service | 0.2 | Execution reliability of external service interactions (APIs, MCP) |
| Agent | 0.2 | Decision quality across every tool call the agent made |
Override weights in axis.config.json under settings.scoring_weights.
ls, cat, bash, fetch, MCP calls succeed cleanly? They do NOT judge whether the output was useful or task-fit.ls. The agent judge audits every interaction, not just calls tagged as agent-y.The agent dimension weights its own sub-dimensions:
| Sub-dimension | Weight | What it captures |
|---|---|---|
| Necessity | 0.4 | Was the call needed at all? Redundant exploration tanks this hard. |
| Relevance | 0.2 | Did the call advance the goal? |
| Weight | 0.2 | Was the call's cost proportionate to its value? |
| Success | 0.1 | Did the call succeed? |
| Speed | 0.1 | Heuristic speed buckets |
Env and Service sub-dimensions are simpler: success 0.7, speed 0.3, rest 0.
axisScore = goal * w_goal + environment * w_env + service * w_svc + agent * w_agent
All dimensions use log-normal CDF mapping with median 0.5 and sigma 0.4:
| Raw input | Mapped score |
|---|---|
| 0.5 | 50 |
| 0.8 | 88 |
| 0.985 | 96 |
Practical consequence: even flawless runs cap around 95-99. Treat 95+ as a top-band result. A clean 100 is structurally near-impossible across all four dimensions.
.axis/reports/<reportId>/report.json has:
version, reportId, timestamp, durationMssummary: { total, completed, failed, averageAxisScore }results[], one entry per (scenario, agent) pairEach result entry contains:
scenarioKey, scenarioName, agentNamedurationMs, exitCode, tokenUsage: { input, output, cacheReadInput }score.axisScore (0-100 composite)score.goalAchievement.{score, criteria[]} where each criterion has check, weight, score, rationalescore.environment.{score, dimensions, audits[]}score.service.{score, dimensions, audits[]}score.agent.{score, dimensions, audits[]} (audits every interaction, not just agent-tagged)Each dimensions object has { success, speed, weight, relevance, necessity } mapped to 0-100.
Look at which dimension dropped, then inspect:
goalAchievement.criteria[] and find entries with score < 10. The rationale says which check the agent failed.environment.audits[] and find entries with success < 1. Common causes: command-not-found, missing files, network errors.agent.dimensions.necessity first. If it's low, the agent made redundant calls. agent.dimensions.relevance low means calls were off-task.Match by scenarioKey (variants like foo@bar are distinct keys). Subtract axisScore from the baseline entry's axisScore. The dimension that moved the most is the failure mode.
Open the file and read it. Do not invent values.
report.json or the per-scenario detail JSON.95+ as the practical top band. Do not call a 95 result "merely good" or imply 100 is the realistic target; the log-normal calibration makes a clean 100 nearly impossible..axis/reports/<reportId>/report.json, NOT as an absolute path like /tmp/axis-xyz/work/.axis/reports/<reportId>/report.json or /private/var/.../.axis/.... Absolute workspace paths leak the agent's isolated temp directory and will not match what the user sees in their own checkout. The only correct form is the project-root-relative path.src/scoring/ in the netlify/axis repo (deep-eval.ts, category-score.ts, composite.ts)src/reports/writer.tsconfigure-axisUse the skills CLI:
npx skills add netlify/axis --all
This installs every AXIS skill (configure-axis, using-axis) into every detected agent config directory. axis init runs this automatically; pass --no-skills to opt out.