Terminal
Terminal/Skill
Runs one read only prompt twice, with and without Teamwork Graph context, to compare token use and answer quality.
By Atlassian
Works with
npx skills add atlassian/twg-cli --skill twg-bench-liteRun one read-only prompt twice: once with free Atlassian/local MCP context and
once with paid Atlassian Teamwork Graph context through TWG CLI. This is a
skill-first workflow with a thin twg benchmark lite run bridge. The benchmark
repo owns reusable runner, judge, and report-template code.
Run twg <command>. On shell command not found, use $HOME/.local/bin/twg
(macOS/Linux) / $env:LOCALAPPDATA\Programs\twg\bin\twg.exe (PowerShell), then
tell user to add that directory to PATH. Do not treat auth or command errors as
PATH failures.
V1 supports Codex and Rovo live runs. Claude is V2. Do not require users to install the benchmark CLI locally.
Use a text-only read task both arms can attempt. Do not compare mutations; for writes, compare a read-only plan or dry-run only after user approval. Preserve the underlying prompt byte-for-byte for both arms.
For the default full lifecycle command, verify the portable runner and selected runtime first:
twg benchmark lite --check
The default check is Codex; use --agent rovo when the run should use Rovo.
For Rovo runs:
twg benchmark lite --check --agent rovo
The quality judge defaults to the selected runtime, so Rovo runs use a Rovo judge by default. If the user explicitly wants a different judge runtime, use:
twg benchmark lite --check --agent rovo --judge-agent codex
The managed custom agents are only required for manual orchestration:
twg-bench-lite-controltwg-bench-lite-testIf a manual run needs either agent and it is missing, tell the user to run:
twg skills install --yes
Then start a fresh Codex thread so agent discovery refreshes. Do not substitute generic agents for manual orchestration because arm isolation is the point.
Before a live run, verify setup with:
twg benchmark lite --check
This checks the bundled runner artifact and the selected live-run runtime.
Codex checks include the Codex executable and Codex home. Rovo checks include
the Rovo executable and OAuth status. If --judge-agent differs from --agent,
the check verifies both runtimes. It also reports the global twg-bench-lite
skill install and managed control/test Codex agent configs as manual-workflow
optional checks when relevant.
For the default full lifecycle, run:
twg benchmark lite run --prompt "<read-only prompt>" --output-dir <dir>
Use --agent rovo to run the same lifecycle with Rovo.
The quality judge uses the same runtime unless --judge-agent codex|rovo is
supplied.
For manual orchestration:
twg benchmark lite plan --prompt-file <prompt-file> --json.twg-bench-lite-control and twg-bench-lite-test.Print lifecycle progress in the parent session: start, prompt captured, agents started, each arm finished, outputs read, quality reviewed, report compiled, done.
Read both answers before interpreting token delta. Classify as equivalent,
twg-better, control-better, capability-gain, or not-comparable.
not-comparable is a real outcome for missing, unusable, or non-comparable
answers, not a placeholder. Claim observed savings only for equivalent or
twg-better; for capability-gain, report capability instead.
Evaluate coverage, correctness, evidence quality, directness, and limitations. Do not assign numeric quality scores.
Return a compact report in the parent answer with:
If token/session metrics are unavailable, say "not measured"; do not estimate. Describe the result as one observed prompt comparison, not a universal efficiency rate.