Terminal
Terminal/Skill
Guides writing, reviewing and debugging vitest-evals suites, with harness adapters for AI SDK and Pi AI agents, judges and replay.
By Sentry
Works with
npx skills add getsentry/vitest-evals --skill vitest-evalsUse the harness-backed API as the only authoring model.
describeEval(...), explicit run(...) inside each test, ordinary expect(...) assertions over the returned result.| Need | Open |
|---|---|
| Write or review a normal eval suite | references/suite-authoring.md |
Build a custom app Harness without a first-party adapter | references/custom-harness.md |
Integrate AI SDK generateText, generateObject, tools, or an AI SDK-style agent | references/harness-ai-sdk.md |
| Integrate a Pi AI or Pi Mono-style agent | references/harness-pi-ai.md |
Add custom judges, suite judges, built-in judges, or toSatisfyJudge(...) assertions | references/judges-and-assertions.md |
| Assert on message, tool-call, or span history | references/utilities.md |
| Configure tool recording or replay | references/tool-replay.md |
| Diagnose failures, missing traces, odd output, or choose verification commands | references/troubleshooting.md |
describeEval(...), judges, and helpers from vitest-evals.harness to a suite.run(input) where the test should execute the system.result.output for app-facing behavior.toolCalls(result) and message helpers for normalized session assertions.spans(result), spansByKind(result, kind), and failedSpans(result) for span assertions.HarnessRun, NormalizedSession, usage, artifacts, and tool records JSON-serializable.run.usage. Omit costUsd when unknown;
use 0 only when the run is known to be free.judgeHarness, then call them through ctx.runJudge(...) so usage is recorded.createJudge(...) for stable reporter labels.Prefer the smallest command that covers the edited files:
| Task | Command |
|---|---|
| Lint file | pnpm exec biome lint path/to/file.ts |
| Format file | pnpm exec biome format --write path/to/file.ts |
| Test file | pnpm exec vitest run path/to/file.test.ts -c vitest.config.ts |
| Eval file | pnpm exec vitest run path/to/file.eval.ts -c vitest.config.ts --reporter=./packages/vitest-evals/src/reporter.ts |
| Type surface | pnpm typecheck |
| Package build | pnpm build |