Terminal
Terminal/Skill
Tests whether an agent skill improves results: runs tasks with and without the skill, then grades them against a rubric.
By Pinecone
Works with
npx skills add pinecone-io/cultivar --skill cultivarcultivar is a CLI that measures whether an agent skill actually improves an agent's behavior. For each task it runs the agent with the skill and without it (and optionally with the source docs), then an LLM grader scores each run against a natural-language rubric. Use this skill when the user wants to create, run, or interpret cultivar evals.
cultivar init <skill> — scaffold tasks/<skill>.yaml + a SKILL.md stub.cultivar run -s <skill> -r <runner> --grade — run all variants and grade.cultivar report / cultivar show latest -t <task> — read the outcome.Always confirm the install first with cultivar hello (or cultivar hello --no-grade
when no ANTHROPIC_API_KEY is available) — it runs a packaged smoke task end-to-end.
cultivar init <skill> [--skills-dir DIR] — scaffold task YAML + SKILL.md stub.cultivar run -s <skill> -r <claude|copilot|gemini> — run. Key flags:
-t <task> one task · -v <variant> one variant (see Variants below for the full list)--remote run in isolated Modal sandboxes · -n N repeat · -p N parallelism--grade grade after running · --title NAME label the run · --dry-run print the
prompt + command without calling anything · --timeout S per-call budget (default 90)--model ID pin the agent's model (Claude runner only, e.g. claude-sonnet-5);
unset uses the CLI's own default. A task's extra_tools: [WebSearch, WebFetch]
field unions extra tools onto whichever variant it requests.cultivar grade <run|latest> -s <skill> [--report] — (re)grade an existing run.
--model picks the grading model (any current Claude model works, including the "-5"
generation) · --max-tokens raises the per-reply budget if evidence/reasoning truncate.cultivar report [run] — summary table across runners/variants.cultivar show <run|latest> -t <task> [--grader|--conversation-only|--workdir] — inspect one run.--dry-run is the safe way to preview exactly what will be sent before spending tokens.
Use the /<skill>.ground_truth.context_refs files are prepended.
Only runs for tasks that declare context_refs. Entries can be local files or
http(s):// URLs (fetched live, cached under ./.docs_cache).Read two deltas: with-skill vs without-skill ("does the skill do anything?") and with-skill vs with-docs ("is the distilled skill better than dumping the raw docs?").
Three more real --variant choices, opt-in only, not part of the default "run every
variant" sweep:
ground_truth.self_navigate_refs gives it a starting page. Tests whether the agent can
find the right doc on its own; with-docs tests whether the content is good once handed
over. Different questions, keep them separate.with-docs:<label> — set ground_truth.doc_versions: {label: [refs]} instead of
flat context_refs; --variant with-docs then auto-expands into one real run per
version, graded and reported side by side. Compares two doc versions (e.g. before/after
a rewrite) in one command instead of two separate runs you diff by hand.tasks/<skill>.yaml holds one or more tasks. Each task:
tasks:
- id: a-short-id
intent: "what you'd ask the agent to do"
category: cli # or: code-gen
# setup / teardown / verify: optional shell hooks
# env: ["SOME_KEY"] # required env vars, checked upfront
# extra_tools: [WebSearch, WebFetch] # unioned onto whichever variant is requested
ground_truth:
criteria: |
PASS requires <2-3 concrete, checkable things>.
FAIL if <a common failure mode>.
# context_refs: [docs/ref.md] # activates the with-docs variant
# self_navigate_refs: [docs/overview.md] # activates self-navigate (a starting point, not the answer)
# doc_versions: {v1: [old.md], v2: [new.md]} # activates with-docs:v1 / with-docs:v2 instead of flat with-docs
Guidance:
cultivar tests exactly one skill per run (the -s one). It resolves the skills root
as: --skills-dir flag → CULTIVAR_SKILLS_DIR env → ./.claude/skills. Keep
skills-under-test outside .claude/ (e.g. ./skills, via CULTIVAR_SKILLS_DIR=skills)
if you don't want your interactive coding agent to auto-load them.
--remote — each (task, variant, repeat) runs in its own Modal sandbox: clean
isolation, parallelism, reproducibility. Requires a Modal account (modal token new)
and a secret holding the agent's ANTHROPIC_API_KEY (default secret name
eval-sandbox-secrets; override with CULTIVAR_MODAL_SECRET). Prefer --remote for
rigorous comparisons. The grader always runs locally and needs ANTHROPIC_API_KEY.results/<timestamp>[__title]/ holds per-run .json (stats), .md (readable trace),
.jsonl (raw events), and .workdir/ (files the agent wrote). grades.json holds the
verdicts. Use cultivar report for the table and cultivar show … --grader for the
grader's reasoning + suggestions on a failure.
ANTHROPIC_API_KEY (loaded from a .env in the cwd). hello --no-grade
and run --dry-run need no key.tasks/, examples/, and results/ are cwd-relative and user-owned.-n 3 (or more) for anything you'll act on.