# braintrust-elicit-eval-criteria

Captures what good looks like from domain experts and real user signals as rubric anchors, before any labeling or scoring starts.

Skill by Braintrust. Category: Testing.

## Install

Project (for this project only):

```sh
npx skills add braintrustdata/eval-library --skill braintrust-elicit-eval-criteria
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

Global (for every project on this machine):

```sh
npx skills add braintrustdata/eval-library --skill braintrust-elicit-eval-criteria -g
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

## Tags

- Runs on your computer: It runs as a program on your machine, with your access.

## Security audits

- Gen Agent Trust Hub: Pass
- Socket: Pass
- Snyk: Warn

Installs: 8. Installs counted by skills.sh. [Open on skills.sh](https://www.skills.sh/braintrustdata/eval-library/braintrust-elicit-eval-criteria)

## Source

[Open the source](https://github.com/braintrustdata/eval-library/tree/HEAD/skills/braintrust-elicit-eval-criteria). Checked 2026-10-10. This summary may contain AI-generated text, so it can be wrong. Check the source before you rely on it.

## SKILL.md

````markdown
---
name: braintrust-elicit-eval-criteria
description: >-
  Extract evaluation criteria out of domain experts and real user desires, and capture them as
  reusable evaluation assets before any labeling or scoring begins — construct facets, anchored
  exemplars, adversarial traps, scoring guidance, audit rules, and the signals that reveal what
  users actually want. Use when nobody can say what "good" means, when a rubric does not exist
  yet, when expert knowledge lives only in reviewers' heads, or when validating that an eval
  reflects user desires rather than team assumptions. Do not use to run the labeling workflow or
  to compare a scorer against finished labels.
---

# Elicit criteria from experts and users

Contract: `references/interaction-contract.md`. Calibration, templates, provenance: `references/elicitation-guide.md`.

## Trigger

- No rubric exists, and review is happening on instinct.
- Experts disagree about what "good" means, or agree without being able to say why.
- An eval built entirely from team assumptions about what users want.

## Do

1. Gather both sources. **Expert knowledge**: what a qualified reviewer knows that the rubric
   does not say. **User desire**: what users actually want, evidenced rather than assumed.
2. Elicit from cases, not abstractions. Put a real trace in front of the expert, ask for a
   judgment, then ask *why* — the reason is the criterion. Repeat until reasons stop being new.
3. Capture each facet as an **anchored exemplar pair**: one output earning the top score, one
   that just misses, and the distinguishing reason. A pair beats a paragraph of description.
4. Harvest user desire from evidence that already exists — escalations, reopened tickets,
   abandoned sessions, thumbs-down, support themes. Where none exists, say so and label the
   criteria `Assumed`.
5. Capture what experts know about **failure**: the traps, the plausible-looking wrong answer,
   what juniors get wrong. Highest-value output, and the part rubrics usually omit.

## Avoid

- Do not ask experts to write a rubric from a blank page; elicit from cases.
- Do not substitute team intuition for user desire, or present it as evidence when you do.
- Do not run the labeling workflow or implement scoring here; this stage produces the criteria
  those steps consume.
- Do not resolve genuine expert disagreement by averaging it — a contested criterion is a
  finding about the construct.

## Check

- Every facet has an anchored exemplar pair with the distinguishing reason.
- Failure knowledge captured explicitly, not just success criteria.
- Each criterion labeled `Confirmed` (evidenced) or `Assumed` (team belief).
- User-desire signals named with their source, or their absence stated.

## Risk

- Experts articulate what they can defend, not always what they use; the reason given may not be
  the reason applied, which is why exemplars beat prose.
- A single expert encodes one house style as a universal standard.
- Available evidence of desire is biased toward users who complain; silent dissatisfaction leaves
  no trace.
- Criteria elicited once ossify — they need the dataset's refresh cadence.

## Braintrust

Run the elicitation session itself as **human review** over real traces with **per-criterion
scoring and free-text notes** — the notes are the output, because they carry the *reason*.
Promote reviewed traces into a **versioned golden dataset** so exemplars travel with the
criteria instead of living in a doc. Each facet then becomes its own scorer with a consistent
name; anchored pairs become the few-shot content of a rubric scorer. Flag traps in `metadata`
(`trap: true`) so they can be sliced out of headline numbers and reported separately. Treat a
criteria revision like a scorer change: bump the version, and do not compare results across the
boundary.
````

- Previous: [braintrust-validate-eval-scorer](https://syn4pse.dev/terminal/braintrust-validate-eval-scorer.md)
- Next: [braintrust-eval](https://syn4pse.dev/terminal/braintrust-eval.md)
- [Terminal](https://syn4pse.dev/terminal.md)
- [Suggest a change](mailto:terminal@syn4pse.dev?subject=Terminal%20suggestion)
