# braintrust-define-eval-objective

Works back from a product decision to what an eval should measure, and which claims it can and cannot support.

Skill by Braintrust. Category: Testing.

## Install

Project (for this project only):

```sh
npx skills add braintrustdata/eval-library --skill braintrust-define-eval-objective
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

Global (for every project on this machine):

```sh
npx skills add braintrustdata/eval-library --skill braintrust-define-eval-objective -g
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

## Tags

- Runs on your computer: It runs as a program on your machine, with your access.

## Security audits

- Gen Agent Trust Hub: Pass
- Socket: Pass
- Snyk: Pass

Installs: 9. Installs counted by skills.sh. [Open on skills.sh](https://www.skills.sh/braintrustdata/eval-library/braintrust-define-eval-objective)

## Source

[Open the source](https://github.com/braintrustdata/eval-library/tree/HEAD/skills/braintrust-define-eval-objective). Checked 2026-10-10. This summary may contain AI-generated text, so it can be wrong. Check the source before you rely on it.

## SKILL.md

````markdown
---
name: braintrust-define-eval-objective
description: >-
  Create, edit, or audit an eval objective by working backward from a product decision to the
  target outcome, construct, population, intended claim, and verification-versus-validation
  questions. Use when a team is unsure what an eval should establish, asks "what are we
  actually trying to measure," "is this eval measuring the right thing," "does this benchmark
  support our claim," or needs to turn a product goal into an eval objective and state which
  claims are out of scope. Do not use to select detailed metrics, design datasets, or implement
  scorers.
---

# Define the eval objective

Contract: `references/interaction-contract.md`. Calibration, templates, provenance: `references/objective-card.md`.

## Trigger

- Nobody can name the decision the eval supports.
- A benchmark score being used to argue a claim it may not license.
- A product goal ("make the agent more helpful") that needs to become measurable.

## Do

1. State the decision in one sentence. If several are plausible, ask which one the eval must
   support — the highest-information question available here.
2. Write one sentence each: **construct**, **population** (and its weighting), **intended use
   and non-use**.
3. Answer both questions, not just the first. **Verification:** do the scorers hold the spec?
   **Validation:** do these scorers reflect outcomes users actually need? Validation is the
   half teams skip.
4. Where validation has no evidence behind it, say so and elicit criteria from experts and
   real user signals rather than asserting what users want. Label unevidenced desires `Assumed`.
5. Write **supported** and **unsupported** claims as separate lists, then name the evidence
   that would change the decision. An objective no result could overturn is not an objective.

## Avoid

- Do not pick metrics, thresholds, or scoring methods.
- Do not let the population drift toward whatever data is easy to collect; most bad datasets
  fail here, before a single item exists.
- Do not accept a benchmark name in place of a product property.
- Do not imply model-level claims when the object under test is a system — harness, tools, and
  context are part of what you measure.

## Check

- Decision, construct, population, use, and non-use each in a sentence.
- Supported and unsupported claims listed separately.
- Both V&V questions answered, with validation's evidence source named.

## Risk

- An ambiguous decision produces an eval that is locally measurable and strategically
  irrelevant.
- A construct defined by convenience gets optimized: the score improves, the product does not.
- Loose population statements ("our users") license claims far broader than the sample
  supports, and nobody notices until the number goes external.

## Braintrust

**Write the decision and intended claim into the experiment description** so the
pre-registered claim travels with its results — a claim in a doc gets reinterpreted after the
fact. Record the population and its weighting in **dataset metadata**. Make non-use
enforceable: name datasets for their population (`prod-weighted-py-backend`, not `eval-set-2`)
so quoting a number out of scope is visibly wrong, and keep benchmark scaffolding in
**separate, clearly named datasets** — once mixed with production-derived items, "measures our
construct" vs. "measures the benchmark's" is unrecoverable. Validation cannot be answered
offline: it needs production scoring sliced by escalation, reopen, and abandonment. If nothing
scores production yet, validation is `Not yet measurable`, and standing up online scoring is the
blocker.
````

- Previous: [braintrust-build-eval-dataset](https://syn4pse.dev/terminal/braintrust-build-eval-dataset.md)
- Next: [braintrust-deploy-evaluator](https://syn4pse.dev/terminal/braintrust-deploy-evaluator.md)
- [Terminal](https://syn4pse.dev/terminal.md)
- [Suggest a change](mailto:terminal@syn4pse.dev?subject=Terminal%20suggestion)
