# braintrust-design-eval-experiment

Plans a controlled comparison of model, prompt or tool changes before running it, with hypotheses, paired designs and stopping rules fixed in advance.

Skill by Braintrust. Category: Testing.

## Install

Project (for this project only):

```sh
npx skills add braintrustdata/eval-library --skill braintrust-design-eval-experiment
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

Global (for every project on this machine):

```sh
npx skills add braintrustdata/eval-library --skill braintrust-design-eval-experiment -g
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

## Tags

- Runs on your computer: It runs as a program on your machine, with your access.

## Security audits

- Gen Agent Trust Hub: Pass
- Socket: Pass
- Snyk: Pass

Installs: 10. Installs counted by skills.sh. [Open on skills.sh](https://www.skills.sh/braintrustdata/eval-library/braintrust-design-eval-experiment)

## Source

[Open the source](https://github.com/braintrustdata/eval-library/tree/HEAD/skills/braintrust-design-eval-experiment). Checked 2026-10-10. This summary may contain AI-generated text, so it can be wrong. Check the source before you rely on it.

## SKILL.md

````markdown
---
name: braintrust-design-eval-experiment
description: >-
  Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or
  agent-architecture changes. Use before data collection to state directional and minimum-effect
  hypotheses, name independent, dependent, and control variables including the serving
  environment and tool surface, choose paired designs, set repetitions and allocation,
  distinguish exploratory from confirmatory comparisons, and pre-specify stopping, exclusion,
  multiplicity, and analysis rules. Do not use primarily to analyze results already collected.
---

# Design the experiment

Contract: `references/interaction-contract.md`. Calibration, templates, provenance: `references/experiment-manifest.md`.

## Trigger

- A planned comparison: prompt A vs. B, M0 → M1, retrieval on/off, a new guardrail.
- "Write a hypothesis." / "Create a pre-analysis plan." / "Name the stopping rule."
- A sweep about to run with no plan for how the winner gets confirmed.

## Do

1. Convert the change into **one tradeoff-aware hypothesis**: directional, minimum effect,
   naming what may not regress, plus population, proxies, and scorers. This **is** the release
   gate stated in advance.
2. Name independent, dependent, and control variables (`reference.md`). Any control that matters
   must be a schema field, or it cannot be held constant or sliced by.
3. Pin what hides inside apparently atomic variables: the **serving path** behind a model
   string, the **tool manifest** behind "the agent." If more than one thing changes per arm, the
   design cannot attribute the effect to any single factor — isolate first.
4. Choose **pairing** (same items across arms — the default, far more sensitive) and K = 3–5 runs
   for anything gating.
5. **Verify the treatment is implementable in every arm.** Publish the per-arm implementation
   table, and exclude arms that cannot receive it from the treatment-effect claim rather than
   recording them as "no benefit."
6. Pre-specify stopping, exclusions, exploratory vs. confirmatory, multiplicity, and estimators —
   the elicitation regime dictates the estimator.

## Avoid

- Do not choose the analysis after seeing outcomes, or relabel a sweep as confirmatory.
- Do not fix a defect mid-comparison. Restart under the corrected config, or finish as designed
  and record the defect — a mid-run correction leaves no trace in the results table.
- Do not run unbounded parallelism; it is the most productive source of fake failures.
- Do not analyze results here.

## Check

- One hypothesis with direction, minimum effect, and guardrail bounds.
- All three variable classes enumerated; every control exists as a field.
- Serving environment and tool manifest pinned per arm.
- Pairing and K stated; stopping rules fixed; exploratory vs. confirmatory labeled in advance.
- Per-arm treatment implementation table drafted; manifest reproducible.

## Risk

- Flexible stopping, undocumented arms, and uncontrolled changes make an apparent improvement
  indistinguishable from selection noise.
- A treatment with no uniform implementation quietly becomes several treatments.
- Metered ceilings fail in the worst available way — partway through one arm, leaving a partial
  result that resembles data.
- Confounds enumerated on a whiteboard stay there unless they become fields.

## Braintrust

**Write H1 into the experiment description** so the pre-registered claim travels with the
results — the cheapest integrity control available. Mark the confirmatory run explicitly in its
**name and description**.

This is the stage that has to get the shared mechanics right, because they are decided here and
cannot be repaired afterwards. From `references/platform-mechanics.md`: **§3 pinning**, since
pairing must be chosen before running and cannot be applied retroactively; **§4 metadata** for
every independent variable, serving path and tool manifest included, and every control named in
step 2; **§5 naming**, or the arms never line up; **§6 separate sweep experiments**, losers kept,
so the denominator survives to the report; **§7 hygiene** for concurrency and smoke-test
deletion.

Each of those is cheap now and impossible later. That asymmetry is the whole argument for
designing the experiment before running it.
````

- Previous: [fusion-to-publish](https://syn4pse.dev/terminal/builderio-fusion-to-publish.md)
- Next: [braintrust-design-eval-instrumentation](https://syn4pse.dev/terminal/braintrust-design-eval-instrumentation.md)
- [Terminal](https://syn4pse.dev/terminal.md)
- [Suggest a change](mailto:terminal@syn4pse.dev?subject=Terminal%20suggestion)
