# braintrust-define-eval-release-gate

Builds ship or hold rules from eval metrics for an LLM app, and turns them into a CI check that explains failures.

Skill by Braintrust. Category: Testing.

## Install

Project (for this project only):

```sh
npx skills add braintrustdata/eval-library --skill braintrust-define-eval-release-gate
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

Global (for every project on this machine):

```sh
npx skills add braintrustdata/eval-library --skill braintrust-define-eval-release-gate -g
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

## Tags

- Runs on your computer: It runs as a program on your machine, with your access.

## Security audits

- Gen Agent Trust Hub: Pass
- Socket: Pass
- Snyk: Warn

Installs: 11. Installs counted by skills.sh. [Open on skills.sh](https://www.skills.sh/braintrustdata/eval-library/braintrust-define-eval-release-gate)

## Source

[Open the source](https://github.com/braintrustdata/eval-library/tree/HEAD/skills/braintrust-define-eval-release-gate). Checked 2026-10-10. This summary may contain AI-generated text, so it can be wrong. Check the source before you rely on it.

## SKILL.md

````markdown
---
name: braintrust-define-eval-release-gate
description: >-
  Create, edit, audit, or apply release gates for LLM applications and agents. Use to combine
  minimum meaningful improvement, statistical significance, regression rate, subgroup consistency,
  worst-run stability, all-attempts reliability, safety upper bounds, latency, and cost into an
  explicit ship-or-hold policy, to turn metrics into a CI gate, or to explain why a candidate
  failed a gate and what evidence would justify reconsideration. Do not use for general result
  analysis without a deployment decision.
---

# Define or apply a release gate

Contract: `references/interaction-contract.md`. Calibration, templates, provenance: `references/gate-rows.md`.

## Trigger

- "Create release gates." / "Should this ship?" / "Why did this candidate fail?"
- A ship decision being made from a single average.
- A gate that exists but has no owner and blocks nothing.

## Do

1. Identify the shipping claim and the failures that cannot be traded away. Draft the table before
   pinning thresholds needing product-owner input — mark those `Needs decision`.
2. Build a row per property the claim implicitly makes, using the row set in `reference.md`:
   magnitude, significance, consistency, stability, reliability, safety, latency/cost. A gate
   should test **every** property, not just the average.
3. Keep improvement metrics and guardrails structurally separate: guardrails are constraints,
   never terms in a weighted sum.
4. Require the run-integrity precondition — effective N and error accounting per arm. A gate on a
   run with unexplained missing items is a gate-shaped artifact of whichever items survived.
5. Enter red-team findings as **existence constraints**, never averaged scores, re-tested every
   round rather than retired once passed.
6. Emit both forms: machine-readable rules for CI, and a human-readable rationale per row. On
   failure, name the failed rows, what each failure **means**, and the evidence that would justify
   reconsideration.

## Avoid

- Do not let an average improvement buy down a safety or reliability constraint.
- Do not gate on point estimates for near-zero rates; size n to the tolerance.
- Do not gate on a scorer not validated for gating, or on the set the candidate was tuned against.
- Do not write thresholds nobody owns; an unowned gate becomes a dashboard.
- Do not compute the statistics here.

## Check

- Every row has a rule, a statistical basis, and a threshold with a stated source and owner.
- Guardrails expressed as constraints; reliability and safety rows use bounds and worst cases.
- Effective-N precondition explicit.
- Machine-readable rules match the human-readable rationale exactly.
- Failure interpretation written per row — each failure means something different.

## Risk

- Thresholds without uncertainty, or without operational ownership, turn gates into decoration.
- A gate that passes says nothing about inputs the set does not cover; completeness is
  unreachable, so gates pair with monitoring rather than replacing it.
- Gates tuned until the current candidate passes are not gates.

## Braintrust

**Encode each row as a scorer-plus-threshold check on the confirmatory experiment and block
promotion on any failing row.** A gate producing a chart instead of a block is not a gate. Reuse
scorers by name and version (`references/platform-mechanics.md` §5); a gate wired to a
differently-named scorer silently stops testing what it was written for. Check each scorer's
fitness statement first — where a scorer is prohibited from gating, **do not attach it at all**,
since absence beats a warning nobody reads.

Rows needing a derived value rather than a native score: the paired difference with clustered CI;
per-category win rate and regression rate; the leave-one-out recomputation; `pass^k` across the K
runs; the Wilson or rule-of-three **upper bound** for safety; p95 per run then the **worst** run.
§8 lists the wiring failures these invite — the last two attract them most.

Gate against the experiment marked **confirmatory**, pinned to the held-out dataset version, on a
cached pull.
````

- Previous: [braintrust-analyze-eval-experiment](https://syn4pse.dev/terminal/braintrust-analyze-eval-experiment.md)
- Next: [fusion-to-publish](https://syn4pse.dev/terminal/builderio-fusion-to-publish.md)
- [Terminal](https://syn4pse.dev/terminal.md)
- [Suggest a change](mailto:terminal@syn4pse.dev?subject=Terminal%20suggestion)
