# braintrust-deploy-evaluator

Takes a tested scorer live on Braintrust production logs, with sampling, an online scoring rule, and optional backfill of past traces.

Skill by Braintrust. Category: Testing.

## Install

Project (for this project only):

```sh
npx skills add braintrustdata/eval-library --skill braintrust-deploy-evaluator
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

Global (for every project on this machine):

```sh
npx skills add braintrustdata/eval-library --skill braintrust-deploy-evaluator -g
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

## Tags

- Runs on your computer: It runs as a program on your machine, with your access.

## Security audits

- Gen Agent Trust Hub: Pass
- Socket: Pass
- Snyk: Warn

Installs: 9. Installs counted by skills.sh. [Open on skills.sh](https://www.skills.sh/braintrustdata/eval-library/braintrust-deploy-evaluator)

## Source

[Open the source](https://github.com/braintrustdata/eval-library/tree/HEAD/skills/braintrust-deploy-evaluator). Checked 2026-10-10. This summary may contain AI-generated text, so it can be wrong. Check the source before you rely on it.

## SKILL.md

````markdown
---
name: braintrust-deploy-evaluator
description: >-
  Take a validated scorer or classifier from definition to running instrument in Braintrust —
  scope selection, inline testing before saving, saving as an evaluator, attaching an
  online-scoring rule, activating it for new traffic, and backfilling history with a rewind. Use
  when a scorer needs to actually run against production logs, when an online-scoring rule needs
  to be created or changed, or when historical traces need scoring. Do not use to decide what the
  scorer should measure, to write its rubric, or to establish that it agrees with human judgment.
---

# Deploy an evaluator to production traffic

Contract: `references/interaction-contract.md`. Calibration, templates, provenance: `references/evaluator-deployment.md`.

## Trigger

- "Run this scorer against our logs." / "Score the last two weeks."
- A validated scorer that still only exists inside an offline experiment.
- An online-scoring rule to create, retarget, resample, or pause.
- Backfilling a newly deployed evaluator over history.

## Do

1. Confirm the evaluator has a **fitness verdict** before deploying it. Trend monitoring and
   release gating are different approvals; a scorer cleared for one is not cleared for the other.
   No verdict yet → `braintrust-validate-eval-scorer` first.
2. Fix **scope** — span for a single LLM call or tool call, trace for behavior spanning a request,
   group for behavior spanning several traces joined by a key. Default to trace. Confirm the
   grouping key exists in real metadata rather than assuming it.
3. Fix the **output contract**: a scorer returns 0–1 and needs its numeric mapping; a classifier
   returns one of a fixed label set and needs its no-match behavior. Declare skip behavior for
   both.
4. **Test inline before saving.** Iterate on the definition against representative examples with
   no saved artifact, then save, then re-test the *saved* version on the same examples — saving is
   a step that can change behavior, and the second test is what catches it.
5. Treat **activation** and **rewind** as two separate authorizations. Neither implies the other.
   A paused rule will not process a rewound range until it is activated; say so rather than
   letting the user believe the backfill is running.
6. Before rewinding, **estimate the eligible target count** over the exact rule configuration and
   window, and account for the sampling rate in the number you report.
7. After results land, read the distribution and inspect representative passes, failures, and
   borderline cases before drawing any conclusion from the aggregate.

## Avoid

- Do not deploy an evaluator whose approved uses are undocumented — absence of a verdict is not
  approval for trend monitoring.
- Do not set 100% sampling without explicit approval; sampling rate is the primary cost control.
- Do not resubmit a rewind to check on it. Resubmission is not a progress query.
- Do not rename an evaluator across deployments (`references/platform-mechanics.md` §5) — the
  name is the join key for every cross-experiment and offline-to-online comparison.
- Do not report a preliminary backfill aggregate without saying it is preliminary.

## Check

- Fitness verdict exists and names this deployment's use as allowed.
- Scope justified; grouping key verified present in real data.
- Output contract declared, including skip and no-match behavior.
- Inline test passed, then the saved version re-tested on the same examples.
- Sampling rate and filters approved; rule status intentional, not defaulted into.
- Rewind window and estimated target count stated before the rewind, not after.

## Risk

- An evaluator deployed without a verdict produces an authoritative-looking series that no one can
  say is valid, and it accumulates history that later looks like a baseline.
- Activation and rewind get conflated constantly. The usual outcome is a user who believes history
  is being scored while a paused rule does nothing.
- Backfill cost scales with eligible units × sampling and is easy to underestimate by an order of
  magnitude on a high-volume project.
- Judge drift under a provider model update is indistinguishable from a product regression in the
  resulting series unless the evaluator version is recorded alongside it.

## Braintrust

**Build and test.** `sql_query` first to find representative data and locate where the behavior
actually appears — `input`, `output`, `metadata`, or elsewhere. Code evaluators (Python or
TypeScript) for anything checkable without a model; LLM evaluators where judgment is required.
Follow the project's existing language. Test with `test_evaluator` and
`function: {inline_evaluator: <definition>}` — the same definition `create_evaluator` accepts, no
save required. Inline code must define a top-level function named `handler`.

**Save.** `create_evaluator` with `output_type: "score"` and `choice_scores` for numeric output, or
`output_type: "classification"` and `choices` for labels. Code evaluators declare `output_type` but
use neither. Then re-run `test_evaluator` with `function: {function_id: "<saved id>"}` on the same
examples to confirm the save round-tripped.

**Rule.** `update_online_scoring_rule` with `operation: "save"`, the evaluator `function_ids`,
scope, filters, sampling rate, and logging behavior. New rules default to `status: "paused"`; pass
`status: "active"` only on explicit authorization. When updating an existing rule, **omit `status`**
to preserve its lifecycle state unless a change was requested. Activate separately via
`set_automation_status`.

**Rewind.** Estimate first with `sql_query` — count spans for span scope, distinct traces for trace
scope, distinct grouping values for group scope — then apply the sampling rate. Call
`update_online_scoring_rule` with `operation: "rewind"`, the `automation_id`, and an inclusive
`start_time`, **once**. For a natural-language window, resolve the timestamp yourself and state the
plan ("the last three full days, roughly 1,400 spans") rather than making the user confirm arithmetic.

**Read the results.** Distribution first, then representative cases at both ends and the boundary.
`generate_monitor_chart` for a histogram of score distribution or a time series of trend; group by a
category only where it improves interpretation. Keep previews unsaved unless asked to persist.
Preprocessors and the `{{preprocessed}}` variable for trace- and group-scoped LLM evaluators are
covered in `braintrust-discover-trace-topics`.
````

- Previous: [braintrust-define-eval-objective](https://syn4pse.dev/terminal/braintrust-define-eval-objective.md)
- Next: [braintrust-design-eval-metric-bundle](https://syn4pse.dev/terminal/braintrust-design-eval-metric-bundle.md)
- [Terminal](https://syn4pse.dev/terminal.md)
- [Suggest a change](mailto:terminal@syn4pse.dev?subject=Terminal%20suggestion)
