# braintrust-discover-trace-topics

Builds a topics pipeline that clusters trace traffic to show what people use an app for and which issues recur.

Skill by Braintrust. Category: Testing.

## Install

Project (for this project only):

```sh
npx skills add braintrustdata/eval-library --skill braintrust-discover-trace-topics
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

Global (for every project on this machine):

```sh
npx skills add braintrustdata/eval-library --skill braintrust-discover-trace-topics -g
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

## Tags

- Runs on your computer: It runs as a program on your machine, with your access.

## Security audits

- Gen Agent Trust Hub: Pass
- Socket: Pass
- Snyk: Warn

Installs: 9. Installs counted by skills.sh. [Open on skills.sh](https://www.skills.sh/braintrustdata/eval-library/braintrust-discover-trace-topics)

## Source

[Open the source](https://github.com/braintrustdata/eval-library/tree/HEAD/skills/braintrust-discover-trace-topics). Checked 2026-10-10. This summary may contain AI-generated text, so it can be wrong. Check the source before you rely on it.

## SKILL.md

````markdown
---
name: braintrust-discover-trace-topics
description: >-
  Design, evaluate, and deploy a Topics pipeline that discovers clusters in trace traffic —
  preprocessor, facet prompt, no-match policy, clustering, and the automation that runs it. Use
  when the label set is unknown and has to come out of the data: finding what a product is
  actually used for, surfacing recurring issues, or building the categorical vocabulary a
  classifier will later enforce. Also use to evaluate or repair an existing facet or
  preprocessor. Do not use when the label set is already known and stable — that is a classifier
  — or to run a hypothesis-driven or adversarial failure hunt.
---

# Discover topics in trace traffic

Contract: `references/interaction-contract.md`. Calibration, templates, provenance: `references/facet-pipeline.md`.

## Trigger

- "What are people actually using this for?" / "What kinds of issues are in our traffic?"
- No label set exists yet and one has to come out of the data.
- An existing facet producing vague, duplicated, or mostly-skipped output.
- A classifier proposal where nobody can enumerate the classes.

## Do

1. Pick the instrument before building one. Known, stable label set → **classifier**. Label set
   unknown and to be discovered → **facet + clustering**. A facet returns 1–2 sentences of
   unstructured, high-signal text; that looseness is exactly what makes clusters findable, and
   exactly what makes it the wrong tool for enforcement.
2. Fix **scope** first — span, trace, or group — and name the grouping key for group scope.
   Default to trace.
3. Get the **preprocessor** right before touching the facet prompt. It must carry user input,
   model output, tool calls, and errors, and drop system and debug noise. **Run it on real
   traces**; do not reason about what it would return.
4. Write the **output policy** before the prompt: what a good facet output looks like, and what
   counts as no-match. Outputs like "no issues" or "neutral" cluster into nothing — they are
   excluded, not scored.
5. Build ground truth with the **strong model**, then evaluate the **cheap facet model** against
   it. Deterministic scorers for output validity and no-match correctness; a semantic-agreement
   scorer for the rest.
6. Iterate under a budget stated in advance (`reference.md`): one hypothesis, one coherent change
   to preprocessor *or* prompt, rerun the same validation slice. Run the winner **once** on the
   untouched split.
7. Estimate the backfill cost and get explicit approval before rewinding.

## Avoid

- Do not enable an automation before validating the facet on the user's own data.
- Do not use a facet where the classes are already known — you are paying discovery cost for
  enforcement work, and getting an unvalidated instrument.
- Do not return one joined transcript string per span from a preprocessor; deduplication operates
  on atomic array items and a joined string defeats it.
- Do not store preprocessed text in the eval dataset. Store the reference and let the pipeline
  re-derive the text, or you are evaluating a frozen snapshot of a preprocessor you are still
  changing.
- Do not keep tuning against the same slice once the budget is spent.

## Check

- Scope named, grouping key named for group scope.
- Preprocessor **executed** on real traces, not assumed; output carries input, output, tool calls,
  and errors, with no system or debug noise.
- No-match policy defined and its pattern tested against real skip cases.
- Facet validated against reviewed examples before any automation is enabled.
- Iteration budget stated up front; test split touched once.
- Backfill cost estimated, presented with its uncertainty, and approved.

## Risk

- Good clusters follow from good facets. A bad preprocessor is invisible until the topic map is
  useless, and by then the cost is already spent.
- Mostly-skipped or duplicated facet output reads as "no signal in this traffic" when it is almost
  always an instrumentation or preprocessing failure.
- A facet promoted into an enforcement role is an unvalidated classifier with no fitness verdict
  attached.
- Clustering finds structure in anything. A clean-looking topic map is not evidence that the facet
  measures what you think it measures.

## Braintrust

**Preprocessor.** `get_project_settings` for the configured and effective defaults; the built-in
`thread` returns the LLM messages per span and the machinery dedups across them. Run
`project_default` on current traces and confirm non-null output before assuming it works — custom
tracing formats are where it silently returns `[]`. A custom preprocessor is TypeScript in QuickJS,
runs independently per source span with no cross-span state, and must return atomic top-level array
items; the system merges them in trace order and exact-dedups. `test_preprocessor_on_trace` before
attaching. If you write one and the project default is still `thread`, set it as the project default
so existing and new facets inherit it.

**Evaluating the facet.** Dataset rows carry **references, never materialized text** (`trace_ref`,
`group_ref`, `_full_row_id_ref` — shapes in `reference.md`). Ground-truth run: `run_eval` with the
facet as task, `expected` unset, **no scorers**; spot-check the outputs. Subsequent runs pass
`{"experiment_name": "<reference-experiment>"}` as the dataset so those outputs become `expected`.
Inline facet tasks wrap the definition in `inline_function` — not `inline_prompt`, not fields
directly under `task`. Facet execution normalizes any `no_match_pattern` hit to `no_match`, so score
the **normalized value**, never source strings like `NONE` or `Neutral`.

**Saving and deploying.** `create_preprocessor`, then `create_facet` with the evaluated prompt and
no-match behavior, then re-run a few reviewed examples through the saved facet to confirm
persistence did not change its behavior. Reuse an existing automation only when data source, scope
and grouping, filter, preprocessor, sampling, topic window, **and** generation cadence are all
compatible — matching scope and filter alone is not sufficient. Sharing an automation batches facet
calls, which is the reason to bother. Pass `automation_id` explicitly whenever more than one exists.

Rewind sizing and the cost formula are in `reference.md`. Present eligible count, window, token
assumptions, base estimate, and uncertainty buffer, and get approval on the window *and* the
estimate — permissions checks on the mutating tool are not a substitute for approving a spend.
````

- Previous: [braintrust-design-human-eval-review](https://syn4pse.dev/terminal/braintrust-design-human-eval-review.md)
- Next: [braintrust-map-eval-evidence](https://syn4pse.dev/terminal/braintrust-map-eval-evidence.md)
- [Terminal](https://syn4pse.dev/terminal.md)
- [Suggest a change](mailto:terminal@syn4pse.dev?subject=Terminal%20suggestion)
