Terminal
Terminal/Skill
Turns bad production traces from an agent that logs to Braintrust into a dataset, scorers and offline evals for fixing repeated failures.
By Braintrust
Works with
npx skills add braintrustdata/braintrust-skills --skill agent-auto-improvementUse real production traces to improve a specific behavior of an agent or LLM app. The source of truth is traced production data and a remote Braintrust dataset, not hand-written examples or hardcoded local JSON.
The loop is:
bt setup instrument first.bt CLI is installed and authenticated (bt setup, or BRAINTRUST_API_KEY in CI).https://www.braintrust.dev/docs/reference/sql (also prefetched at .bt/skills/docs/reference/sql.md if bt setup skills ran).Do not try to eval "the whole agent" at once. Pick one decision point that is observable in traces and cheap to replay offline, for example:
Find where that behavior lives in the repo: the system prompt, the tool schema/description, and the runtime code that executes it. These are the files the loop will improve.
Then confirm bt points at the right org and project, and capture the project ID — SQL table functions like project_logs(...) require IDs, not names:
bt status --json
bt projects list --json
Start with trace-derived evidence, not prompt intuition. Query for failures with bt sql, scoped to the spans that capture the target behavior. Every query needs a time-range filter on created (the query linter blocks unscoped queries), and queries that project large fields like input/output need LIMIT 100 or less:
# Errors on a specific tool's spans
bt sql "SELECT id, root_span_id, input, output, error \
FROM project_logs('<project-id>') \
WHERE created > now() - interval 30 day \
AND span_attributes.name = '<tool-or-span-name>' AND error IS NOT NULL \
LIMIT 100" --json
# Low-scoring or negative-feedback traces
bt sql "SELECT id, root_span_id, input, output, scores \
FROM project_logs('<project-id>') \
WHERE created > now() - interval 30 day AND scores['<score-name>'] < 0.5 \
LIMIT 100" --json
# Slow spans (duration in seconds)
bt sql "SELECT id, root_span_id, metrics.duration AS duration \
FROM project_logs('<project-id>') \
WHERE created > now() - interval 30 day AND metrics.duration > 20 \
ORDER BY duration DESC LIMIT 100" --json
If the project has no scores and the error field is never set, failures are usually latent in span contents instead: search tool span output payloads for failure signatures (non-zero exit codes, "not found", rejection messages), and look for behavioral patterns like retry loops, repeated identical calls, or a cheaper tool that should have been used. Aggregate by span_attributes.name first to see where the volume is.
Use bt view trace --object-ref project_logs:<project-id> --trace-id <root-span-id> to read full traces, and bt view span --id <row-id> for full payloads. Read enough individual failures to understand them concretely before generalizing.
Group what you saw into named categories, and separate them into three buckets:
Focus the loop on hard failures first. Write the taxonomy down (category id, description, how to detect it) — it becomes the spec for your scorers.
Turn real failing traces into a remote dataset using a dataset pipeline. Each row should contain everything needed to replay the decision offline:
input: the captured context at the decision point (prior messages, available state)expected (optional): a known-good output, if one existsmetadata: the baseline production output and its failure categories, plus the source trace idimport { DatasetPipeline } from "braintrust";
DatasetPipeline({
source: {
projectName: "<project-name>",
filter:
"span_attributes.name = '<tool-or-span-name>' AND error IS NOT NULL",
scope: "span",
},
transform: ({ id, input, output, metadata, trace }) => ({
input,
metadata: {
source_span_id: id,
baseline_output: output,
baseline_categories: categorize(output),
},
}),
target: {
projectName: "<project-name>",
datasetName: "<agent>-<behavior>-offline",
},
});
bt datasets pipeline run ./pipeline.ts --limit 200 --window 30d
The pipeline runner needs tsx (or vite-node) installed as a local dev dependency for TypeScript pipelines; pass --runner tsx if auto-detection fails.
Implement categorize from the taxonomy in step 3 — it is shared between the pipeline and the scorers. Refresh the dataset (rerun the pipeline) whenever it feels stale, too small, or skewed toward one category. Then verify the dataset actually contains the failures you want to attack:
bt datasets view <agent>-<behavior>-offline --limit 50
bt sql "SELECT metadata.primary_category AS cat, count(1) AS n \
FROM dataset('<dataset-id>') \
WHERE created > now() - interval 1 day \
GROUP BY cat ORDER BY n DESC" --json
Prefer deterministic code scorers for hard categories — they are free, fast, and stable across runs. Use LLM-as-a-judge (e.g. from autoevals) only for qualities a deterministic check cannot capture, and keep judges few and focused.
The scorer set that makes this loop work:
1 only when the output has zero hard failure categories.metadata.baseline_output.[0,1] score over the soft-inefficiency categories.import { categorize, HARD_CATEGORIES } from "./categorize";
export function hardQuality({ output }) {
const issues = categorize(output).filter((c) => HARD_CATEGORIES.has(c));
return {
name: "hard_quality",
score: issues.length === 0 ? 1 : 0,
metadata: { issues },
};
}
Unit-test the categorizer against known-bad and known-good outputs before trusting eval scores built on it.
The task function replays the captured context against the current system and returns the target behavior's output (for tool-call behaviors, capture the first relevant tool call rather than running the tool):
import { Eval, initDataset } from "braintrust";
import {
hardQuality,
issueCount,
beatsBaseline,
softEfficiency,
} from "./scorers";
Eval("<project-name>", {
experimentName: process.env.EVAL_EXPERIMENT,
data: initDataset({
project: "<project-name>",
dataset: "<agent>-<behavior>-offline",
useOutputAsExpected: false,
}),
task: async (input) => runAgentDecision(input),
scores: [hardQuality, issueCount, beatsBaseline, softEfficiency],
});
runAgentDecision must call the same prompt, tools, and model configuration as production — import them from the app code rather than copying them, so prompt improvements are picked up automatically.
Two practical replay issues:
tool_result blocks whose matching tool_use was trimmed away, and start the history on a user message. Otherwise the provider API rejects the request.BRAINTRUST_API_KEY.Bounded smoke run first, full run once it is directionally better:
bt eval --first 20 first-step.eval.ts # bounded, non-final
bt eval first-step.eval.ts # full dataset, final
Note the experiment name printed in the summary, then resolve its ID — experiment(...) requires the ID, not the name:
bt experiments list --json
bt sql "SELECT root_span_id, scores \
FROM experiment('<experiment-id>') \
WHERE created > now() - interval 1 day \
AND span_attributes.type = 'score' AND scores['hard_quality'] < 1" --json
Priority order:
Do not trade away the hard score to optimize the soft score.
Prefer fixes that generalize across many traces:
If a fix only helps one or two dataset rows, it is overfitting; push on repeated categories instead.
After every meaningful change:
When new production failures appear later: refresh the dataset from the pipeline, rerun, keep iterating.
Only after the offline loop is healthy:
bt functions push, keeping it aligned with the offline categorizer.Until then, the offline eval is the primary optimization surface.
A good iteration improves the target behavior across many real traced cases by:
If a change improves the bounded run but regresses the full dataset, keep iterating — the full dataset is the real check.