Terminal
Terminal/Skill
Decides what an LLM app or agent should log, such as spans, metadata and resolved config, so failures can be debugged and sliced later.
By Braintrust
Works with
npx skills add braintrustdata/eval-library --skill braintrust-design-eval-instrumentationContract: references/interaction-contract.md. Calibration, templates, provenance: references/trace-schema.md.
reference.md) — one span per LLM
call, tool call, and scorer call, nested to mirror actual control flow.Of the four objects (references/platform-mechanics.md §1), this stage owns traces — the one
everything else consumes, and the only one you cannot backfill.
Field placement decides what the platform can ever do: scores in native scores (0–1);
tokens/latency/cost in native metrics, or they cannot be charted or gated on; slicing
variables in record metadata, or subgroup analysis needs re-labeling; scorer name/version in
span metadata; raw non-text input as a span Attachment (input.audio = Attachment(...)),
which renders inline so "listen to the actual clip" is one click. Note experiments surface
attachments more prominently than datasets — if human review of raw artifacts is part of the
loop, make an experiment the review surface.
A stratum not in metadata is a stratum you cannot report, with no retroactive fix. Include
scenario_id wherever items are variants of one scenario, so clustered standard errors stay
computable.
Instrumentation also fixes the evaluation scopes available later. Evaluators run against a
span, a trace, or a group of traces joined by a metadata key — so multi-turn behavior is only
measurable if something like conversation_id, session_id, or thread_id was emitted at the
time. Same no-retroactive-fix rule as any other stratum, with a sharper edge: a grouping key that
is present but inconsistently populated is worse than one that is absent, because group-scoped
evaluation will silently drop the traffic where it is null and report a confident number over the
remainder. Assert its coverage the way you assert any other transform invariant.