Report eval results
Contract: references/interaction-contract.md. Calibration, templates, provenance: references/presentation-checklist.md.
Trigger
- Writing up completed results: technical report, release summary, chart set, external post.
- Auditing a draft for claims the design does not support.
- A "state-of-the-art" or "best model" claim being drafted.
Do
- Give every headline metric point estimate + 95% CI + n + K. "82.4%" is not a result;
"82.4% [79.1, 85.7], n=520, K=3" is.
- Report the effective n — attempted, completed, errored per arm, plus how errors were
treated. "n=520" on a run where 80 items never completed is a misreport.
- Run the full presentation checklist in
reference.md. It is the deliverable's definition of
done.
- Disclose the search: how many arms were tried before the reported winner, and label every
number exploratory or confirmatory.
- Pin the configuration — model string, decoding params, prompt/scorer/dataset versions, serving
path, date. A result without re-run fields is a rumor.
- Calibrate language to evidence. "Ranks first on this benchmark, +9 pp [3, 15]" is defensible;
"state-of-the-art" implies consistency and robustness that must be demonstrated separately.
Avoid
- Do not imply a ranking from overlapping uncertainty.
- Do not hide attempted arms, killed runs, or abandoned configurations.
- Do not use language broader than the population and design support.
- Do not present a discovery or red-team yield as a population rate.
- Do not recompute or revise the analysis here; if the numbers look wrong, fix them in the
analysis and re-derive, rather than adjusting them in the writeup.
Check
- Every headline number carries interval, n, K — and every claim maps to a reported number.
- Effective-N accounting present per arm; search denominator disclosed.
- Every figure has uncertainty where applicable; captions state overlap.
- Re-run fields complete enough for someone else to reproduce the table.
- Known limitations — scorer artifacts, label error, uncovered strata — in the body, not a footnote.
Risk
- Polished presentation makes weak or exploratory evidence look decisive; the cleaner the chart, the
stronger the implied claim.
- A number quoted out of the report loses its interval and population immediately — write the
sentence you would accept being quoted alone.
- Selective reporting of the best arm is the most common integrity failure in eval writeups and is
invisible unless disclosed.
- External readers situate your numbers against benchmarks with different constructs; say what
yours measures.
Braintrust
Shared mechanics: references/platform-mechanics.md. Every number in the report comes off one
cached pull (§2) — a truncated pull produces a confidently wrong n, and n is the number readers
check first. Effective N is completed-row vs. error count, and the search denominator is the
experiment list (§6), not what anyone remembers trying; clean smoke tests and partial arms before
counting or they inflate it.
Re-run fields should be readable straight out of experiment metadata, because the experiment
design put them there. If they are missing, the report cannot honestly claim reproducibility — say
so as a limitation rather than reconstructing from memory.
Link the experiment or comparison view rather than pasting a screenshot — a permalink carries the
versions, metadata, and item-level data a reader needs to check a claim.