Terminal
Terminal/Skill
Turns graded evaluation traces and production logs into fine-tuning examples and preference pairs, including rejection sampling and DPO pairs from passing and failing runs.
By Seth Hobson
Works with
npx skills add wshobson/agents --skill trace-to-training-dataThis skill assumes eval-harness-first
already graded the traces being
converted here — goldens, graders,
and runs/<run-id>/results.json
all exist before conversion
starts. This is the flywheel edge
that skill names in its own flow:
"the same labeled traces become
the training set." Conversion
happens here; grading already
happened upstream.
Input: graded traces —
eval/goldens.jsonl plus
runs/<run-id>/results.json, each
row carrying a task_id, a
verdict from the grader, and a
reward when the task supports a
scalar score (judge score,
execution partial-credit, or an
RLVR verifier):
{"task_id": "t-042", "trace_id": "t-042-a3",
"messages": [{"role": "user", "content": "..."}],
"verdict": "pass", "reward": 0.91,
"grader": "exact_match"}
Output format: rows shaped
exactly like dataset-curation's
Format Selection table — SFT
messages rows or DPO
prompt/chosen/rejected
pairs — so this skill's output is
that skill's input with no
reshaping step in between.
The eval harness already did the
labeling work: every trace in
results.json carries a verdict,
and often a reward, before this
skill ever touches it. Converting
a graded trace into a training
row is mechanical — pick a shape
from dataset-curation's table,
map fields, write JSONL.
Curation is the work that
remains — which traces clear a
quality bar, which pairs are
informative, and which rows must
never enter the training set at
all.
Treat any conversion step that
requires re-judging a trace as a
sign the harness is missing a
grader, not a gap this skill
should paper over. A trace with
no verdict or reward isn't
convertible yet — route it back
to eval-harness-first first,
don't hand-label it here to
unblock conversion.
preference-optimization's
Pair Construction section owns
the full selection formula;
this skill supplies the graded
trajectories it consumes.eval/goldens.jsonl ID
out of every converted SFT and
DPO set — a trace that also
appears as a golden trains on
the exact item the checkpoint
gets graded against later,
silently inflating every
subsequent eval run.dataset-curation's dedup
method field, run against
whatever training data already
exists before this batch merges
in.run_id and trace_id
— dataset-curation's
Provenance field checks for
exactly this link back to
trace-to-training-data
output; a row with no traceable
source isn't ready to merge.eval-harness-first — produces
the graded traces this skill
converts; a trace with no
verdict or reward isn't
convertible yet, route it back
there before conversion.dataset-curation — owns the
target formats and the dataset
card this skill's provenance
data feeds; converted rows must
match its Format Selection
table field names exactly, not
an approximation of them.preference-optimization —
consumes the DPO pairs this
skill builds and owns the full
μ−2σ rejection-selection
formula referenced above.Worked JSONL-to-JSONL conversions
— graded trace to SFT row, trace
pair to DPO pair, correction to
SFT row, the rejection-sampling
loop, and the goldens-holdout
check — live in
references/conversion-recipes.md.