# eval-dataset

Curate replayable AI eval cases from traces, feedback, or grounded synthetic scenarios; preserve provenance and build leakage-resistant splits. Not a generic dataset generation tool.

Skill by Confident AI (DeepEval). Category: Testing.

## Install

Project (for this project only):

```sh
npx skills add confident-ai/eval-skills --skill eval-dataset
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

Global (for every project on this machine):

```sh
npx skills add confident-ai/eval-skills --skill eval-dataset -g
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

## Tags

- Runs on your computer: It runs as a program on your machine, with your access.

## Security audits

- Gen Agent Trust Hub: Pass
- Socket: Pass
- Snyk: Pass

Installs: 1. Installs counted by skills.sh. [Open on skills.sh](https://www.skills.sh/confident-ai/eval-skills/eval-dataset)

## Source

[Open the source](https://github.com/confident-ai/eval-skills/tree/HEAD/skills/eval-dataset). Checked 2026-10-10. This summary may contain AI-generated text, so it can be wrong. Check the source before you rely on it.

## SKILL.md

````markdown
---
name: eval-dataset
description: Curate replayable AI eval cases from traces, feedback, or grounded synthetic scenarios; preserve provenance and build leakage-resistant splits. Not a generic dataset generation tool.
license: Apache-2.0
metadata:
  author: Confident AI
  version: "1.1.0"
---

# Build a dataset from evidence

Read [the operating agreement](references/workflow.md), [data sourcing](references/data-sourcing.md), and [artifact records](references/artifacts.md). For hosted storage use [managed guidance](references/managed.md).

For bundled local tools and exact commands, read [toolkit setup](references/toolkit.md). Run [the workspace installer](scripts/setup_workspace.py) for this skill; it preserves existing customized files.
For the managed route, use [concrete service operations](references/api-managed.md) after confirming the connected schema.

## Reuse and curate

Inspect existing cases and ask only what cannot be derived: whether they still match traffic, who verified references, and what decisions the eval must support. Preserve useful cases. Distinguish inputs from reference answers, rubrics, and baseline outputs; `expected` does not always mean a single correct sentence.

For every case retain source, permitted retention, failure/category tags, conversation or source group, necessary attachments/context, and reference origin. Capture the environment/fixture necessary to replay an agent. Keep source records unchanged.

Use confirmed failures as regression examples, but also include normal and negative cases. A challenge suite and a representative benchmark answer different questions; label them separately. Generate synthetic cases only to fill specific gaps and have a knowledgeable reviewer verify realism.

## Protect comparisons

Deduplicate and group related conversations, documents, customers where appropriate, and paraphrase families before splitting. Split by group so related examples do not cross boundaries. Stratify by relevant task/category where enough groups exist; do not select development cases just because their baseline scores are low.

Use development for diagnosis, validation for candidate selection, and a final test slice only once when enough data exists. With a small dataset, reserve a credible holdout if possible; otherwise report exploratory results and collect new confirmation cases. Freeze the split before optimization. Never redraw because scores are inconvenient.

Keep answers in grader-only storage outside an acting model's filesystem and tool reach. Remove public benchmark solution access when relevant. Do not rely on “do not read answers” instructions.

## Review and commit

Show actual cases with inputs, attachments, reference origins, and category counts in the chosen review surface. Confirm relevance and retention before committing production-derived data. Store source identifiers or sanitized substitutes when full records cannot be retained.

Record immutable dataset version/hash, sample purpose, grouping rule, and split IDs. Use the optional eval-run artifact helper to validate exported records and create a reproducible group split, or the existing runner's equivalent. Its simple splitter is not a statistical sampling service; inspect category coverage before locking the manifest.

Deliver the runnable input set, protected reference set, review status, and any coverage gaps. Do not mark unreviewed synthetic references as expert-approved.
````

- Previous: [eval-build](https://syn4pse.dev/terminal/confident-eval-build.md)
- Next: [eval-descent](https://syn4pse.dev/terminal/confident-eval-descent.md)
- [Terminal](https://syn4pse.dev/terminal.md)
- [Suggest a change](mailto:terminal@syn4pse.dev?subject=Terminal%20suggestion)
