# eval-discover

Guides a human through reviewing real AI app traces to find failure modes before writing quality metrics, with a managed or local review UI.

Skill by Confident AI (DeepEval). Category: Testing.

## Install

Project (for this project only):

```sh
npx skills add confident-ai/eval-skills --skill eval-discover
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

Global (for every project on this machine):

```sh
npx skills add confident-ai/eval-skills --skill eval-discover -g
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

## Tags

- Runs on your computer: It runs as a program on your machine, with your access.

## Security audits

- Gen Agent Trust Hub: Pass
- Socket: Pass
- Snyk: Warn

Installs: 1. Installs counted by skills.sh. [Open on skills.sh](https://www.skills.sh/confident-ai/eval-skills/eval-discover)

## Source

[Open the source](https://github.com/confident-ai/eval-skills/tree/HEAD/skills/eval-discover). Checked 2026-10-10. This summary may contain AI-generated text, so it can be wrong. Check the source before you rely on it.

## SKILL.md

````markdown
---
name: eval-discover
description: Find realistic AI application examples and conduct human-led trace review to discover failure modes. Use before inventing quality metrics; supports managed review or an AI-built local annotation UI.
license: Apache-2.0
metadata:
  author: Confident AI
  version: "1.1.0"
---

# Discover failures with a human

Read [the operating agreement](references/workflow.md), [data sourcing](references/data-sourcing.md), and the selected surface: [managed](references/managed.md) or [local review](references/local-review.md).

For bundled local tools and exact commands, read [toolkit setup](references/toolkit.md). Run [the workspace installer](scripts/setup_workspace.py) for this skill; it preserves existing customized files.

## Assemble evidence

Reuse existing traces and review notes. Seek production examples, complaints/incidents, support tickets, expert examples, and targeted synthetic variations in that order as practical. Ask about unresolved source access and retainability before pulling content. Inspect one real execution before a new managed connection.

Create a diverse discovery sample. Include ordinary traffic and random exploration alongside difficult examples; cluster only when it helps. Explain which dimensions the sample covers and which it misses. Fifteen to twenty-five examples can start a review, but stop/grow based on discovery saturation and domain breadth, not a universal count.

## Review loop

1. Show full behavior in its natural form, not a raw JSON cell. Start with open-ended notes: what went wrong, what should have happened, and why it matters.
2. Save human observations immediately with stable annotation and trace IDs. Note provisional themes to guide further sampling; defer taxonomy consolidation and assignment review to the error-analysis stage.
3. Alternate breadth (new task types) and depth (more instances of a known failure). Revisit earlier examples when the meaning of failure changes.
4. After several human-reviewed records establish a category, optionally delegate one category per background analyzer to find candidates. Without delegation, scan sequentially. Reviewers accept or dismiss suggestions; suggestions never become human ground truth automatically.
5. Track confirmed versus suggested findings, reviewed coverage, and new-category discovery. Pause when further review mostly repeats known categories, then propose the next missing coverage or curation step.

Do not monitor indefinitely or claim a background watcher persists after the session ends. Local autosave must work independently of the agent. Resume from saved annotations and revision IDs on the next turn.

## Exit artifact

Deliver the reviewed sample, durable human notes, provisional themes, and coverage gaps linked to source traces. Separate discovery counts from prevalence estimates. Use `eval-error-analysis` when installed to consolidate annotations into a reviewed failure taxonomy; otherwise describe that next outcome using the saved evidence. Confirm failure definitions before creating graders.
````

- Previous: [eval-descent](https://syn4pse.dev/terminal/confident-eval-descent.md)
- Next: [eval-error-analysis](https://syn4pse.dev/terminal/confident-eval-error-analysis.md)
- [Terminal](https://syn4pse.dev/terminal.md)
- [Suggest a change](mailto:terminal@syn4pse.dev?subject=Terminal%20suggestion)
