Terminal
Terminal/Skill
Guides a human through reviewing real AI app traces to find failure modes before writing quality metrics, with a managed or local review UI.
By Confident AI (DeepEval)
Works with
npx skills add confident-ai/eval-skills --skill eval-discoverRead the operating agreement, data sourcing, and the selected surface: managed or local review.
For bundled local tools and exact commands, read toolkit setup. Run the workspace installer for this skill; it preserves existing customized files.
Reuse existing traces and review notes. Seek production examples, complaints/incidents, support tickets, expert examples, and targeted synthetic variations in that order as practical. Ask about unresolved source access and retainability before pulling content. Inspect one real execution before a new managed connection.
Create a diverse discovery sample. Include ordinary traffic and random exploration alongside difficult examples; cluster only when it helps. Explain which dimensions the sample covers and which it misses. Fifteen to twenty-five examples can start a review, but stop/grow based on discovery saturation and domain breadth, not a universal count.
Do not monitor indefinitely or claim a background watcher persists after the session ends. Local autosave must work independently of the agent. Resume from saved annotations and revision IDs on the next turn.
Deliver the reviewed sample, durable human notes, provisional themes, and coverage gaps linked to source traces. Separate discovery counts from prevalence estimates. Use eval-error-analysis when installed to consolidate annotations into a reviewed failure taxonomy; otherwise describe that next outcome using the saved evidence. Confirm failure definitions before creating graders.