Terminal
Terminal/Skill
Builds an LLM evaluation for your app or agent, runs it, and returns an Opik experiment with scores and its link.
By Comet
Works with
npx skills add comet-ml/opik-skills --skill opik-evaluateDefinition of done: an experiment in Opik with scores, its link, and a score summary the user can act on — produced by running the user's app (or prompt) over a set of cases with scoring that targets real failure modes. If the evaluation can't be built or run, stop at the first genuine blocker and return exactly one next step. A dataset with no run, a judge prompt with no scores, or a plan in prose is not success.
Operate: ground the cases in what actually goes wrong, score with code before judges, judge one failure mode at a time, run once end to end, read the scores back from Opik — and change no application code. The only file this skill writes is a runner outside the repo.
The entry point is /opik-evaluate (evaluate the app in this repo), /opik-evaluate <what> ("the RAG answers", "the refund flow"), or /opik-evaluate <dataset-or-suite> (run against existing cases). Infer the rest; treat these as optional overrides:
evaluate() when you need built-in metrics or exact matching) · judge model · sample size (default: 20–50 items).Ask only at a genuine, non-inferable blocker (see Blockers).
Confirm Opik is reachable (~/.opik.config or OPIK_API_KEY; otherwise → Blocker). Find the entrypoint to evaluate (the function a trace's root span names, or the one the user points at). Check what already exists — client.get_test_suites(project_name=…), client.get_datasets() — and reuse before creating.
Have an eval already? Audit it first (references/eval-audit.md): unvalidated judges, no error analysis, vanity metrics. Fix the worst gap, then run.
Read real traces before writing any scorer — client.search_traces(project_name=…, max_results=100) (errors, low scores, long durations first). Categorize what goes wrong and how often (references/error-analysis.md). No traces yet → generate cases (references/generate-synthetic-data.md) and say so in the report.
| Situation | Use |
|---|---|
| Agent / chatbot, expectations are behaviors ("mentions Paris", "declines legal advice") | Test suite — items + string assertions checked by a judge; opik.run_tests() |
Exact expected outputs, or built-in metrics (Hallucination, AnswerRelevance, RAG ContextPrecision/ContextRecall) | Dataset + evaluate() with scoring_metrics |
| The thing under test is a prompt version, not code | Server-side: client.rest_client.experiments.execute_experiment(...) — no runner |
Prefer the test suite for agents; it is what /opik-test and /opik-compare operate on.
import opik
client = opik.Opik()
suite = client.get_or_create_test_suite(
name="<project>-eval", project_name="<project>",
global_assertions=["<one behavior every answer must show>"], # optional
)
suite.insert([
{"data": {"input": t.input, "source_trace_id": t.id}, "assertions": ["<what a correct output does>"]}
for t in sampled_traces
])
For the dataset path: client.get_or_create_dataset(name, project_name) then dataset.insert([{"input": …, "expected_output": …}]). Store inputs verbatim; keep source_trace_id so cases trace back to production.
Equals, Contains, RegexMatch, IsJson, JsonSchemaMatch, LevenshteinRatio from opik.evaluation.metrics whenever the check is mechanical.references/write-judge-prompt.md). As a suite assertion, or as a GEval / custom BaseMetric on the dataset path.references/evaluate-rag.md).references/validate-evaluator.md) — TPR/TNR, not accuracy.Write the task adapter as a temp file outside the repo (needs the app's provider credentials — absent → Blocker). Never run a production entrypoint that writes, sends, or spends.
# Test suite
results = opik.run_tests(test_suite=suite, task=lambda item: {"input": item["input"], "output": str(app(item["input"]))},
experiment_name="baseline-<sha>", model="<judge model>",
generate_report=False) # default True writes opik_test_suite_reports/ into cwd — keep the repo clean
# Dataset
from opik.evaluation import evaluate
res = evaluate(dataset=dataset, task=task, scoring_metrics=[...], experiment_name="baseline-<sha>",
scoring_key_mapping={"reference": "expected_output"}) # map dataset keys onto metric args
project_name matters: datasets, suites, prompts, and experiments are project-scoped, and it must match the tracing project if the app uses @track. Set it when creating the dataset or suite — evaluate() inherits the dataset's project, and its own project_name kwarg is deprecated (the SDK warns and ignores it).
Judge credential guard: if the LLM judge (suite assertions, or an LLM metric) has no provider key, run_tests/evaluate do not raise — every item scores 0 with scoring_failed=True and a "Missing credentials" reason, and the experiment is still created. Check for that before reporting; it is a Blocker ("set the judge's provider key and rerun"), not a result.
exp = client.get_experiment_by_id(results.experiment_id) # or res.experiment_id
items = exp.get_items() # dataset_item_data, evaluation_task_output, feedback_scores [{name,value,reason}], assertion_results [{passed,reason}], trace_id
# aggregate: mean per score name; pass rate = items where every assertion passed / items with assertions
On the dataset path res.aggregate_evaluation_scores().aggregated_scores gives per-metric statistics directly. Name the worst items and the failure mode each hit — that is the actionable part. (get_experiment_by_name is deprecated; use get_experiments_by_name / get_experiment_by_id.)
Experiment link, the score table, the three worst items with their reasons, what the cases were grounded in (traces vs synthetic), and one next step (see Output). This run is the baseline /opik-compare will compare against.
Stop at the earliest blocker and return exactly one next step:
opik configure, then rerun /opik-evaluate."answer(question))."OPENAI_API_KEY (or the relevant key) and rerun."synthetic and I'll generate them."User-facing: a short human message — the experiment link, the score table, the worst items with reasons, the case source, and the single next step. Not a raw dump of every item.
Underneath (for composition / evals), one shape:
status: evaluated | blockedshape: test_suite | dataset | server_sidecases: name, id, count, source (traces | provided | synthetic | existing)scoring: list of {name, kind: heuristic|judge|assertion, failure_mode}experiment: id, name, url, projectscores: list of {metric, value} (pass rate included)worst: list of {dataset_item_id, input, score_or_assertion, reason, trace_url}next_step: exactly oneInvariants: evaluated carries an experiment.url and non-empty scores; each judge in scoring names one failure_mode; blocked carries exactly one next_step; every path leaves the codebase unchanged.
Agent, from traces. /opik-evaluate. 100 traces read: 30% give wrong refund windows, 10% invent policies. Suite support-bot-eval, 40 items from traces, assertions "States the refund window as 5–7 business days" and "Does not invent a policy not present in the docs". Runner built, run_tests → 27/40. Read back: worst items all hit the refund assertion. → evaluated; next step = "/opik-test isn't needed — the suite exists; fix retrieve() and run /opik-compare support-bot-eval".
RAG, dataset path. /opik-evaluate the RAG answers. Dataset with input, expected_output, context; evaluate() with ContextPrecision, ContextRecall, Hallucination. Recall 0.61 is the weak stage. → evaluated; next step = "raise top-k / fix the retriever, then /opik-compare".
Prompt only. The target is a prompt version in the library. execute_experiment with the variant; poll until items fill; read back. → evaluated with shape: server_side.
Blocked. No entrypoint is inferable from the repo. → blocked: "Which function should I evaluate?"
project_name where the object is created. To get_or_create_dataset, get_or_create_test_suite, create_prompt. evaluate() and run_tests() inherit it from the dataset/suite (the evaluate(project_name=…) kwarg is deprecated)./opik-compare reads the same numbers later.Building a judge before reading a single trace; a "quality 1–10" judge; scoring with a judge what Equals could check; a dataset with no run; reporting an aggregate without naming the worst cases; writing the runner into the repo; editing app code to make the eval pass; skipping project_name; deprecated get_experiment_by_name.
Methodology, in this skill's own references: references/eval-audit.md (audit an existing pipeline), references/error-analysis.md (failure categorization from traces), references/generate-synthetic-data.md (dimension-based inputs), references/write-judge-prompt.md (binary judges), references/validate-evaluator.md (TPR/TNR calibration), references/evaluate-rag.md (retrieval vs generation).
SDK detail lives in the opik skill, installed beside this one — paths relative to this file: ../opik/references/evaluation-test-suites.md (suites, run_tests, results), ../opik/references/evaluation-datasets.md (evaluate(), 60+ metrics, OQL, datasets from traces), ../opik/references/production.md (search_traces). If your host lays skills out differently, locate the opik skill's references/ directory.
If the opik skill isn't installed, say so in the report and use https://www.comet.com/docs/opik/ rather than working from memory.