Terminal
Terminal/Skill
Puts an LLM judge or Python metric on live production traces, with sampling and a cost cap, and confirms new traces get scored.
By Comet
Works with
npx skills add comet-ml/opik-skills --skill opik-online-evalDefinition of done: an online evaluation rule exists on the project, is enabled, and has scored at least one new trace — confirmed by reading a fresh trace's feedback score, not by the create call returning. The rule scores one failure mode, samples at a rate the project's volume can afford, carries a cost cap, and maps its variables to the fields the traces actually have. If traffic hasn't arrived yet, the done state is "rule live, unverified — watch this filter". If the rule can't be created, stop at the first genuine blocker and return exactly one next step.
Operate: one failure mode per rule, sample before you scale, cap the spend, verify on a real trace — and change no application code. This skill writes to Opik only.
The entry point is /opik-online-eval <what to score> ("hallucinations", "the refund-window answer", "refusals"), /opik-online-eval right after /opik-evaluate validated a judge (take that judge live), or /opik-online-eval <project>. Infer the rest; treat these as optional overrides:
span for one model call, thread for whole conversations) · sampling rate (default: 1.0 under ~1k traces/day, else 0.1–0.2) · filters (default: none; typical: an environment or tag filter) · judge model · cost cap (default: set one) · variable mapping (default: input → input, output → output).Ask only at a genuine, non-inferable blocker (see Blockers).
Confirm Opik is reachable (~/.opik.config or OPIK_API_KEY; otherwise → Blocker). Resolve the project id from a trace or by name. Decide the judge:
/opik-evaluate already validated → reuse its prompt and output schema verbatim.../opik-evaluate/references/write-judge-prompt.md).Scope: trace by default; span when the check is about one LLM/tool call; thread when the check needs the whole conversation (thread rules wait for the thread to go inactive — 15 min by default).
Read three recent production traces and note the real shape of input and output. Experiment runs land in the same project with a different input shape (their metadata carries test_suite_experiment_id), so skip those — filtering on the app's entrypoint is the reliable way: client.search_traces(project_name=…, max_results=3, filter_string='name = "<entrypoint>"'). (OQL has no is_empty for metadata.* keys.) Variables are plain field paths, dot-notation for nested keys (output.answer, input.messages) — never {{ }} templates. A wrong path is the most common reason a rule silently scores nothing.
import opik
client = opik.Opik()
existing = client.rest_client.automation_rule_evaluators.find_evaluators(project_id="<project_id>")
Same name or same failure mode already there → don't create a second one; report exists (offer to adjust sampling/enable). When the hosted MCP is connected, list('online_rule', project_id=…) and list('score_name', project_id=…) show the same, with each rule's type, enabled flag, and sampling rate.
There is no high-level SDK wrapper; use the REST client. LLM-as-judge, trace scope:
from opik.rest_api.types import (
AutomationRuleEvaluatorWrite_LlmAsJudge, LlmAsJudgeCodeWrite,
LlmAsJudgeModelParametersWrite, LlmAsJudgeMessageWrite, LlmAsJudgeOutputSchemaWrite,
)
rule = AutomationRuleEvaluatorWrite_LlmAsJudge(
action="evaluator", # required literal; the model rejects the payload without it
name="refund_window_correct", # becomes the feedback-score name on every scored trace — use underscores, not hyphens: OQL parses `feedback_scores.a-b` as an operator
project_ids=["<project_id>"],
sampling_rate=0.2, # fraction of SDK-logged traces scored
enabled=True,
filters=[], # e.g. [{"field": "tags", "operator": "contains", "value": "production"}]
code=LlmAsJudgeCodeWrite(
model=LlmAsJudgeModelParametersWrite(name="<judge model>", temperature=0.0),
messages=[LlmAsJudgeMessageWrite(role="USER", content="<the validated judge prompt using {{input}} and {{output}}>")],
variables={"input": "input", "output": "output"}, # field paths from step 2
schema_=[LlmAsJudgeOutputSchemaWrite(name="refund_window_correct", type="BOOLEAN",
description="True if the response states 5-7 business days")],
max_cost_usd=5.0, # per-rule spend cap — set it
),
)
created = client.rest_client.automation_rule_evaluators.create_automation_rule_evaluator(request=rule)
Notes: on Opik Cloud without your own provider key, model.name="opik-free-model" uses the workspace's built-in free provider; the Python attribute is schema_ (wire name schema); sampling_rate applies to production traces only (experiment traces are always scored in full); trigger_scope defaults to production. Span and thread variants: AutomationRuleEvaluatorWrite_SpanLlmAsJudge, AutomationRuleEvaluatorWrite_TraceThreadLlmAsJudge. Python metric: AutomationRuleEvaluatorWrite_UserDefinedMetricPython with code={"metric": "<python source defining a BaseMetric>", "arguments": {"output": "output"}}.
Endpoint, if scripting outside Python: POST /v1/private/automations/evaluators/ with the same body.
import time
for _ in range(12): # ~2 min
scored = client.search_traces(project_name="<project>", max_results=1,
filter_string='feedback_scores.refund_window_correct is_not_empty')
if scored: break
time.sleep(10)
If the score name already contains a hyphen (an existing rule), double-quote the key or the OQL parser reads the hyphen as an operator: filter_string='feedback_scores."refund-window-correct" is_not_empty'.
A scored trace → live. None, and the project had no new traces in the window → live_unverified with the filter to watch. None, but traces did arrive → read the rule's logs (get_evaluator_logs_by_id(id)) — a variable-path error or model failure shows there; fix and re-verify.
Rule name/id, scope, sampling, cost cap, the score name, the verification trace link, and one next step (see Output). Natural next steps: /opik-diagnose will now surface low scores on this name; an alert on the score threshold; or, if the judge wasn't validated first, "validate it against 20 human labels (/opik-evaluate) before anyone acts on it".
Stop at the earliest blocker and return exactly one next step:
opik configure, then rerun /opik-online-eval."/opik-online-eval <what> <project>."output is {answer, sources} — should the judge read output.answer? (I'll map it that way unless you say otherwise.)" — ask only when the mapping is genuinely ambiguous.User-facing: a short human message — the rule (name, scope, sampling, cap), the score name, the verification trace as a clickable Opik UI link (or the watch filter if unverified), and the single next step. Not JSON.
Underneath (for composition / evals), one shape:
status: live | live_unverified | exists | blockedrule: id, name, type (llm_as_judge | user_defined_metric_python | span/thread variants), scope, sampling_rate, filters, max_cost_usd, enabledscore_namevariables: the field-path mapping usedverification: trace_id, trace_url, value (when live); watch_filter (when unverified)source: sdk | mcpnext_step: exactly oneInvariants: live carries a verification.trace_id; every created rule has a max_cost_usd and a sampling_rate; one failure mode per rule; exists created nothing; blocked carries exactly one next_step; every path leaves the codebase unchanged.
Validated judge goes live. /opik-evaluate calibrated "states the refund window as 5–7 business days" (TPR 0.95). /opik-online-eval: project support-bot, ~5k traces/day → sampling 0.1, filter tags contains "production", cap $5, mapping output → output.answer (traces nest it). Created; 40 s later a trace carries refund-window-correct = 1. → live; next step = "alert when the 1h mean drops below 0.8".
Mechanical check. "Make sure every prod answer is valid JSON." → Python metric rule (IsJson on output), sampling 1.0 (cheap), no judge. → live.
No traffic yet. Rule created on a project that receives traces only in business hours. → live_unverified: "watch feedback_scores.<name> is_not_empty — or send one traced request".
Already there. A rule named hallucination exists, enabled, at 0.5. → exists; next step = "lower sampling to 0.1 if cost is the concern".
A rule at sampling_rate: 1.0 on a high-volume project without saying what it costs; no max_cost_usd; {{input}}-style template syntax in variables (they are field paths); a holistic "quality" judge as a rule; a judge for a mechanical check; a second rule for the same failure mode; declaring success from the create call without a scored trace; taking an unvalidated judge live and calling its scores truth; editing application code.
Production and observability detail live in the opik skill, installed beside this one — paths relative to this file: ../opik/references/production.md (online evaluation, variable mapping, feedback scores, alerts), ../opik/references/observability.md (trace/span/thread model), ../opik/references/evaluation-datasets.md (OQL filters, feedback_scores.<name> operators). Judge design: ../opik-evaluate/references/write-judge-prompt.md, ../opik-evaluate/references/validate-evaluator.md. If your host lays skills out differently, locate the opik skill's references/ directory.
If the opik skill isn't installed, say so in the report and use https://www.comet.com/docs/opik/ rather than working from memory.