Terminal
Terminal/Skill
Scores AI agent outcomes in production with Inngest, covering deferred scorers, sessions, traces, and variant experiments to compare prompts, models, and tools.
By Inngest
Works with
npx skills add inngest/inngest-skills --skill inngest-agent-evalsUse this skill when the user wants to evaluate AI agents or AI workflows in production, add scoring, compare prompts/models/tools, group related runs, or debug why an agent outcome was good or bad.
Agent Evals is not a separate package. It is the production evaluation workflow built from Inngest functions, durable steps, scores, deferred scorers, sessions, traces, experiments, and Insights.
Check the public docs first when exact API details matter:
group.experiment() reference: https://www.inngest.com/docs/reference/typescript/v4/functions/group-experimentStart from the outcome, not the mechanism:
step.score() or
inngest.score().createScorer() and defer(),
or explicit inngest.score({ runId }).meta.sessions for repeated high-cardinality identifiers such as
conversation_id, ticket_id, agent_run_id, or import_id.group.experiment() for prompt, model, provider, tool, workflow, or
rollout comparisons.experimentRef to deferred or later scores when the score should be
credited to the selected variant.npm install inngest@latest.step.score() requires scoreMiddleware() from inngest/experimental on
the Inngest client.createScorer() is imported from inngest/experimental and the scorer must
be registered in the serve handler with other functions.group.experiment() and the
experiment helper from inngest.Client shape:
import { Inngest } from "inngest";
import { scoreMiddleware } from "inngest/experimental";
export const inngest = new Inngest({
id: "support-agent",
middleware: [scoreMiddleware()],
});
Add sessions when events belong to a user flow, conversation, ticket, import, or agent task that someone will inspect repeatedly.
await inngest.send({
name: "support/ticket.created",
data: {
ticketId: "tk_123",
message: "I can't sign in.",
},
meta: {
sessions: {
ticket_id: "tk_123",
},
},
});
Rules:
conversation_id; put the actual ID in the
value.step.invoke() and step.sendEvent() when
downstream runs should join the same session.environment: prod; use Insights for
that style of filtering.Use direct scoring when the score is known during the function run.
Good direct scores:
export const answerTicket = inngest.createFunction(
{
id: "answer-ticket",
triggers: [{ event: "support/ticket.created" }],
},
async ({ event, step }) => {
const answer = await step.run("generate-answer", () =>
generateAnswer(event.data.ticketId)
);
const passed = await step.run("check-answer", () => validateAnswer(answer));
await step.score("score-answer-quality", {
name: "answer-quality",
value: passed,
});
return { answer, passed };
}
);
Use stable score names. Changing a score name creates a separate metric. Score values must be finite numbers or booleans.
Use deferred scoring when the useful signal arrives after the parent workflow finishes, such as user feedback, conversion, ticket reopen, retention, or a slow LLM-as-judge run.
import { createScorer } from "inngest/experimental";
import { z } from "zod";
export const feedbackScorer = createScorer(
inngest,
{
id: "support-feedback-scorer",
schema: z.object({ ticketId: z.string() }),
},
async ({ event, step }) => {
const feedback = await step.waitForEvent("wait-for-feedback", {
event: "support/feedback.received",
timeout: "7d",
if: `async.data.ticketId == '${event.data.ticketId}'`,
});
return {
name: "user-feedback",
value: feedback?.data.helpful ? 1 : 0,
};
}
);
Trigger it from the producing function:
async ({ event, step, defer }) => {
const answer = await step.run("generate-answer", () =>
generateAnswer(event.data.ticketId)
);
defer("score-feedback", {
function: feedbackScorer,
data: { ticketId: event.data.ticketId },
});
return { answer };
}
Guardrails:
serve({ functions: [...] }).defer() is fire-and-forget; do not await it.null when a signal times out, based on the
product semantics.Use group.experiment() when comparing prompts, models, providers, tools,
workflow rewrites, or operational settings against real traffic.
import { experiment } from "inngest";
const { result, variant, experimentRef } = await group.experiment(
"answer-style",
{
variants: {
concise: () => step.run("answer-concise", () => answerConcise(event.data)),
detailed: () =>
step.run("answer-detailed", () => answerDetailed(event.data)),
},
select: experiment.bucket(event.data.accountId, {
weights: { concise: 50, detailed: 50 },
}),
}
);
Selection strategy:
experiment.weighted() for run-level traffic splits.experiment.bucket(stableId, { weights }) when a user, account, or tenant
should usually keep the same experience.experiment.custom() when assignment comes from a database, flag service, or
rollout table.experiment.fixed() to force one variant during testing or after choosing a
winner.Rules:
step.* tool.experimentRef and the parent run ID if a later, separate process
will score the selected variant.When deferred scoring a variant, pass the ref:
defer("score-answer-feedback", {
function: feedbackScorer,
data: { ticketId: event.data.ticketId },
experiment: experimentRef,
});
When scoring from a later run explicitly:
await inngest.score.experiment({
name: "clickthrough",
value: 1,
experiment: experimentRef,
runId: originalRunId,
});
When adding Agent Evals to an existing app:
meta.sessions at event producers before building dashboards around
correlation.group.experiment() only around one isolated decision at a time.Use inngest-brownfield-audit first when the repo has many candidate
workflows. Use inngest-agents with this skill when the work is an AgentKit or
durable agent workflow.
NaN, or Infinity.scoreMiddleware() and then assuming step.score() is missing.runId.meta.sessions.experimentRef plus the original run ID when needed.