Terminal
Terminal/Skill
Workflow for tracing production issues to a root cause in Honeycomb, using broad queries, BubbleUp, and trace analysis.
By Honeycomb
Works with
npx skills add honeycombio/agent-skill --skill production-investigationStructured workflows for debugging production issues. The MCP tools document their own parameters — this skill focuses on the sequence of tool calls and how to interpret results to reach a root cause.
This workflow implements the core analysis loop (Define → Visualize → Investigate → Evaluate) from the observability-fundamentals skill. If BubbleUp returns nothing useful, the issue is often an instrumentation gap — add the missing attributes (see the otel-instrumentation skill) and try again.
get_workspace_context → environments and datasetsget_slos → any SLOs in violation? (frames severity)get_triggers → any alerts firing? (narrows scope)find_queries → has anyone investigated this before?Run a broad query to see the shape of the issue:
error=true) by service/route/category, then
separately count exception event rows using event.name=exception and exception.type exists;
use sampled trace.trace_id values to drill into representative tracesAlso call get_service_map — it shows P95 durations between services and can immediately reveal which dependency is slow.
Exception data has two query surfaces: operation failures belong on spans (error=true, span
status, low-cardinality exception.slug/error category); full exception diagnostics may belong on
trace-correlated Logs API event rows. Do not assume exception.* exists on the containing span.
When investigating exceptions, discover the dataset schema first, query event.name=exception
with exception.type exists and trace.trace_id exists, take a sample, then pass its
trace.trace_id to get_trace(show_events=true). For legacy span-event exceptions, also check
name=exception and meta.signal_type=trace; Logs API events use event.name/body and
meta.signal_type=log.
If a service uses an exception-promoting LogRecordProcessor, some exception.* fields may also
appear on the containing span. Treat that as an explicit client-side compatibility feature, not a
Honeycomb guarantee: the event row remains authoritative for full diagnostics, and absence of
parent-span fields does not mean the exception event is missing.
This is the highest-value step. Once you have a query showing the anomaly:
run_bubbleup on the query result, selecting the outlier regionHow to interpret BubbleUp results:
deployment.version=v2.3.1 is 90% of slow requests but only 20% of baseline)db.query_duration is much higher in outliers)After BubbleUp identifies suspects:
get_trace to fetch the full traceWhat to look for in the trace waterfall:
Form a hypothesis from BubbleUp + trace analysis, then confirm:
Call create_board with:
HEATMAP first → BubbleUp the slow region → trace a slow request → verify with filtered queries
Count failed operation spans by service/route/category → count Logs API exception events by
event.name=exception and exception.type → sample trace.trace_id → get_trace(show_events=true)
→ verify with filtered queries. Do not use exception.message on the parent span as the only
exception search.
P99 grouped by deployment.version → BubbleUp comparing new vs old → trace from new version → verify
get_service_map → P99 on the slow dependency → relational query (any.service.name) to measure user impact → trace an affected request
If you find yourself reasoning any of these, follow the workflow anyway:
find_columns, expand time range, verify environment/dataset${CLAUDE_PLUGIN_ROOT}/skills/production-investigation/references/investigation-playbooks.md — Step-by-step playbooks for latency spikes, error surges, deployment regressions, dependency failures, SLO budget burn, and health checks${CLAUDE_PLUGIN_ROOT}/skills/production-investigation/references/bubbleup-guide.md — Detailed BubbleUp usage: selection types, time specifications, pagination, result interpretation${CLAUDE_PLUGIN_ROOT}/skills/production-investigation/references/trace-exploration.md — Trace structure, get_trace parameters and view modes, waterfall analysis, span events and links