Terminal
Terminal/Skill
Diagnoses why a Synthetic Monitoring check is failing by classifying the failure scope and breaking results down per probe.
By Grafana Labs
Works with
npx skills add grafana/gcx --skill synth-investigate-checkInvestigate Synthetic Monitoring check failures by triaging probe data, classifying failure scope, and identifying root cause.
-o json for agent processing, default format for user displaygcx synthetic-monitoring checks status <ID>
If the user provided a name instead of ID, list first with a job glob (the numeric ID is the NAME suffix, e.g. web-check-1001 is ID 1001):
gcx synthetic-monitoring checks list --job '*<name>*'
Early exit — OK: Status command reports OK (success rate at or above the check's alertSensitivity threshold: high = 95%, medium/default = 90%, low = 75%).
Report: "Check <job> is healthy. Success rate: %. <probe_count> probes up."
Stop unless the user asks for more.
Early exit — NODATA: No Prometheus metrics available.
enabled: true (gcx synthetic-monitoring checks get <ID> -o json | jq .spec.enabled)gcx synthetic-monitoring checks get <ID> -o json
Extract: job name, target, check type (http/ping/dns/tcp/traceroute), probe list, frequency, timeout, alertSensitivity, enabled flag.
For HTTP checks also note: any assertion settings, TLS config, expected status codes.
gcx synthetic-monitoring checks timeline <ID> --from now-1h --to now
Show the graph output to the user. Then analyze the pattern:
| Pattern | Classification |
|---|---|
| All probes at 0 (or near 0) | Target down |
| Subset of probes at 0, others healthy | Regional / network |
| Intermittent drops across multiple probes | Flapping / timeout |
| All probes drop at a specific point in time | Sudden onset — possible deployment or config change |
| Gradual decline | Degradation — timeout drift or resource exhaustion |
Use a longer window if the failure started more than 1h ago:
gcx synthetic-monitoring checks timeline <ID> --from now-6h --to now
Get the probe list for geographic mapping:
gcx synthetic-monitoring probes list -o json
Cross-reference the probe names from the check config against each probe's region field. Map failing probes to their regions.
All probes failing: Target/service issue — likely target down, SSL error, or DNS failure.
Subset of probes failing: Regional or network issue. Note which regions are affected:
Intermittent failures: Flapping. Consider: rate limiting, timeout too tight, flaky connectivity.
Resolve datasource UID if not already known:
gcx datasources list --type prometheus
If the filtered list comes back empty, the stack may leave the type field
blank in list payloads (known issue) — rerun without --type and pick the
Prometheus datasource by name.
Run per-probe success rate to pinpoint failing probes (use -o json for parsing, -o graph to show the user):
gcx metrics query -d <datasource-uid> \
'avg by (probe) (probe_success{job="<job>",instance="<target>"})' \
--from now-1h --to now --step 1m -o graph
For HTTP checks, also run HTTP phase latency to locate where time is spent:
gcx metrics query -d <datasource-uid> \
'avg by (phase) (probe_http_duration_seconds{job="<job>",instance="<target>"})' \
--from now-1h --to now --step 1m -o graph
For SSL/TLS cert expiry, DNS latency, per-probe error rates, and other patterns, see sm-promql-patterns.md.
Cross-reference signals against failure-modes.md (full signal/cause/next-action table and decision tree) to select the most likely failure mode:
connect or tls → TimeoutSynthesize findings into an actionable report (see Output Format below). Take next actions from the "Next Action" column of the failure mode's row in failure-modes.md.
If deeper investigation is needed (e.g., logs, infra repos), ask the user if they want to proceed.
If check config needs changes (probe selection, frequency, assertions), route to synth-manage-checks.
Early exit (OK):
Check: <job> (<target>)
Status: OK
Success rate: <rate>%
Probes up: <count>/<total>
Early exit (NODATA):
Check: <job> (<target>)
Status: NODATA
Enabled: <yes/no>
Next: <datasource check / re-enable instruction>
Full investigation:
Check: <job> (<target>)
Type: <http|ping|dns|tcp|traceroute>
Status: FAILING
Success rate: <rate>% (window: <from> – <to>)
[Timeline graph]
Failure classification: <Target down | Regional/CDN | SSL/TLS | DNS | Timeout | Content/assertion | Private probe infra | Rate limiting>
Affected probes: <count>/<total>
- <probe-name> (<region>): failing since <time>
- <probe-name> (<region>): intermittent
Onset: <time/duration or "unknown">
Diagnosis:
<2-4 sentences describing what the data shows and the most likely cause>
Next actions:
1. <action>
2. <action>
3. <action>
Use minimal formatting. Avoid excessive bold text. Trust the user to prioritize.
gcx synthetic-monitoring checks status returns no rows: check ID may be wrong — list all checks and confirmgcx synthetic-monitoring probes list fails: skip geographic mapping; classify probes by name where possiblegcx metrics query fails with datasource error: note it, skip PromQL steps, classify using timeline data only--from now-6h --to now before concluding NODATA