Terminal
Terminal/Skill
Sets up Temporal metrics collection and writes queries and alerts for Datadog, Grafana, Prometheus, and similar tools.
By Temporal
Works with
npx skills add temporalio/skill-temporal-observability --skill temporal-observabilityThis skill helps users set up metrics collection, write monitoring queries, configure alerts, and diagnose performance issues using Temporal metrics. It operates in three modes:
You produce configs, queries, and code snippets. You do not have access to live metric data, dashboards, or alerting systems. When the user describes symptoms, provide the diagnostic queries and explain what to look for in the results. This skill owns the metrics collection and query layer. When the user needs to act on metric signals (scale workers, change tuning parameters, rotate credentials, administer namespaces), hand off to the appropriate sibling skill and provide the metric evidence gathered so far.
Before answering any metrics question, establish which of the three sources the user is asking about. This determines which reference files apply, which query functions are valid, and which auth/scrape patterns to use:
temporal_cloud_v1_* only) — from metrics.temporal.io, API key auth. Never use rate(). Never recommend deprecated temporal_cloud_v0_* metrics.temporal_* without cloud) — emitted by Workers/Clients, scraped from app endpoints. rate() is correct here.service_requests, persistence_latency, etc.) — emitted by the Temporal Service itself. rate() is correct here.If the source is ambiguous, ask before writing queries.
Cloud, self-hosted, or dev server. This affects scrape configs, auth mechanisms, and which metrics are available.
Prometheus, Datadog, Grafana Cloud, New Relic, ClickStack, or OTel Collector. This affects config format, query language, and available features.
Histogram units differ: milliseconds for Core-based SDKs (TypeScript, Python, .NET) vs seconds for Go and Java. Getting this wrong produces thresholds that are off by 1000x.
Most real questions require 2-3 reference files:
openmetrics-endpoint.md (auth + scrape) + integrations.md (platform config) + service-health-monitoring.md (what to alert on) + worker-health-monitoring.md (worker fleet).ops-diagnostics-reference.md (USE framework, decision trees, metric mappings) + worker-health-monitoring.md (thresholds, queries) + service-health-monitoring.md (failure rates, limits) + troubleshooting.md (per-bottleneck root causes). For worked examples: diagnosis-examples.md. If they need to act on findings: hand off to skill-temporal-workertuning.migration-v0-to-v1.md (mapping table + auth changes). Only relevant for users already on v0 — never suggest v0 to new users. v0 is deprecated and unavailable to new accounts.service-health-monitoring.md + worker-health-monitoring.md for the essential observation set.When the user describes a symptom (slow workflows, growing backlog, activity failures, high latency), follow this systematic process. Never diagnose without metrics data.
Map the user's description to a category:
| User says... | Category |
|---|---|
| "workflows slow to start", "backlog growing", "schedule-to-start high", "tasks queuing up" | Throughput/Backlog |
| "activities failing", "workflow failures spiking", "retries not working", "error rate high" | Failure/Error |
| "workflows taking too long end-to-end", "execution slow", "replay latency" | Latency |
| "resource exhausted errors", "throttling", "hitting limits" | Resource Pressure |
| "can't connect", "auth error", "certificate expired" | Not this skill — hand off to skill-temporal-cloud |
| "need to tune workers", "autoscaling", "slot sizing" | Not this skill — hand off to skill-temporal-workertuning |
If the symptom doesn't clearly map, ask what behavior they're seeing, when it started, and what changed.
Use context clues to determine:
See references/ops-diagnostics-reference.md § Environment Detection for clue table. If confidence is below 95%, ask.
You do not have access to live metrics. Establish how to get values:
Always provide queries in the user's platform syntax. See references/ops-diagnostics-reference.md § Query Examples by Platform for Datadog and New Relic translations. For PromQL, see references/worker-health-monitoring.md and references/service-health-monitoring.md.
Do not proceed with diagnosis without metrics data.
For each Temporal resource relevant to the symptom, check all three USE dimensions:
Consult references/ops-diagnostics-reference.md:
Cross-reference thresholds and detailed metric descriptions from:
references/worker-health-monitoring.md — backlog, sync match, poll success, task slots, sticky cachereferences/service-health-monitoring.md — failure rates, error rates, limits, replicationreferences/troubleshooting.md — per-bottleneck root causes and diagnostic stepsStructure every diagnosis response using this format:
What's Happening: Plain-language explanation of the mechanics. Connect the user's metrics to the behavior they're seeing. Explain how Temporal's internals work in this situation so the user builds a mental model.
USE Analysis:
| Resource | Utilization | Saturation | Errors |
|---|---|---|---|
| (resource) | (metric: value + interpretation) | (metric: value) | (metric: value) |
Why: Root cause grounded in metric evidence. Explain the causal chain ("X is high because Y, caused by Z"). If multiple factors interact, show how.
Fix: Ordered list, most impactful first. For each action, explain why it helps in one sentence.
skill-temporal-workertuningskill-temporal-opsskill-temporal-developerMonitor After: Which metrics to watch, target values, and expected timeline for improvement.
If Not Resolved: What persistence of the issue would indicate, and next diagnostic step or skill handoff.
For fully worked examples of this protocol in action, see references/diagnosis-examples.md.
| Pitfall | Correct Behavior |
|---|---|
Applying rate() to temporal_cloud_v1_* metrics in queries | v1 metrics are pre-computed rates — use directly. rate() is only valid for SDK and self-hosted metrics. |
| Confusing SDK metric units across SDKs | Go/Java use seconds, TypeScript/Python/.NET use milliseconds. A 200ms threshold is 0.2 in Go/Java and 200 in Core SDKs. |
Assuming approximate_backlog_count = 0 means no backlog | This metric resets to zero on idle queues. Confirm with schedule-to-start latency. |
| Jumping to "add more workers" without checking slots | First check if existing workers have available slots. Slot depletion with low host CPU = configuration issue, not capacity. |
| Diagnosing without knowing the environment | Cloud vs self-hosted changes which metrics exist. Always establish environment first. |
| Ignoring the failure conversion rate | High activity failures with low workflow failures = healthy error handling. Always compute the ratio in failure scenarios. |
| Prescribing without explaining | Users who understand the mechanics make better decisions. Explain before recommending. |
metrics.temporal.io, API key auth, 30s recommended scrape interval, 30k datapoints per scrape, 180 requests/hour rate limit, ~3-minute data latency.rate(), increase(), irate(), or histogram_quantile().temporal_cloud_v1_approximate_backlog_count is approximate — can overcount and resets to zero on idle queues.worker_version label is deprecated — use temporal_worker_deployment_name and temporal_worker_build_id.| User intent | Action |
|---|---|
| Wants to know which Cloud metrics exist, their types, or labels | Consult references/cloud-metrics-catalog.md for the complete temporal_cloud_v1_* catalog |
| Needs to set up or configure the OpenMetrics endpoint (auth, scrape config, rate limits) | Consult references/openmetrics-endpoint.md |
| Wants to integrate metrics with Datadog, Grafana Cloud, ClickStack, New Relic, Prometheus, or OTel Collector | Consult references/integrations.md |
| Needs to configure SDK metrics scrape endpoints or understand SDK metric units | Consult references/sdk-metrics.md |
| Setting up monitoring for a self-hosted Temporal Service | Consult references/self-hosted-monitoring.md |
| Wants to monitor Temporal Cloud service health (latency, error rates, failures, limits) | Consult references/service-health-monitoring.md |
| Needs to monitor Worker fleet health (backlog, greedy workers, misconfigured workers, sticky cache) | Consult references/worker-health-monitoring.md |
| Migrating from PromQL/v0 to OpenMetrics/v1 | Consult references/migration-v0-to-v1.md |
| Troubleshooting performance bottlenecks using metrics | Consult references/troubleshooting.md |
| Describing a symptom that needs systematic diagnosis (slow workflows, backlog, failures, latency) | Follow the Diagnosis Protocol above. Consult references/ops-diagnostics-reference.md for USE framework and decision trees |
| Wants a worked example of a diagnostic workflow | Consult references/diagnosis-examples.md |
| Wants to write Workflow/Activity/Worker SDK code (not metrics config) | Defer to skill-temporal-developer |
| Wants to tune Worker performance (slot suppliers, tuners, cache sizing) | Defer to skill-temporal-workertuning -- this skill owns collection/query layer only |
| Wants to administer or diagnose Temporal Cloud/self-hosted environments | Defer to skill-temporal-ops -- this skill provides metric queries for diagnosis |
| Wants Cloud connectivity, auth, or namespace config (not metrics) | Defer to skill-temporal-cloud |
temporal_cloud_v0_* metrics and the PromQL endpoint are deprecated and no longer available to new users. Never recommend v0 metrics for any purpose. Do not include v0 metric names in configs, queries, dashboards, or examples. Always use the temporal_cloud_v1_* equivalents.references/migration-v0-to-v1.md and guide them to the v1 replacements.temporal_cloud_v0_frontend_service_request_count), respond with the v1 equivalent and note that v0 is deprecated and being shut down.temporal_cloud_v0_frontend_service_request_count → temporal_cloud_v1_service_request_count — note the dropped frontend_). When helping users migrate, consult the mapping table in references/migration-v0-to-v1.md; do not attempt to guess v1 names by swapping the version prefix.curl -s "http://localhost:9090/api/v1/query?query=temporal_cloud_v1_service_request_count" should return results with recent timestamps.temporal_cloud_v1 — do matching metrics appear with recent values?").<account-id>.tmprl.cloud/prometheus).