Terminal
Terminal/Skill
Administers and troubleshoots Temporal Cloud and self-hosted servers with the temporal and tcld CLIs, from namespaces to stuck Workflows.
By Temporal
Works with
npx skills add temporalio/skill-temporal-ops --skill temporal-opsThis skill operates and diagnoses Temporal environments. It has two modes:
It does not teach how to write workflows or activities (use skill-temporal-developer for that), and it does not reproduce exhaustive CLI flag tables — run temporal <cmd> --help for those, and see cli-conventions.md for cross-command CLI conventions. The boundary is: if the user needs to administer or troubleshoot a running Temporal environment, this skill applies.
skill-temporal-developer.temporal <cmd> --help; cross-command CLI conventions → cli-conventions.md.skill-temporal-workertuning.If the conversation drifts into one of these areas, hand off to the relevant sibling skill rather than improvising.
When the user wants to perform an operational task:
tcld) or a self-hosted operation (temporal operator)? Data-plane operations (temporal workflow, temporal batch, etc.) work on both. If the backend is ambiguous, ask before proceeding — do not assume Cloud or self-hosted and do not output environment-specific commands until you know.get, list, describe, count, show) run freely. Anything listed under Destructive operations is proposed to the user first.An operation belongs to this tier if it is irreversible (tcld namespace delete), revokes access for a live identity (tcld apikey delete), moves
production traffic (tcld namespace failover), or fans out to every match
(any --query form). Apply the test to the operation in front of you — this is a
rule, not a list, and a command's absence from any list in this skill does not
place it outside the tier.
For anything in the tier: gather the evidence and propose. Do not run it on your own initiative, and do not run one to find out what it would do. The reference file for each command states its specific blast radius; read that before proposing, not after.
--query form, run
temporal workflow count --query '<query>' with the byte-identical query
first and carry the result into the proposal. A filter with no narrowing
predicate beyond ExecutionStatus="Running" matches every open Execution in
the Namespace.TEMPORAL_* env vars or a
config-file profile, so the target is frequently not visible in the command
text. If the backend or Namespace was inferred from context rather than stated
by the user, say so — a destructive command aimed at the wrong Namespace is the
most common way this goes wrong.--query form — the count from step 1 to the user as a direct
question, and wait for an answer. Once they approve, run it with --yes on the
--query batch forms; that flag is what lets an approved batch finish, since
the interactive prompt needs a terminal and without one the command reports
user denied confirmation and does nothing. --yes belongs in a command the
user approved, never in a retry of one that failed its prompt. Do not substitute
a loop over single-target workflow terminate --workflow-id. Approval covers one
command against one target; it does not carry to the next command, a widened
query, or a second Namespace.get,
describe, or count. A batch job drains asynchronously:
temporal batch describe --job-id <id> shows how far it has gotten and
temporal batch terminate --job-id <id> stops it before it reaches the rest
of its matches.When a reversible sibling reaches the same goal, propose it alongside: cancel
lets Workflow cleanup code run where terminate does not; apikey disable is
reversible where delete is not; accepted-client-ca add appends where set
replaces.
Assume nothing in the environment will stop a destructive command on your behalf. Credential scope, command denylists, and confirmation prompts may or may not be configured, and their possible presence is not a reason to skip any step above — you are the safeguard the user is relying on.
When the user arrives with a failure or anomaly:
Bottom-up diagnosis. Verify the lower layer before blaming the upper one. The layers, from bottom to top:
The full ladder lives in diagnostic-ladder.md.
Always verify the next layer up rather than prescribing a speculative fix. If TLS works, prove auth works before blaming the workflow. If pollers are present, prove the workflow's last event before blaming the worker.
Attach a confidence score (1-10) to every proposed diagnosis:
Name ambiguity explicitly. Errors like context deadline exceeded are not self-describing, and a single code such as RESOURCE_EXHAUSTED can mean more than one condition (account-limit throttling vs. per-Workflow lock contention). Surface that, gather more context, and scope the next step narrowly.
These are skill conventions, not docs-derived facts.
Find the row that matches the user's intent. The reference file contains the commands and procedures.
| Intent | Category | Reference |
|---|---|---|
| Create, get, list, delete a Cloud namespace | Cloud namespace admin | cloud-namespace-admin.md |
| Add/remove region, failover, HA config | Cloud namespace admin | cloud-namespace-admin.md |
| Set retention, tags, codec-server, connectivity rules | Cloud namespace admin | cloud-namespace-admin.md |
| Add or rename Cloud search attributes | Cloud namespace admin | cloud-namespace-admin.md |
| Check current APS / capacity mode | Cloud capacity | cloud-capacity.md |
| Switch On-Demand ↔ Provisioned, set TRUs | Cloud capacity | cloud-capacity.md |
| Understand APS / RPS / OPS limits and throttling | Cloud capacity | cloud-capacity.md |
| Create, disable, enable, delete an API key | Cloud IAM | cloud-iam.md |
| Invite, list, remove users; set roles/permissions | Cloud IAM | cloud-iam.md |
| Manage user groups and service accounts | Cloud IAM | cloud-iam.md |
| Generate mTLS certs, upload CA, set cert filters | Cloud certs | cloud-certs.md |
| Rotate mTLS certificates | Cloud certs | cloud-certs.md |
| Set up Workflow History Export (S3 / GCS) | Cloud namespace admin | cloud-namespace-admin.md |
| Set up PrivateLink / PSC, manage connectivity rules | Cloud connectivity | cloud-connectivity.md |
| Self-hosted cluster health, describe, namespace CRUD | Self-hosted admin | self-hosted-admin.md |
| Self-hosted search attributes, Nexus endpoints | Self-hosted admin | self-hosted-admin.md |
| Check or manage a Cloud Nexus Endpoint's caller-Namespace allowlist; the 1,000-caller default | Cloud namespace admin | cloud-namespace-admin.md#tcld-nexus-endpoint-allowed-namespace |
| Find stuck/hung/unhealthy workflows via list queries | Workflow health | workflow-health.md |
| Task queue poller status, workflow counts | Workflow health | workflow-health.md |
| Cancel, terminate, or reset workflows | Workflow recovery | workflow-stuck.md#recovery-commands |
Bulk / batch operations on workflows (--query) | CLI conventions | cli-conventions.md |
| Schedule CRUD, time-spec, and operations | CLI conventions | cli-conventions.md |
| Complete or fail an activity externally | CLI conventions | cli-conventions.md |
| Cloud Ops API access, rate limits, Go SDK | Cloud Ops API | cloud-ops-api.md |
| View billing, generate billing report, cost attribution | Cloud billing | cloud-billing.md |
| Audit Logs: view, query via API, configure sink (AWS/GCP) | Cloud audit logs | cloud-audit-logs.md |
| Terraform provider: Namespace/User/SA/API Key/Nexus CRUD | Cloud Terraform | cloud-terraform.md |
| Expiry alerts (cert, API key, credit), status page | Cloud notifications | cloud-notifications.md |
| SAML SSO, SCIM provisioning, IdP integration | Cloud SAML/SCIM | cloud-saml-scim.md |
| Migrate self-hosted to Cloud (automated or manual), migrate between Cloud regions | Cloud migration | cloud-migration.md |
| End-to-end ops playbook (setup, rotation, audit, billing, Terraform) | Ops recipes | ops/recipes.md |
Find the row that matches the user's symptom. Start the investigation at the first check, then read the linked reference.
| Symptom | Category | First check | Reference |
|---|---|---|---|
connection refused, cannot reach frontend | Connectivity | nc -zvw10 <host> 7233 | connectivity.md#connection-refused |
no such host, DNS resolution fails | Connectivity | dig +short <host> or nslookup <host> | connectivity.md#dns |
tls: handshake failure, server rejects handshake | Certificates | openssl s_client -connect <host>:7233 -servername <host> </dev/null | certificates.md#handshake-failure |
x509: certificate has expired or not yet valid | Certificates | openssl x509 -enddate -noout -in cert.pem | certificates.md#expired-or-not-yet-valid |
x509: certificate signed by unknown authority | Certificates | openssl verify -CAfile ca.pem client.pem | certificates.md#unknown-authority |
tcld session / auth fails, Cloud role unclear | Authentication | tcld account get | authentication.md#cloud-role-and-permission-model |
UNAUTHENTICATED, API key rejected | Authentication | env | grep -i TEMPORAL_API_KEY, then tcld apikey get --id <apikey_id> | authentication.md#things-to-check-when-unauthenticated-is-returned-with-an-api-key |
namespace not found / wrong namespace string with an API key | Authentication | Confirm Regional Endpoint form <region>.<cloud_provider>.api.temporal.io:7233 | authentication.md#address-form-for-api-key-connections |
RESOURCE_EXHAUSTED gRPC status | Rate limits | Identify which limit fired: throttle metrics on Cloud v1, the resource_exhausted_cause label on v0 / self-hosted | rate-limits.md#identifying-which-limit-was-hit |
| Task queue shows no pollers | Worker health | temporal task-queue describe --task-queue <q> | worker-health.md#what-no-pollers-looks-like |
| Workflow stuck on a pending activity / timer / child / signal | Workflow stuck | temporal workflow describe --workflow-id <id> | workflow-stuck.md#the-primary-inspection-command-temporal-workflow-describe |
NondeterminismError, repeating WorkflowTaskFailed | Non-determinism | Identify the last WorkflowTaskFailed cause in the Event History | non-determinism.md#the-wft-failure-signature-of-non-determinism |
| Replay fails locally but prod workflow was running | Non-determinism | Fetch the history and run the SDK replayer in one test | replay.md#step-2--run-the-sdk-replayer-all-supported-sdks |
| HA failover did not route traffic to failover region | HA failover | tcld namespace get --namespace <ns>.<acct> vs. DNS CNAME | ha-failover.md#start-here-establish-ground-truth |
| Serverless Worker (AWS Lambda) stopped processing work after a Namespace failover | HA failover | Confirm a FailoverNamespace audit event, then compare the new active region against the Lambda ARN on the Worker Deployment Version | ha-failover.md#symptom-serverless-workers-kept-running-in-the-old-region-after-failover |
context deadline exceeded (unknown layer) | Runtime errors | Identify which operation and SDK emitted it | runtime-errors.md#deadline-exceeded |
Workflow is busy / ResourceExhausted on signal/update/query to one Workflow (BusyWorkflow) | Runtime errors | Rule out account-limit throttling, then split temporal_cloud_v1_resource_exhausted_error_count by operation | runtime-errors.md#workflow-lock-contention-busyworkflow |
PAYLOADS_TOO_LARGE, exceeds size limit, payload/gRPC blob size error | Blob size limits | Check whether the issue is payload (2 MB) or gRPC message (4 MB) | blob-size-limits.md |
| Workflow stuck in invisible retry loop (gRPC message too large) | Blob size limits | Check Worker logs for ResourceExhausted, reduce batch size | blob-size-limits.md |
| High schedule-to-start latency, task slot depletion, slow Workflow Tasks | Performance bottlenecks | Check temporal_workflow_task_schedule_to_start_latency P95 | performance-bottlenecks.md |
| High replay latency, cache evictions, deadlock detected | Performance bottlenecks | Check workflow_task_replay_latency and sticky cache metrics | performance-bottlenecks.md |
| Schedule did not fire, missed catchup window | Missed Schedule Actions | Alert on temporal_cloud_v1_schedule_missed_catchup_window_count | schedule-missed.md |
If a symptom does not map to a row, start at diagnostic-ladder.md and work up from whichever layer was last known healthy.
Determine what the user wants to do and whether it targets:
tcld commands (requires tcld login)temporal operator commandstemporal workflow, temporal batch, temporal schedule, etc.If the backend is unambiguous from context — .tmprl.cloud address, tcld command, Cloud namespace format ns.account → Cloud; Kubernetes/Helm, docker-compose, self-hosted config files → self-hosted — proceed without asking. Otherwise, stop and ask: "Are you on Temporal Cloud or self-hosted?" before outputting any environment-specific commands. Do not default to either environment. Once known, save the answer to memory so you don't ask again in future conversations.
Look up the intent in the Operations table above. Read the linked reference file for the exact commands, flags, and expected output. Run the command and interpret the result for the user.
After a mutating operation (create, update, delete, rotate), confirm the new state:
get or describe commandAsk the user for the exact, copy-pasted error text. Do not accept paraphrases — the exact string often encodes the layer (e.g., x509: prefix means TLS/cert layer, RESOURCE_EXHAUSTED: prefix means gRPC rate limit, NondeterminismError means workflow replay layer). Note that RESOURCE_EXHAUSTED alone does not tell you which condition fired — account-limit throttling and per-Workflow lock contention (Workflow is busy, i.e. BusyWorkflow) share the code. Split the resource-exhausted metric by its label (operation on the Cloud v1 family, resource_exhausted_cause on v0 and self-hosted) rather than parsing the free-text message.
Confirm three things before continuing:
The context the investigation needs depends on the category. At minimum:
temporal workflow describe --workflow-id <id> (pending-operation state lives here, not in the Event History alone). Event History via temporal workflow show is the companion view.temporal task-queue describe --task-queue <q>, and temporal worker describe --task-queue <q> for per-worker details.If the user has pasted SDK connection code — even just the address/namespace/auth fields — review it against sdk-snippet-review.md before descending the ladder. Wrong endpoint family, short namespace, or mismatched auth method will make every network-layer probe below look broken when nothing lower actually is.
Skip this step when the user has an established, previously-working config and the symptom is new — the snippet is not the culprit, the environment changed. Otherwise treat snippet validation as Layer 0.
Use diagnostic-ladder.md to pick the right starting layer. As a rule of thumb:
Each layer has a command that proves it healthy and a failure signature that tells you whether the problem lives at that layer or higher.
Prescribe the fix scoped to the root cause. Then verify by re-running the layer's healthy-check command and, if possible, the original user operation. Attach the confidence score to the diagnosis.
If the layer above the fix is still failing, return to step 4 and continue walking upward — the first broken layer is rarely the only one.
temporal) — required for data-plane operations and self-hosted admin. Install: brew install temporal or see Temporal CLI docs.brew install temporal-cloud-cli or see tcld docs. Authenticate with tcld login before use.tcld: create, get, list, delete, failover, add-region, retention, tags, codec-server, HA config, connectivity rules, search attributes, accepted-client-ca, certificate filters, export, and the tcld nexus endpoint allowed-namespace caller allowlist (1,000-caller Access Policy ceiling).tcld namespace capacity update, default limits, throttling, APS management best practices.tcld apikey), users (tcld user), user groups (tcld user-group), service accounts, account operations (tcld account), roles, namespace permissions.tcld generate-certificates, uploading CAs, certificate filters, cert rotation, switching mTLS ↔ API keys.tcld migration commands, 5 phases), manual self-hosted→Cloud (client changes, workflow strategies), within-Cloud region-to-region (HA add-region/failover).saas-api.tmprl.cloud), Go SDK, protobuf compilation, rate limits (160 RPS account, 40 user, 80 SA, 10 concurrent async), API version header, use cases.TEMPORAL_CLOUD_API_KEY), temporalcloud_namespace/temporalcloud_user/temporalcloud_service_account/temporalcloud_apikey/temporalcloud_nexus_endpoint CRUD, import, data sources (regions, namespaces), limitations (API keys not importable, cannot manage Account Owner).noreply@temporal.io sender.urn:auth0:prod-tmprl:ACCOUNT_ID-saml), callback URL (login.tmprl.cloud), IdP configuration steps, support ticket workflow. SCIM: supported vendors, prerequisites (SAML first), 10-minute sync window, group-to-role mapping.temporal operator: cluster health/describe, namespace CRUD, search-attribute create/list/remove, Nexus endpoint CRUD.temporal workflow list with List Filters, temporal workflow describe/show/count, temporal task-queue describe for poller status.temporal CLI conventions: connection/identity (TEMPORAL_* env vars ↔ --address/--namespace/--api-key, --identity), output/formatting (--output, --time-format, payload shorthand), the --query ⇒ batch-job bridge (with temporal batch describe/list/terminate), and schedule time-spec forms. Ends with an operation→command index that routes each data-plane operation to its owner file. Delegates exhaustive flags to temporal <cmd> --help.TEMPORAL_* env vars, common misconfigurations. Run before the diagnostic ladder.tcld namespace accepted-client-ca, Cloud mTLS certificate requirements, rotation and expiry notifications, openssl recipes.UNAUTHENTICATED vs PERMISSION_DENIED, API-key lifecycle (tcld apikey commands, env var propagation, required Regional Endpoint form), mTLS after TLS (certificate filters, identity-to-role mapping), Cloud account-level roles and namespace-level permissions.temporal workflow describe as the primary inspection command, Event History via temporal workflow show, pending activities / child workflows / signals / Nexus operations / Workflow Tasks, WorkflowTaskFailed retry loops, recovery commands (signal, terminate, cancel, reset, pause/unpause).temporal task-queue describe, reachability and versioning, worker-level describe, schedule-to-start latency, worker task slots, sticky execution and sticky cache, worker heartbeating, Cloud namespace-level poller limits, worker log signatures.RESOURCE_EXHAUSTED means (and does not), Cloud APS / RPS / OPS under On-Demand and Provisioned capacity modes, self-hosted frontend.rps / frontend.namespaceRPS dynamic config, identifying which limit fired via the throttle metrics (Cloud v1) or the resource_exhausted_cause label (v0 / self-hosted), and separating account-limit throttling from single-resource exhaustion.tcld namespace get vs. DNS view), clients that did not follow the failover, PrivateLink after failover, failover-not-executing, handover-window errors, platform limits, RPO/RTO semantics, and Serverless Workers (AWS Lambda) not following a failover because compute-provider configuration is region-scoped.operation breakdown, routing for no pollers / INVALID_ARGUMENT / unspecified UNAVAILABLE.TEMPORAL_DEBUG and the deadlock detector, interpreting divergent and successful replays, and the TypeScript-only VS Code extension.temporal_cloud_v1_schedule_missed_catchup_window_count / schedule_missed_catchup_window, investigation via temporal schedule list + temporal schedule describe, DescribeSchedule fields (missedCatchupWindow, overlapSkipped, bufferDropped), default catchup window (one year), root causes, overlap policies (6 values), backfill remediation.If you (the AI) find this skill's explanations are unclear, misleading, or missing important information, draft a GitHub issue body describing the problem encountered and what would have helped, then ask the user to file it at https://github.com/temporalio/skill-temporal-ops/issues/new. Do not file the issue autonomously.