Terminal
Terminal/Skill
Diagnoses and tunes an existing Runpod Serverless endpoint: stuck queues, failing workers, scaling, GPU pins, and queue cancels, changing only approved settings.
By RunPod
Works with
npx skills add runpod/runpod-plugins-official --skill endpoint-opsYou operate a Serverless endpoint that already exists. Everything starts by reading its current state (the endpoint config, its health, its workers, its logs) before you say anything or change anything. Diagnosis is always read-only. You change a setting only when the user asked for a change, and then only the exact setting agreed.
One property of the Runpod update path drives the central rule here: an endpoint update is a PATCH. Only the fields present in the body change and omitted fields are left untouched, so you send exactly the setting you were asked to change and nothing else. The one exception is the GPU selection: gpu.pools and gpu.excludedTypes are a single selection, so a PATCH that sends pools without excludedTypes clears the exclusions. When you touch either, send both. What the PATCH does not give you is proof: its own response is not a read. Read the config first for the before value, and read it back afterward for the after value.
Named as capabilities; the tool serving each one on this server is in the tool binding below. This skill reads before it mutates, and mutates only what the user authorized.
excludedTypes selection and gpu.count; set-endpoint-gpus builds it and preserves the other settings.Resolve the endpoint (list endpoints and match by name if needed), read its config, then read health and workers before forming a conclusion. Classify against the symptom:
IN_QUEUE. Read health and the worker list. If max workers is 0 or all workers are throttled/unhealthy, the queue has nothing to run on. An endpoint idle for 3 days has workers.max lowered to 2 and after 7 days to 0, and it stays there until raised, so a long-idle endpoint "stuck" on its first job is usually this. Separate the causes: worker limit (max-workers too low or lowered by idleness, or min 0 with a cold scaler) vs capacity (workers can't acquire the selected GPU: throttled above 0; cross-check that GPU type's availability with product:["SERVERLESS"], and its CUDA versions with capacity when the endpoint pins a CUDA floor) vs crash loop (workers cycling: read their logs with source=system; a container that exits before the handler runs can still report RUNNING or INITIALIZING while the job stays queued, so the logs settle it, not the worker status). Name which one, quoting the health/worker numbers you read.HEALTH_CHECK_PATH (default /ping) on PORT_HEALTH (default 80), which is separate from the serving PORT (default 80). Workers stuck un-ready while requests arrive usually means nothing answers that path on that port; the endpoint read gives you the exact health URL being polled. The fix is a redeploy that exposes the port and serves the health path (see serverless-deploy), not a scaler change.End phase 1 with the diagnosis and the recommended change, but do not apply it unless the prompt authorized a change. If the ask was only "why / what's wrong", you are done here.
Advance only when the prompt asks to change a named endpoint's setting.
workers.idleTimeout, and the account's worker quota can refuse a higher workers.max). Change only the fields the user agreed; state the cost of a higher min-worker.gpu.excludedTypes (at least one type must remain), with the CUDA floor if given. set-endpoint-gpus builds that exclusion list for you and re-sends the endpoint's other settings unchanged; update-endpoint can carry the same gpu object, but there pools and excludedTypes are one selection, so send both together. allowedCudaVersions and minCudaVersion are mutually exclusive (400 if both are sent). Changing the CUDA constraint or the pool does not evict running workers; bounce workers.max to 0 and back when the change must apply now. Read the endpoint back after a pin and confirm the scaling settings survived.get-job-status read. The cancel call's own response is not verification (an unread write is an unverified write); the final answer reports the re-read state. If the re-read shows the job already reached a terminal state before your cancel, say exactly that; it is a complete, honest outcome.get-endpoint read AFTER it. The before/after values the final answer quotes come from reads (1) and (3). The update call's own response is not a read; never claim a read-back you did not perform.run/runsync "test probe" while diagnosing: a submitted job bills compute and changes the queue, so it is a mutation, and on a broken endpoint it burns money to reproduce what the worker states already show. Diagnose from the endpoint/health/worker/log reads only; a recommended fix is presented, not applied, unless the prompt authorizes it.gpu.pools sent without excludedTypes, which clears the exclusions). Look at the other write paths: a GPU pin that re-sent the whole config, a template save, a redeploy, or a concurrent change by someone else. Re-read before re-applying anything.IN_QUEUE with ready workers and nothing in progress → do not wait it out: read the worker logs (source=system); a container that exits before the handler runs looks exactly like this and points at a broken image, not at the scaler.serverless-deploy), not a scaler tweak.get-job-status answers 404 for a job id that was valid → the job's TTL (24 h from submission by default) expired and removed it; it is not a wrong id.set-endpoint-gpus is missing from the session → make the same selection through update-endpoint, sending gpu.pools and gpu.excludedTypes together.| Capability | Tool |
|---|---|
| List / inspect endpoints | list-endpoints, get-endpoint |
| Endpoint health | endpoint-health |
| List workers (per-worker detail) | list-endpoint-workers |
| Read a worker's logs | stream-worker-logs (source=system for the container lifecycle) |
| GPU stock per data center | list-gpu-types / get-gpu-type (include:["AVAILABILITY"], product:["SERVERLESS"]) |
| CUDA versions with capacity, per GPU type | the same read (cudaVersions); get-capacity for the matrix view |
| Update endpoint settings | update-endpoint (PATCH; send gpu.pools and gpu.excludedTypes together) |
| Pin GPUs / set GPU count | set-endpoint-gpus |
| Release history | list-endpoint-releases |
| Cancel one job | cancel-job |
| Purge the queue | purge-endpoint-queue |
| Submit / poll a job | run-endpoint, get-job-status, runsync-endpoint |
If a tool named here is missing from the session's tool list, say so and use the nearest read instead of inventing a result.
The cross-journey answer contract (definite facts from reads, commit-don't-hedge, mutations bind to this conversation's creations, a standalone final message, honest failure, granted tools only) is defined in the runpod-mcp skill and applies to every reply from this journey. If the runpod-mcp skill has not been loaded in this conversation, load it now, before the next tool call: its rules on which resources you may change, when a request is the go, and when to stop for the user are not repeated here.