Terminal
Terminal/Skill
Hosts models on dedicated Together AI GPUs with autoscaling, traffic splits, metrics, and LoRA uploads, and covers moving off the retired v1 endpoints API.
By Together AI
Works with
npx skills add togethercomputer/skills --skill together-dedicated-model-inferenceDedicated model inference (DMI) serves a model on reserved single-tenant GPUs. It bills per minute per running replica (by hardware, not by token), has no hard rate limits, and uses the same inference API as serverless models.
The resource model has six parts: Project → Model → Config → Endpoint → Deployment → Replica.
Inference goes to https://api-inference.together.ai/v1 with the endpoint string
(<project_slug>/<endpoint_name>) as the model parameter. Management goes through the
Together CLI (tg beta ... — install with uv tool install "together[cli]"; tg and
together are interchangeable), the client.beta.* SDK namespaces, or the /v2 REST API at
https://api.together.ai.
The legacy v1 dedicated endpoints API is retired. The old flow (
client.endpoints.createwithmodel=+hardware=, hardware IDs like1x_nvidia_h100_80gb_sxm, thetogether endpointsCLI withoutbeta) no longer accepts new endpoints:POST /v1/endpoints,client.endpoints.create(...), andtg endpoints createreturnendpoints_v1_create_access_disabled(HTTP 403), and a stopped/paused v1 endpoint can't be restarted. Already-running v1 endpoints keep serving until further notice; everything new — and any redeploy of a stopped v1 endpoint — uses the v2 flow this skill describes.
tg beta endpoints / tg beta models (a.k.a. together beta ...) CLI commandsclient.beta.endpoints / client.beta.models SDK namespacestogether-chat-completions for serverless inference and request-shaping questions
(the request shape is identical once the endpoint is up).together-fine-tuning for training the model you'll deploy here.together-dedicated-containers for custom Docker runtimes (Sprocket/Jig).together-gpu-clusters for raw multi-node compute.tg beta models public (or upload custom weights first).deploymentProfiles (they carry quantization
like BF16/FP8, gpuCount, and the exact model+config resource names). tg beta models configs <id> is often empty for catalog architectures — see
models-and-configs.md. deploy auto-picks only when the
model has a single profile.tg beta endpoints deploy <model> --endpoint <name> — creates the
endpoint, attaches a deployment, and routes 100% of traffic in one step. For a public
catalog model with one profile, pass its name (Qwen/Qwen2.5-7B-Instruct); the catalog
ml_/arch_ ID is owned by a platform project and won't resolve as a deploy/ab
positional in your project. If the model has multiple profiles (e.g. BF16 + FP8), a
bare-name deploy errors — pass the chosen profile's full resolved model resource path as
the positional plus its --config (see models-and-configs.md).status.state is DEPLOYMENT_STATE_READY. The CLI get now accepts an
endpoint or deployment ID: tg beta endpoints get dep_... --json | jq -r '.status.state'
is the scripted polling loop; the SDK equivalent is
client.beta.endpoints.deployments.retrieve(dep_id, project_id=..., endpoint_id=...).
First-time provisioning commonly takes up to ~20 minutes.https://api-inference.together.ai/v1 with the endpoint string as model.tg beta endpoints update <dep_id>
(--min/--max-replicas, --scaling-metric/--scaling-target, --traffic-weight); split traffic or
experiment as needed.tg beta endpoints update <dep_id> --min-replicas 0 --max-replicas 0 to stop
billing, or tg beta endpoints rm <ep_id> --force to tear everything down (it scales
deployments to zero itself).inactive_timeout auto-stop was removed — always scale to
zero (min_replicas: 0, max_replicas: 0) or delete when the user is done; a forgotten
deployment bills until someone stops it.READY deployment serves nothing until it's in the endpoint's traffic split. The CLI's
deploy routes automatically; otherwise set a weight with tg beta endpoints update <dep_id> --traffic-weight N (upserts one entry) or replace the split via SDK
endpoints.update(traffic_split=[...]). A routing_error/503 on a READY deployment almost
always means a missing/zero weight.ep_, dep_, cr_, ml_,
abx_, exp_); inference takes the endpoint string <project_slug>/<endpoint_name>.project_id on every method — derive it with client.whoami().project_id.
The SDK also takes models/configs as resource names (projects/{p}/models/{ml},
projects/{p}/configs/{cr}), while the CLI takes bare IDs. Use the config's own projectId
(often a platform project) when building its resource name.min_replicas: 0 with a positive max_replicas is
rejected. 0/0 stops; on a running deployment the floor is otherwise 1.gpu_utilization, inflight_requests,
ttft, ...) — raw Prometheus series names are rejected in scaling policies. Charts live in
the dashboard (https://api.together.ai/endpoints); raw series can be scraped from the
org-scoped Prometheus-compatible metrics endpoint (beta — see api-reference.md).STOPPED), delete it, then
delete the endpoint. The CLI's rm smart-deletes by ID prefix and auto-detaches from the
split. Deleting still requires a stopped deployment, but rm now scales down for you:
rm dep_... on a running deployment sets 0/0 and asks you to retry once STOPPED, and
rm ep_... --force scales the endpoint's deployments to zero itself as part of teardown.model field only echoes the endpoint string — identical for
every deployment. The routing headers distinguish them: x-cluster is the per-deployment
cluster ID and worker_url (inside the x-i-router-log-event header) is the replica pod.
This is the only way to verify a split or A/B empirically. See
traffic-routing.md (Observing routing).READY, then shift traffic gradually with --traffic-weight (and
replica counts), watching the dashboard between steps. Take the old deployment out with
--traffic-weight 0, then scale it down and delete it.client.beta.* SDK surface and together beta CLI are beta: pin a current SDK
release (uv pip install --upgrade together) and expect the surface to evolve.