Terminal
Terminal/Skill
Installs the Kueue job queue controller on a Together Kubernetes GPU cluster so quotas admit jobs only when GPUs are free, pinned to v0.18.3.
By Together AI
Works with
npx skills add togethercomputer/skills --skill together-kueueKueue is a Kubernetes-native job queueing controller. It holds jobs in a queue and admits them only when their quota is free, suspending the rest. Use it to share a fixed GPU pool across teams or workloads without overcommitting it. Unlike Volcano, Kueue does not replace the scheduler; it gates when jobs start by toggling their suspend flag.
Public cookbook: https://docs.together.ai/docs/kueue-on-gpu-clusters. Pair this skill with the together-gpu-clusters skill, which covers creating the cluster and configuring kubectl.
Ready state, with kubectl pointed at it (tg beta clusters get-credentials <cluster_id> --set-default-context).nvidia.com/gpu (NVIDIA device plugin preinstalled on Together clusters).Pin the version. The API group is kueue.x-k8s.io/v1beta2 in v0.18.x; older examples using v1beta1 will not apply.
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.18.3/manifests.yaml
kubectl -n kueue-system wait --for=condition=Available deployment/kueue-controller-manager --timeout=240s
--server-side is required; the bundled CRDs exceed the client-side apply annotation limit. Helm is an alternative: helm install kueue oci://registry.k8s.io/kueue/charts/kueue --version 0.18.3 --namespace kueue-system --create-namespace.
Three objects: ResourceFlavor (a node class), ClusterQueue (the quota pool), and LocalQueue (the namespaced entry point users submit to).
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
name: gpu-flavor
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
name: gpu-cluster-queue
spec:
namespaceSelector: {} # accept jobs from every namespace
resourceGroups:
- coveredResources: ["cpu", "memory", "nvidia.com/gpu"] # MUST list every resource jobs request
flavors:
- name: gpu-flavor # must match the ResourceFlavor
resources:
- name: "cpu"
nominalQuota: 64
- name: "memory"
nominalQuota: 512Gi
- name: "nvidia.com/gpu"
nominalQuota: 8 # admit at most 8 GPUs worth of jobs at once
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
namespace: default
name: gpu-queue
spec:
clusterQueue: gpu-cluster-queue
Together provisions a static PersistentVolume named after the cluster's shared volume. Bind a ReadWriteMany PVC to it so queued jobs share datasets and checkpoints, then mount it in the job (below). Storage is not a quota resource, so it does not affect admission.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shared-pvc
spec:
accessModes: ["ReadWriteMany"]
storageClassName: shared-wekafs # Together default shared storage class
volumeName: <shared-volume-name> # the static PV named after your shared volume
resources:
requests:
storage: 100Gi
Add the queue-name label and start the job suspended. Kueue flips suspend to false on admission. The volumes/volumeMounts blocks are optional; drop them if the job needs no shared storage.
apiVersion: batch/v1
kind: Job
metadata:
name: gpu-job
namespace: default
labels:
kueue.x-k8s.io/queue-name: gpu-queue # route to the LocalQueue
spec:
parallelism: 1
completions: 1
suspend: true # Kueue unsuspends on admission
template:
spec:
restartPolicy: Never
containers:
- name: worker
image: nvidia/cuda:12.4.0-base-ubuntu22.04
command: ["bash", "-c", "nvidia-smi -L; sleep 180"]
resources:
requests:
cpu: "2"
memory: "8Gi"
limits:
nvidia.com/gpu: 6
volumeMounts:
- name: shared
mountPath: /mnt/shared
volumes:
- name: shared
persistentVolumeClaim:
claimName: shared-pvc
kubectl get workloads shows ADMITTED True once the job fits.suspend: true with no pod. This is correct queueing, not a failure. Read the reason:WORKLOAD=$(kubectl get workloads -o name | grep <job-name>)
kubectl get "$WORKLOAD" -o jsonpath='{.status.conditions[?(@.type=="QuotaReserved")].message}'
kueue.x-k8s.io/v1beta2 for v0.18.x.coveredResources MUST include every resource a job requests. A GPU job that also requests CPU and memory is never admitted if the ClusterQueue only covers nvidia.com/gpu. This is the most common failure.flavors[].name must exactly match a ResourceFlavor name, or the ClusterQueue admits nothing.kueue.x-k8s.io/queue-name label must name a LocalQueue in the job's own namespace. A missing or wrong label means the job runs immediately, bypassing quota.schedulerName: volcano. Do not point one job at both.