Terminal
Terminal/Skill
Installs Volcano on a Together Kubernetes GPU cluster so multi pod jobs, such as distributed training, start all at once or not at all.
By Together AI
Works with
npx skills add togethercomputer/skills --skill together-volcanoVolcano is a Kubernetes-native batch scheduler. Its key property is gang scheduling: a group of pods is placed all-at-once or not at all, so distributed training never starts with only some workers running. Use it when a job needs N pods to run together or not at all.
Public cookbook: https://docs.together.ai/docs/volcano-on-gpu-clusters. Pair this skill with the together-gpu-clusters skill, which covers creating the cluster and configuring kubectl.
Ready state, with kubectl pointed at it (tg beta clusters get-credentials <cluster_id> --set-default-context).nvidia.com/gpu; the NVIDIA device plugin is preinstalled on Together clusters. Confirm with:kubectl get nodes -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
Pin the version. Do not use master.
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.15.0/installer/volcano-development.yaml
kubectl -n volcano-system wait --for=condition=Available deployment --all --timeout=180s
A default queue is created automatically (kubectl get queue). Helm is an alternative: helm repo add volcano-sh https://volcano-sh.github.io/helm-charts && helm install volcano volcano-sh/volcano -n volcano-system --create-namespace.
A queue caps and weights the resources a set of jobs can use.
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
name: research
spec:
reclaimable: true # give idle capacity back to other queues
weight: 1 # relative share under contention
capability: # hard ceiling for this queue
nvidia.com/gpu: 8
Together provisions a static PersistentVolume named after the cluster's shared volume. Bind a ReadWriteMany PVC to it so every worker shares datasets and checkpoints, then mount it in the job (below).
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shared-pvc
spec:
accessModes: ["ReadWriteMany"]
storageClassName: shared-wekafs # Together default shared storage class
volumeName: <shared-volume-name> # the static PV named after your shared volume
resources:
requests:
storage: 100Gi
The gang guarantee comes from minAvailable on a Volcano Job (vcjob). Set schedulerName: volcano and a queue. The volumes/volumeMounts blocks are optional; drop them if the job needs no shared storage.
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
name: gpu-gang
spec:
minAvailable: 4 # schedule all 4 pods together or none
schedulerName: volcano
queue: research
policies:
- event: PodEvicted
action: RestartJob # restart the whole gang if a pod is evicted
tasks:
- replicas: 4
name: worker
template:
spec:
restartPolicy: OnFailure
containers:
- name: worker
image: nvidia/cuda:12.4.0-base-ubuntu22.04
command: ["bash", "-c", "nvidia-smi -L; sleep 300"]
resources:
limits:
nvidia.com/gpu: 2 # 4 x 2 = 8 GPUs
volumeMounts:
- name: shared
mountPath: /mnt/shared
volumes:
- name: shared
persistentVolumeClaim:
claimName: shared-pvc
kubectl get vcjob <name> should show STATUS Running and RUNNINGS equal to minAvailable.kubectl get podgroup shows the gang; a Running phase means the whole group is placed.Inqueue with zero pods running (all-or-nothing). This is the signal Volcano is working, not a failure. Never "fix" it by lowering minAvailable below the job's real parallelism.schedulerName: volcano must be on the pod template (on a vcjob it goes under spec). Without it, the default scheduler grabs the pods and there is no gang guarantee.capability is rejected. Raise the cap or split the job.schedulerName: volcano and attach a PodGroup. See https://volcano.sh/en/docs/podgroup/.schedulerName: volcano; Kueue admits jobs on the default scheduler. Pick one per workload; do not point a single job at both.