Terminal
Terminal/Skill
Gates TAO job launches on Docker, SLURM, Kubernetes or Brev: checks credentials, image, data and compute, then submits, monitors and cancels.
By NVIDIA
Works with
npx skills add nvidia/skills --skill tao-launch-workflowStandalone install? If this session was not initialized by the TAO skill bank plugin, run the
tao-setupskill first (host preflight, credentials, cross-skill discovery).
Use this skill before launching any TAO workflow or model action.
Run the platform helper, ask for platform and monitoring preferences, then run the selected platform detail helper before asking for credentials.
This gate is model-agnostic. Apply it to every TAO model, data action, and application workflow before launching side-effecting work.
Do not create runner scripts, launch scripts, compatibility shims, workspace folders, state files, logs, or dependency-install side effects until the launch preflight passes.
Preflight passes only after all of these are true:
image=<override>.If any item is missing, ask for the missing input and stop before generating artifacts. This applies to AutoML, normal train/eval/infer/export/TRT, and DEFT/application workflows.
When preflight work clears a blocker, keep track of the original user request. After the fix, rerun the relevant preflight and continue toward that request; do not stop at "blocker fixed" unless the user explicitly asked only for the repair.
Once the launch gate passes and the producing model/data skill has authored the
spec-bundle (schema: tao-artifacts), execution is exactly four verbs. Every
platform skill implements them over its native CLI — the bank ships five
(tao-run-on-docker, -slurm, -kubernetes, -brev, -virtualenv), and any
externally installed platform skill joins the same contract (§ External
platform skills); nothing else is platform-specific.
$BANK = ${TAO_SKILL_BANK_PATH}.
tao-data-io only on a
frame mismatch (remote URIs, cross-host paths, PTM fetches, tier-C result
uploads). Then lint the assembled command with redact_secrets.py lint and
open the record and launch, in that order:
JOB_ID=$("$BANK/scripts/tao_job_record.py" open --platform <p> --image <img> \
--network-arch <arch> --action <action> --storage-tier <A|B|C> --results-root <root>)
# <native launch, naming the backend object after $JOB_ID>
"$BANK/scripts/tao_job_record.py" mark "$JOB_ID" --state RUNNING --backend-ref <ref>
PENDING RUNNING COMPLETE ERROR CANCELED UNKNOWN; the native sub-state
(ImagePullBackOff, PENDING-resources, slurm COMPLETING) rides in the
transition message. Never read "what's running" from records — poll the backend.mark <id> --state CANCELED.Record-then-launch is the ordering invariant. open mints the id and binds
results_dir before any launch, and the id it returns is the only handle the
launch can use — a submit that skipped the gate or the open has no id, so it
cannot launch. This is what keeps a run recoverable across a context break:
results_dir is recorded before the backend object (which K8s TTL or docker
--rm may later delete) ever exists.
When the producing spec-bundle declares execution, preserve it as model-owned
action semantics across every application that reuses that model skill. The
selected platform consumes the lifecycle; an application must not copy its
commands into a private launcher. Platform-independent pre/post commands,
runtime attestations, helper dependencies, distributed intent, and completion
evidence belong in the producer's spec-bundle. Scheduler syntax, mounts,
secrets, timeouts, ranks, and child-exit preservation remain platform-owned.
No registry, no interface file: a platform skill declares the contract by
documenting the four verbs, and you verify by reading before first use. A
skill with only native primitives may be used by inferring the mapping
(bank invariants still bind; the mapping goes in the launch review; persist
what worked). Rules and the no-equivalent hard floor:
references/external-platforms.md.
When status reaches ERROR, read the log tail and classify before any
retry — infrastructure faults are retriable (new record, --retry-of, up
to 10), program faults never are. Full criteria, the two judgment calls
(device-side asserts, downstream tracebacks), and the post-turn poller
rules: references/failure-analysis-retry.md.
After the user confirms what they want to do, ask which execution platform
should run it. Discover the choices from the platform skills installed in this
session — you already see them by name and description (tao-run-on-docker,
-slurm, -kubernetes, -brev, -virtualenv, plus any externally installed one such as the
official brev-cli skill). There is no central platform registry to read. If
your runtime surfaces only the core router skills (e.g. Codex), list the bank's
platform skills by reading skills/platform/tao-run-on-*/SKILL.md frontmatter
(name + one-line description) under ${TAO_SKILL_BANK_PATH}.
Then ask:
Use long_running_enabled=true and status_interval_minutes=5 when the user
accepts the defaults.
When monitoring is enabled, do not send a final summary just because several
polls have elapsed or the job is still PENDING. Keep the turn attached and
emit status every status_interval_minutes until a terminal state or explicit
user stop/detach request. If the runtime environment cannot keep the chat turn
open, say that clearly and leave a durable watcher/log path; do not imply that
chat updates will continue after the turn ends.
Final-answer rule: a final response ends chat-side monitoring. While
long_running_enabled=true and any launched job is non-terminal, status
messages must be sent as in-progress updates and the agent must continue
polling. Only send a final response when the workflow reaches terminal state,
the user explicitly asks to detach/stop monitoring, or the runtime genuinely
cannot keep the turn open; in that last case, say it is a runtime limitation
and provide the exact durable status command/log path.
When intake inputs are missing, ask with the exact prompt shape in
references/intake-prompts.md (one consolidated ask, concrete examples,
no invented defaults).
After model ownership resolution, inspect the selected model's
references/skill_info.yaml. If it declares backend_contracts, resolve the
implementation before selecting an image or authoring a spec. An explicit
backend wins when it supports the model/action; otherwise apply the packaged
backend_selection policy and show its rationale. The selected backend
metadata in skill_info.yaml owns its image. The referenced backend contract
owns the entrypoint, configuration schema, data mappings, topology, checkpoint
format, output layout, and status behavior. Never use a legacy top-level image
fallback for a multi-backend frontend, and never treat one backend as a version
of another.
Pass action, backend, and workload hints to the model resolver. When metadata
declares a backend planner, use it. The shared Cosmos frontend, for example,
uses scripts/cosmos_workflow.py plan to generate backend-native TOML and a
launch sequence.
Before creating specs, runner scripts, workspaces, logs, state files, or submitting a job, resolve the image for the selected model/action:
${TAO_SKILL_BANK_PATH:-~/tao-skill-bank}/scripts/resolve_tao_image.py \
--skill-bank ${TAO_SKILL_BANK_PATH:-~/tao-skill-bank} \
--model <network> --action <action> --backend <auto-or-explicit> \
--workload <workload-hint> --format text
If the helper is unavailable, read skills/models/<network>/config.json
directly. Resolve image fields in this order:
backend_contracts.<selected-backend>.container_image, when presentactions.<action>.container_imageactions.<action>.imagecontainer_imageimageShow the exact image and ask:
Container image for <network>/<action>:
default=<resolved image>
Use this image, or provide image=<override>?
If the user accepts, pass the resolved image as the job image. If the user
overrides, require a non-empty image reference and pass that value instead.
Do not silently launch on the default image. This confirmation applies to
training, AutoML recommendations, evaluation, inference, export, TensorRT
engine generation, and application workflows that submit TAO containers.
After the user chooses a platform, get the credential list for only that
platform from the chosen skill itself — its ## Credentials section and, if
present, references/skill_info.yaml (required_credentials, credential_groups,
optional_credentials). The launch preflight (check_tao_launch_preflight.py)
reads that same per-skill skill_info.yaml to enforce the credential gate; a
credential-free platform (e.g. Docker) may ship only prose, in which case rely on
its Preflight section.
Ask only for credentials that platform actually needs, plus model-specific
credentials from the selected model skill. Do not ask for Brev credentials on
SLURM, Kubernetes, or Docker. Do not ask for SLURM credentials on Brev,
Kubernetes, or Docker. Ask S3 credentials only when the selected
platform and the dataset/result URIs require s3:// access.
Credentials may already be present in the process environment or in a
user-approved secret env file such as ~/.tao/secrets.env or
~/.config/tao/.env; source such files only when needed and never print,
grep, cat, paste, or log their contents. Verify only variable presence.
For initial launch intake, ask for required credentials and required credential
groups only. Treat the helper's optional credentials/settings section as
reference material; do not request those values unless their only_when
condition applies, the selected workflow cannot proceed without them, or the
user asks to customize that setting.
When the helper output includes a "Required credential groups" section, satisfy one credential from each group before proceeding. Explain each requested value using the helper's description and "How to get it" text.
For SLURM, user-facing prompts should ask for SSH_KEY_PATH first. Mention
SSH_AUTH_SOCK only if the user says they already use an SSH agent.
If a required CLI/library is missing, say exactly what is missing and why it is needed, then ask before installing. Examples:
aws.After user approval and installation, rerun the same preflight. Do not create runner files or launch jobs between the failed check and the rerun.
Accept dataset inputs in either mode:
custom.train_dataset.annotation_path=<root>/annotations.json and
custom.train_dataset.media_path=<root>.custom.train_dataset.annotation_path=<TRAIN_ANNOTATION_PATH>
and custom.train_dataset.media_path=<TRAIN_MEDIA_PATH>.Ask for dataset examples that match the selected platform:
s3://bucket/path/train and
s3://bucket/path/eval unless the platform profile mounts shared storage./data/tao/<model>/train, or direct spec paths visible inside the planned
container mount.DOCKER_HOST, not paths on the local agent machine.Do not assume "dataset root" is the only acceptable input. When direct spec paths are supplied, validate the exact spec paths rather than appending default filenames.
Run the selected platform's preflight checks before any launch artifact is
created — prefer the packaged helper scripts/check_tao_launch_preflight.py
(--platform <p> --container-image <img> --path <label>=<path> ...). It verifies
credentials, client tools, platform/cluster/object-store access, dataset paths
from the compute frame, GPU/runtime health, and image-architecture fit; treat any
failure as blocking. Never use --skip-platform-access for a real launch.
See references/platform-preflight.md for the full per-platform detail (SLURM
SSH/key setup + resource defaults, docker/remote-docker GPU + bind-mount checks,
Brev/Kubernetes API + object-store checks, annotation content-field checks, and
data staging).
Before any side-effecting launch, show a concise review:
For AutoML, also show the algorithm, metric/direction, recommendation budget,
search parameters, ranges, and generated/default recommendation details as
described in skills/applications/tao-run-automl/SKILL.md. Ask for confirmation after
this review. If the user supplied a time limit, flag any plan that exceeds it
and offer concrete reductions before launch.
Never end a successful launch review with only “nothing was launched.” End
with one direct action prompt, for example: Ready to materialize the sealed plan and submit the job. Reply "launch", "go ahead", or "yes" to proceed.
The next unambiguous affirmative chat message authorizes materialization,
job-record creation, submission, and the previously reviewed monitoring mode;
execute immediately without another intake or confirmation round.
When the model contract declares a structured status path or metric extractor, poll it alongside the native backend. Scheduler/container completion is not a successful training result by itself: require the model's terminal structured success record, collect concrete checkpoint events, and return final train loss plus every epoch validation-complete loss. Do not promote validation heartbeat/batch metrics or a train-loss line to epoch validation loss. If the process fails before its native logger exists, invoke the packaged status finalizer or report the real process exit failure; use raw log parsing only as a fallback.