Terminal
Terminal/Skill
Lets an agent operate terminal apps, browsers, and desktop apps to run QA, record demos, or capture proof.
By Factory
Works with
npx skills add factory-ai/factory-plugins --skill droid-controlAutomate terminals, browsers, and desktop apps. Route by the user's requested method first, then load only the mechanics and evidence stages needed.
wait / wait-idle. Never substitute fixtures or mocked data.tctl owns recorded terminal sessions. It wraps asciinema rec around the PTY; browser and desktop drivers own their separate lifecycles. Never call tuistory launch directly. Resolve TCTL to an absolute path only for terminal workflows or worker handoffs./tmp/tctl-sessions/). At the start of every workflow, generate a run ID (RUN_ID=$(date +%s)-$$ or similar) and use it as a prefix for all session names and a scoped temp directory for all output files:
RUN_ID="$(date +%s)-$$"
RUN_DIR="$(mktemp -d /tmp/droid-run-${RUN_ID}-XXXXXX)"
# Session names: -s ${RUN_ID}-before, -s ${RUN_ID}-after
# Output paths: ${RUN_DIR}/before.cast, ${RUN_DIR}/after.cast
Never use bare session names like -s demo, -s before, -s after — they will collide with concurrent runs.
Separate names and paths do not isolate shared desktop focus or keyboard input. Keep one controller for a visible desktop.Three independent lookups. Do all three, then load the union of skills they produce.
| Target | Load these skills |
|---|---|
| User explicitly requests cua-only, native GUI input, or desktop control (including Electron) | desktop-use; method constraints override the defaults below |
Droid CLI (droid-dev, droid exec) | terminal-use + droid-cli |
| Other terminal TUI | terminal-use |
| Web page or Electron app | browser-use |
| Native desktop GUI app | desktop-use |
| Raw terminal byte sequences | terminal-use + pty-capture |
terminal-use selects the terminal backend behind ${DROID_PLUGIN_ROOT}/bin/tctl: tuistory by default, and it loads true-input when real terminal rendering or keyboard-encoding evidence is needed. Desktop-use includes compositor-specific guidance; inspect live Cua capabilities rather than assuming all Linux targets are X11-only or abandoning the user's chosen method.
Every workflow passes through stages. Load the atoms for each stage you'll use.
| Stage | Skill | When to load |
|---|---|---|
| Capture | capture | Recording, scripted multi-step evidence, or a demo/QA deliverable; ordinary desktop operation uses the driver's observe/verify loop |
| Compose | compose | When the deliverable is a produced artifact (video, annotated screenshots, comparison image) |
| Verify | verify | Formal proof, demo, or QA deliverable; every action still needs verification even without this stage |
Only relevant when compose is loaded.
| Artifact need | Also load |
|---|---|
| Showcase polish (window chrome, branded frame, cinematic background) | showcase |
| Effects and keystroke overlays | (compose handles this — they're fields in the Remotion props JSON) |
Command (intent + commitments)
→ Target route (load driver atoms)
→ Capture (record / screenshot / byte-capture)
→ Compose (assemble deliverable, if needed)
→ Verify (check against commitments)
→ Report
Commands declare what to produce. Atoms own how.
Default: single. One clip showing the target/final state. Pick this unless the deliverable is fundamentally a comparison.
| Case | Layout |
|---|---|
| Brand-new feature (no meaningful prior state) | single |
| Bug fix, single-clip proof of the working path | single |
| Walkthrough / tutorial / readme hero | single |
| Regression proof (broken vs fixed) | side-by-side |
| Behavior-preserving refactor (visual parity is the point) | side-by-side |
| User explicitly asks for a comparison | side-by-side |
Do not synthesize a "before" state to justify side-by-side. If there is no real baseline, use single.
Keep short interactive desktop tasks in the parent: it owns observations, input, user permission waits, and cleanup. Delegate independent capture environments or rendering, not individual screenshots interleaved with another controller's input.
| Task | Delegate? | Why |
|---|---|---|
| Interactive shared desktop | NO — parent | One controller owns focus, snapshots, input, permission waits, and cleanup |
| Capture clip in an isolated terminal/browser environment | YES | Worker owns the complete interaction and recording lifecycle |
| Capture both clips (comparison layout) | YES, only with independent environments | Worktrees and session labels alone do not isolate a desktop |
| Remotion render | YES | Needs only props JSON, clip paths, output path. Runs render-showcase.sh (handles .cast conversion, per-render staging, fidelity profiles, longest-clip duration, cleanup) |
| Planning, interaction scripting | NO — parent | Requires PR context and editorial judgment |
| Layout and prop construction | NO — parent | Requires editorial decisions about effects, timing, labels |
| Verification | NO — parent | Requires commitment context |
| Single ffprobe / file-existence check | NO — inline | Too trivial for subagent overhead |
Step 0: Resolve paths and generate a run ID. Workers don't inherit ${DROID_PLUGIN_ROOT}. Resolve once, paste everywhere:
TCTL="$(realpath "${DROID_PLUGIN_ROOT}/bin/tctl")"
RENDER="$(realpath "${DROID_PLUGIN_ROOT}/scripts/render-showcase.sh")"
RUN_ID="$(date +%s)-$$"
RUN_DIR="$(mktemp -d /tmp/droid-run-${RUN_ID}-XXXXXX)"
Use ${RUN_DIR} for all output files (recordings, props, rendered video). Use ${RUN_ID}- as a prefix for all session names. Never use bare names like -s before or hardcoded paths like /tmp/before.cast.
Give workers exact commands with the resolved absolute paths — not abstract instructions, not tuistory, not ${DROID_PLUGIN_ROOT}. The parent does the thinking; the worker executes:
Task prompt for a capture worker:
"Run these commands in order. Report the output file path and any errors.
1. /abs/path/to/bin/tctl launch "droid-dev" -s 1712345678-42-before --backend tuistory \
--repo-root /abs/path/to/baseline/worktree \
--cols 120 --rows 36 --record /tmp/droid-run-1712345678-42-xxxx/before.cast \
--env FORCE_COLOR=3 --env COLORTERM=truecolor
2. /abs/path/to/bin/tctl -s 1712345678-42-before wait ">" --timeout 15000
3. /abs/path/to/bin/tctl -s 1712345678-42-before type "hello world"
4. /abs/path/to/bin/tctl -s 1712345678-42-before press enter
5. /abs/path/to/bin/tctl -s 1712345678-42-before wait-idle
6. /abs/path/to/bin/tctl -s 1712345678-42-before close"
Task prompt for a Remotion render worker:
"Run this command. Report the output file path and any errors.
/abs/path/to/scripts/render-showcase.sh \
--props /tmp/droid-run-1712345678-42-xxxx/showcase-props.json \
--output /tmp/droid-run-1712345678-42-xxxx/demo.mp4 \
/tmp/droid-run-1712345678-42-xxxx/before.cast /tmp/droid-run-1712345678-42-xxxx/after.cast"
Only applicable when the Layout default table selects side-by-side and the capture environments are independent. Serialize shared-desktop captures. For a single interactive desktop task, keep control in the parent.
For before/after comparison demos, launch both capture workers simultaneously:
1. Parent constructs the interaction script (identical for both branches)
2. Launch worker A: capture the baseline/reference branch with `--repo-root` set to that worktree
3. Launch worker B: capture the candidate/change branch with `--repo-root` set to that worktree
4. Wait for both to complete (TaskOutput)
5. Collect .cast paths from results
6. Continue to compose
Terminal drivers use the unified tctl wrapper. Browser-use and desktop-use have their own CLIs (agent-browser, cua-driver) and do not use tctl.
Drivers can be combined in one workflow — e.g., tctl for a CLI and agent-browser for a web UI it interacts with.
Deterministic recipe for reproducing degraded transcript tails in the droid CLI — stranded live tool rows and queued steering messages — without waiting for a slow model turn. The trick: a slow PreToolUse hook pins a tool in its executing state for as long as you need.
Scratch project. Create a throwaway directory (never a real repo — the hook fires on every matching tool call) with a project-local hook that sleeps:
SCRATCH="$(mktemp -d /tmp/degraded-tail-XXXXXX)"
mkdir -p "$SCRATCH/.factory"
cat > "$SCRATCH/.factory/settings.json" <<'JSON'
{
"hooks": {
"PreToolUse": [
{
"matcher": "TodoWrite",
"hooks": [{ "type": "command", "command": "sleep 120" }]
}
]
}
}
JSON
Pick a sleep long enough to interact mid-hook (60–180s) and a matcher for a tool the prompt will reliably trigger (TodoWrite fires on any multi-step ask).
Launch with --cwd pointed at the scratch project — --repo-root stays on your dev worktree so droid-dev provenance still records the code under test:
$TCTL launch "droid-dev" -s ${RUN_ID}-tail --cwd "$SCRATCH" \
--repo-root /abs/path/to/dev/worktree --record ${RUN_DIR}/tail.cast
Trigger the hook, then degrade the tail while the tool row shows executing:
press escape) — strands the live tool row: it never resolves to a completed/canceled state in the transcript tail.type "..." + press enter) — the steering message queues behind the executing tool instead of interleaving.Gotcha: dev-scope hook settings can silently disable project hooks. If the tool completes instantly, run /hooks in the session and check the "Hooks enabled" toggle before debugging the hook config itself.
| Stage | Platform | Required | Optional |
|---|---|---|---|
| terminal-use (tuistory) | All | tuistory, asciinema, agg | tmux |
| true-input | Linux/Wayland | cage, wtype, Wayland terminal, /dev/dri/* | grim, wf-recorder |
| true-input | Windows (KVM) | libvirt, qemu, KVM VM with SPICE + SSH, DROID_VM_* env vars | virt-manager |
| true-input | macOS (QEMU) | qemu, socat, macOS VM with SSH, DROID_MAC_* env vars | — |
| browser-use | All | agent-browser (+ agent-browser install) | — |
| desktop-use | All | cua-driver in the intended graphical session; approved OS permissions | Documentation is bundled; no separate skill install |
| compose | All | ffmpeg, ffprobe, agg | — |
| showcase | All | Node.js (>= 18), Chrome/Chromium | — |
# tuistory driver + recording
npm install -g tuistory # virtual PTY driver
pip install asciinema # terminal recording (tctl wraps this)
cargo install --git https://github.com/asciinema/agg # .cast -> .gif converter (compose needs this)
# true-input driver (Linux/Wayland)
sudo apt-get install -y cage wtype # required: headless compositor + keystroke injection
sudo apt-get install -y grim wf-recorder # optional: screenshots + video recording
# agent-browser driver
agent-browser install # one-time: downloads bundled Chromium
# desktop-use: follow its setup instructions only if the binary
# is missing and installation is approved. No separate skill install.
# compose + showcase (video rendering)
sudo apt-get install -y ffmpeg # video processing (includes ffprobe)
cd ${DROID_PLUGIN_ROOT}/remotion && npm install # Remotion dependencies
# Chrome or Chromium must be installed for Remotion rendering