Terminal
Terminal/Skill
Runs mutation testing on a codebase to find surviving mutants and coverage gaps, ranked by risk.
By CircleCI
Works with
npx skills add circleci-public/skills --skill chunk-testing-gapsPerform mutation testing on the current codebase to find gaps in test coverage.
Perform mutation testing on this codebase. A mutation is a small, deliberate change to production code that mimics a realistic bug a human might introduce. A mutant is the resulting modified codebase. If the test suite catches the bug (tests fail), the mutant is killed. If tests still pass, the mutant survives — exposing a gap in test coverage.
The objective is to find as many surviving mutants as possible, prioritised by risk.
This prompt expects to be run in the context of a local clone of a VCS project. If that is not the case, stop and explain why you cannot proceed.
Before any testing, ensure local dependencies are running (docker compose up -d or equivalent). On compose failure, check whether containers from another project are occupying ports and kill them.
Stage 2 runs mutants on chunk sidecars, which needs validation.sidecarImage set in .chunk/config.json. Check it with cat .chunk/config.json. If it is missing, stop and tell the user to run the one-time setup in the chunk-sidecar skill (Step 3) first — booting a fresh unconfigured sidecar per mutant is slow enough to make the whole run impractical.
A mutant is a deliberately broken copy of the code. It must never reach the remote.
Earlier versions of this skill pushed a branch per mutant and opened a draft PR to force a CI run. That flooded repositories with stale draft PRs. If you find yourself reaching for git push or gh pr create in Stage 2, you have taken a wrong turn — re-read Stage 2.
== ↔ !=, < ↔ <=), removed or inverted if branches, swapped switch/case fall-throughs, early returns removed or addedFocus mutations where test coverage is likely weakest. Look for these patterns:
Investigate the codebase and identify candidate mutations. Use subagents to parallelise discovery across packages, modules, or directories — one subagent per logical area.
Each mutation must be assigned a unique sequential number (e.g. MUT-001, MUT-002, ...).
Target density: aim for roughly 1 mutation per 50 lines of production code (e.g. ~1000 mutations for a 50k LOC codebase). This is a guideline, not a hard rule — dense areas may yield more, sparse areas fewer.
Time cap: if no new viable mutation has been identified for 3 minutes, stop discovery and proceed with what you have.
For each candidate, record:
Validation is two passes: a cheap local triage that discards the obvious kills, then a sidecar run for whatever is left. Triage first — it is what keeps the sidecar count, and therefore the cost, proportionate to the number of interesting mutants rather than to the number of candidates.
For each candidate mutation, working in your normal checkout one at a time:
./do, Taskfile, or whatever is configured). If the project has no local tooling, check CI config for what tools are used and attempt to run them locally. Do not install missing tools without prompting.-race -timeout 1m). If tests take longer than 1 minute, treat the mutation as viable and move on.A patch is the output of git diff for that single mutation, captured before the revert.
Collect every local survivor into a single variants file at .chunk/variants.json (gitignored) — a JSON array of objects with id, description, and patch:
[
{
"id": "MUT-001",
"description": "Inverted auth check in verifyToken",
"patch": "diff --git a/pkg/auth/verify.go b/pkg/auth/verify.go\n@@ ... @@\n-\tif !ok {\n+\tif ok {\n"
}
]
The patch value must be the literal git diff text with newlines escaped, and must apply cleanly against the current HEAD.
Then run the whole set in one command:
chunk validate variants .chunk/variants.json --parallel 5
This boots one throwaway sidecar per variant (5 at a time), applies that variant's patch there, runs the project's remote: true validate commands, and deletes the sidecar. Nothing is pushed and no PR is created.
It prints a JSON array, one entry per variant:
killed: true → the test suite ran and failed. Mark as killed. command names the validate command that caught it.killed: false with no error → the tests passed on broken code. Mark as survivor — this is a real coverage gap.error non-empty → the mutant was never assessed. Mark as inconclusive. Causes include: the patch did not apply, the sidecar failed to boot, the sync failed, the validate command was not present on the snapshot, or the command timed out. Either fix the cause and re-run just that variant, or report it as not assessed.killed and error are mutually exclusive, and an inconclusive variant is neither a kill nor a survivor. Never record an inconclusive variant as either. Counting one as killed is the worst outcome available here: it converts a broken run into a clean bill of health.
Raise --parallel for a faster run at higher concurrent cost; lower it if the org hits sidecar limits. Use --name <command> to run a single validate command instead of every remote one. --timeout <seconds> bounds each command for variants whose mutation makes the suite hang; a command's own timeout in .chunk/config.json takes precedence.
Prefer a command that runs the whole suite. Template variables like {{CHANGED_PACKAGES}} expand against your local working tree before the command is sent, and the mutation exists only on the sidecar — so a changed-packages command can skip the very package it is meant to be testing and report a false survivor.
A high kill rate is the result an environmental failure produces, because a snapshot missing the project's tooling fails the same way for every variant. Before reporting results:
stdout/stderr of two or three results and confirm the failures are test assertions failing, not a missing binary, a dependency that was never installed, or a shell error.If the environment is at fault, fix it — usually by rebuilding the snapshot via the chunk-sidecar skill — and re-run. Do not report a coverage verdict from a run you could not sanity-check.
Cleanup: the command deletes its own sidecars, including on Ctrl-C, and sweeps orphans from a previous crashed run before it starts. The sweep only takes sidecars old enough that no live run could still own them, so a second validate variants in another worktree or repo is safe to run alongside the first. You do not need to clean up sidecars by hand. Do delete .chunk/variants.json when the run is done.
Once the variants run has completed, produce a summary table:
| # | Mutation | File | Line | Status |
|---|---|---|---|---|
| MUT-001 | Inverted auth check | pkg/auth/verify.go | 42 | Survivor |
| MUT-002 | Removed nil guard | pkg/api/handler.go | 118 | Killed (sidecar) |
| MUT-003 | Weakened length check | pkg/api/parse.go | 55 | Killed (local) |
| MUT-004 | Removed retry | pkg/api/client.go | 73 | Not assessed (timed out) |
Report any inconclusive variants separately, with the reason they did not run, so the user can see what was left unassessed rather than reading their absence as a pass.
Attempt to determine whether the surviving mutants' code paths are exercised in production. Use any available observability tooling — Honeycomb, Datadog, or other connected MCP servers or CLIs.
Concrete approaches:
If no observability access is available, note this and skip to Stage 4.
Update the summary table with a Production Traffic column indicating: High (clear evidence of regular traffic), Low (occasional or indirect traffic), None found (no evidence), or Unknown (no observability access).
For each survivor, assess overall risk based on:
Assign a risk level: Critical, High, Medium, or Low.
Present the final summary table sorted by risk (highest first):
| # | Mutation | File | Line | Production Traffic | Risk | Rationale |
|---|---|---|---|---|---|---|
| MUT-001 | Inverted auth check | pkg/auth/verify.go | 42 | High | Critical | Auth bypass on a hot path, no test coverage |
| MUT-017 | Hardcoded timeout to 0 | pkg/worker/poll.go | 89 | Low | Medium | Would cause tight loop but only in batch worker |