# craft-experiment-design

Helps a product manager write an A/B test plan with a hypothesis, success metrics, guardrails, and holdout strategy.

Skill by Amplitude. Category: Testing.

## Install

Project (for this project only):

```sh
npx skills add amplitude/builder-skills --skill craft-experiment-design
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

Global (for every project on this machine):

```sh
npx skills add amplitude/builder-skills --skill craft-experiment-design -g
```

Works with: Claude Code, Codex CLI, Cursor, Antigravity, Copilot CLI, opencode.

## Tags

- Runs on your computer: It runs as a program on your machine, with your access.

## Security audits

- Gen Agent Trust Hub: Pass
- Socket: Pass
- Snyk: Pass

Installs: 14. Installs counted by skills.sh. [Open on skills.sh](https://www.skills.sh/amplitude/builder-skills/craft-experiment-design)

## Source

[Open the source](https://github.com/amplitude/builder-skills/tree/HEAD/product-skills/skills/craft-experiment-design). Checked 2026-10-10. This summary may contain AI-generated text, so it can be wrong. Check the source before you rely on it.

## SKILL.md

````markdown
---
name: craft-experiment-design
description: Write a hypothesis, define success metrics, and plan a holdout strategy. Use when designing A/B tests or experiment plans.
suggest_when: User wants to test something, says "A/B test", "experiment", "should we test this", "hypothesis", or is planning a holdout strategy for a feature launch.
---

# Experiment Design

**Write a hypothesis, define success metrics, and plan a holdout strategy.**

You want to run an A/B test but need to get the plan straight first. This skill helps you go from "we should test this" to a well-structured experiment design that your team and data scientists can review.

---

## Prompt Template

```
You are an experienced product manager and experimentation specialist.

Here is what I want to test:

<context>
$ARGUMENTS
</context>

> If the above is blank, ask the user: "{{DESCRIBE THE CHANGE YOU WANT TO TEST AND WHY}}"

Help me design an experiment plan that includes:

1. **Hypothesis** — A clear, falsifiable statement in the format: "If we [change], then [outcome], because [rationale]."
2. **Primary Metric** — The single metric that determines success or failure.
3. **Secondary Metrics** — 2-3 supporting metrics to watch for unintended effects.
4. **Guardrail Metrics** — Metrics that must not degrade (e.g., error rates, latency, retention).
5. **Audience & Allocation** — Who should be in the test? What percentage split do you recommend?
6. **Holdout Strategy** — Should we maintain a holdout group after the test? Why or why not?
7. **Duration Estimate** — How long should we run the test and what assumptions drive that?
8. **Risks & Considerations** — What could go wrong or bias the results?

Be specific. Use real metric names where possible. Call out any assumptions I should validate with data or eng.
```

---

## Tips

- Include any prior data or context you have — conversion rates, traffic volume, previous test results. It helps with duration and allocation recommendations.
- If you're unsure about your guardrail metrics, ask the skill to suggest some based on your product area.
````

- Previous: [craft-spec](https://syn4pse.dev/terminal/amplitude-craft-spec.md)
- Next: [jobs-to-be-done](https://syn4pse.dev/terminal/amplitude-jobs-to-be-done.md)
- [Terminal](https://syn4pse.dev/terminal.md)
- [Suggest a change](mailto:terminal@syn4pse.dev?subject=Terminal%20suggestion)
