PlugWiki

ab-testing Skill for Claude Code: Design A/B Tests and Experiment Programs

Designs A/B tests with a hypothesis, sample size and metrics, and sets up a growth experimentation program.

Installs
81.1K
on skills.sh
Stars
57.9K
on GitHub
Files
3
besides SKILL.md
Updated
Oct 2026
last repo commit

ab-testing makes your assistant act as an experimentation expert. It checks for your product marketing context file, then asks what you want to improve, your baseline conversion rate, traffic and constraints. It writes a hypothesis, picks a test type (A/B, A/B/n, multivariate or split URL), gives sample sizes per variant and defines primary, secondary and guardrail metrics.

It also covers variant design, traffic allocation, client-side versus server-side implementation, a pre-launch checklist and the peeking problem, then how to read significance. A second half covers a growth experimentation program with an experiment loop, ICE prioritization and a playbook. It bundles a sample-size guide, a test-templates file and an evals/evals.json file, and no scripts.

Good for

  • Marketers testing landing pages, pricing pages or emails
  • Product teams who want a repeatable experiment process
  • Developers choosing client-side or server-side testing

Install ab-testing

Claude Code

  1. 1

    Copy this prompt.

    Install the ab-testing skill in Claude Code. Run this command:

npx skills add https://github.com/coreyhaines31/marketingskills --skill ab-testing

Then check that it loaded and tell me how to use it.
  2. 2

    Paste it into Claude Code. It shows the change and asks before running it.

Requires Node.js.

From its listing on skills.sh. Source.

What's in the skill

When it loads

When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," "how long should I run this test," "growth experiments," "experiment velocity," "experiment backlog," "ICE score," "experimentation program," or "experiment playbook." Use this whenever someone is comparing two approaches and wants to measure which performs better, or when they want to build a systematic experimentation practice. For tracking implementation, see analytics. For page-level conversion optimization, see cro.

Sections in SKILL.md

  1. Initial Assessment
  2. Core Principles
  3. 1. Start with a Hypothesis
  4. 2. Test One Thing
  5. 3. Statistical Rigor
  6. 4. Measure What Matters
  7. Hypothesis Framework
  8. Structure
  9. Example
  10. Test Types
  11. Sample Size
  12. Quick Reference
  13. Metrics Selection
  14. Primary Metric
  15. Secondary Metrics
  16. Guardrail Metrics

Key facts

Author
Corey Haines
Repo
coreyhaines31/marketingskills
Installs on skills.sh
81,094 (Nov 2, 2026)
GitHub stars (whole repo)
57,855 (Nov 2, 2026)
License
MIT
Last repo commit
Oct 31, 2026
Files in the skill folder
4 files: SKILL.md, evals/evals.json, 2 in references/
skills.sh audits
Gen Agent Trust Hub pass, Socket pass, Snyk pass
Bundled references
sample-size-guide.md, test-templates.md

What it includes

  • Asks for test context, current state and constraints

    Initial Assessment

  • Turns an idea into a structured, specific prediction

    Hypothesis Framework

  • A/B, A/B/n, MVT and split URL, with per-variant sample sizes

    Test Types and Sample Size

  • Primary, secondary and guardrail metrics

    Metrics Selection

  • What to vary and how to split traffic

    Designing Variants and Traffic

  • Client-side and server-side options, with tools named for each

    Implementation

  • Checklists, the peeking problem and reading significance

    Running and Analyzing

  • Experiment loop, ICE scoring, velocity and a playbook

    Growth Experimentation Program

From the official docs

Questions

What does the ab-testing skill do?

It helps you design an A/B test that can give a valid result. It covers the hypothesis, sample size, metrics, setup and analysis, and how to run experiments as an ongoing program.

How do I install the ab-testing skill?

Use the install box at the top of this page. The repo's README also offers a Claude Code plugin called marketing-skills that installs every skill in the collection, and a Codex plugin route for the same set. Pick one skill only if you want just this one. If you ask an agent session to run npx skills for you, the README says to pass -a claude-code, because the CLI may otherwise install only to .agents/skills/, which Claude Code does not read.

Do I need the product-marketing skill first?

No. The ab-testing SKILL.md says to read your product marketing context file if one exists, and to ask you for anything the file does not cover. The README calls product-marketing the foundation that the other skills check first, so it helps to run it once. Its Related Skills section also names cro, analytics and copywriting for test ideas, measurement and variant copy.

Does it calculate sample size?

It gives a quick-reference table for baselines of 1% to 10% and links two outside calculators, from Evan Miller and Optimizely. A bundled guide holds longer tables and duration math.

Does it run the test for me?

No. It plans and analyzes. The Implementation section names tools such as PostHog, Optimizely, VWO, LaunchDarkly and Split, but the skill does not connect to them.

Does it work in Codex, Cursor or only Claude Code?

The README says the collection works with Claude Code, OpenAI Codex, Cursor, Windsurf, and any agent that supports the Agent Skills spec. It does not list test results per skill.

Is the ab-testing skill safe to install?

It is a markdown skill. The folder bundles reference files and an evals/evals.json file, and no scripts. The SKILL.md names no API key. It changes how your assistant works in your project, so read the SKILL.md if you need to review it first.

Information

updated Nov 2, 2026

Repository
coreyhaines31/marketingskills
Folder
skills/ab-testing

coreyhaines31/marketingskillsskills.sh

Sources

Docs, issue trackers and discussions this page is built from.

Show all 3
  1. ab-testing SKILL.md · github.com
  2. marketingskills README · github.com
  3. ab-testing on skills.sh · skills.sh

Updated Nov 2, 2026.