Design a measurable experiment with ak:ab-test-setup

Use ak:ab-test-setup to turn a proposed page, feature, or campaign change into a reviewable experiment plan. The Skill defines the hypothesis, variants, audience, metrics, allocation, implementation approach, and analysis rules you need before exposing participants to a test.

Choose ak:ab-test-setup for a testable decision

Use ak:ab-test-setup when

• You have an observation or data point that motivates a specific change.
• You can define one primary decision metric and relevant harm guardrails.
• You know the baseline, eligible audience, traffic, and smallest worthwhile effect well enough to state assumptions.
• You need an A/B, A/B/n, multivariate, or split-URL design and a pre-launch checklist.

Choose another workflow when

• You only need event instrumentation. Use the Marketing analytics workflow.
• You still need conversion hypotheses for a form or onboarding flow. Start with ak:form-cro or ak:onboarding-cro, then bring the selected hypothesis here.
• You want to roll out a known change without estimating a causal effect. A feature flag or staged release may fit better than an experiment.
• You cannot assign variants consistently, measure outcomes reliably, or expose participants fairly. Fix those constraints before calling a result causal.

Prepare evidence and runtime access

Before you start, provide the current experience, the observed problem, the eligible audience, baseline metric definitions, expected traffic, known seasonality, technical constraints, and the decision the test should inform. State any consent, privacy, legal, brand, or accessibility requirements.

Complete Onboarding, and confirm Marketing Kit is installed for the runtime and scope you are using.

See Runtime adapters for component-level differences.

Run the Skill

Ask for a plan first when implementation authority, production data, or an experiment provider is not already approved.

/ak:ab-test-setup "Plan an A/B test for the demo-request headline. Use qualified demo requests as the primary metric, protect lead quality and accessibility, state every sample-size assumption, and do not edit or launch anything"

/ak:ab-test-setup "Plan an A/B test for the demo-request headline. Use qualified demo requests as the primary metric, protect lead quality and accessibility, state every sample-size assumption, and do not edit or launch anything"

$ak:ab-test-setup "Plan an A/B test for the demo-request headline. Use qualified demo requests as the primary metric, protect lead quality and accessibility, state every sample-size assumption, and do not edit or launch anything"

Useful input separates observation from prediction:

Because [evidence], we believe [one change] will cause [directional outcome]
for [eligible audience]. We will decide using [primary metric], while monitoring
[guardrails].

Do not insert a target lift merely to make the hypothesis look precise. Define the minimum detectable effect from decision value, traffic, cost, and risk.

Understand the experiment stages

1. Confirm the decision. The Skill records the observation, change, audience, expected direction, and action each possible result would support.
2. Choose the design. It distinguishes A/B, A/B/n, multivariate, and split-URL designs and explains the traffic and interpretation trade-offs.
3. Pre-register measurement. It defines one primary metric, supporting metrics, guardrails, eligibility, exposure, analysis unit, segments, and stopping rule before launch.
4. Plan sample and duration. It uses the baseline, minimum detectable effect, significance threshold, power, variant count, traffic, and business cycles as explicit assumptions.
5. Describe variants and delivery. It documents the unchanged control, isolated change, allocation, persistence, QA, tracking, and client-side, server-side, or feature-flag approach.
6. Prepare analysis and learning. It records confidence intervals, effect size, guardrail outcomes, planned segment checks, limitations, and the decision as winner, loser, or inconclusive.

Keep consent and statistical decisions explicit

• Randomize only an eligible population and keep assignment stable for the declared analysis unit.
• Collect the minimum data needed, document retention and access, and obtain required consent before tracking sensitive behavior.
• Do not use deceptive scarcity, hidden prices, inaccessible variants, or unequal treatment that participants would reasonably consider harmful.
• Do not repeatedly inspect a fixed-horizon test and stop at a favorable result. If sequential monitoring is required, pre-specify a valid sequential method.
• A threshold such as p < 0.05 is not the probability that the result is random, and statistical significance is not proof of practical value.
• Treat unplanned segment findings as exploratory until independently tested.

Review the outputs and evidence

A useful result includes:

• A complete hypothesis and the decision it is intended to support.
• Control and variant descriptions with one attributable difference where possible.
• Metric definitions, guardrails, eligibility, allocation, and exposure rules.
• Sample-size and duration calculations with assumptions and method named.
• An implementation and QA checklist, not an unreviewed production mutation.
• A pre-specified analysis plan and a results template that can record an inconclusive outcome without forcing a winner.

Recalculate the design with your analytics or experimentation owner. The Skill's plan cannot validate unseen tracking, traffic quality, interference, novelty, seasonality, or external events.

Troubleshoot the workflow

Know the limits

• The Skill does not supply trustworthy benchmarks, traffic, baselines, or causal validity when your evidence does not.
• Sample size depends on the chosen method and assumptions; generic lookup tables are not a substitute for a calculation reviewed for this experiment.
• Low traffic, network interference, novelty, multiple comparisons, and metric changes can make a test inconclusive or misleading.
• The documented package releases contain identical ak:ab-test-setup content and invocations.

Continue with the Marketing Kit overview, or use ak:form-cro, ak:onboarding-cro, or ak:pricing-strategy to develop a candidate hypothesis before designing its test.