What is A/B Testing?

TL;DR

• A/B testing compares variants using real user behavior.
• Participants should be randomly assigned to versions.
• The primary metric and hypothesis should be defined before looking at results.
• A/B tests show which variant performed better, not automatically why.
• Experiments need enough traffic and time to avoid misleading conclusions.

What does an A/B test look like?

Version A shows the current onboarding page. Version B moves the team-invite step earlier. New eligible users are randomly split between the two variants.

The team measures whether the change improves the predefined activation outcome while watching guardrail metrics such as errors or drop-off.

A/B testing vs. usability testing

Usability testing helps explain where and why users struggle in a designed experience. A/B testing measures which variation performs better at scale.

Teams often use qualitative research to generate hypotheses and A/B testing to measure impact.

What makes a good A/B test?

• one clear hypothesis
• random assignment
• a predefined primary metric
• enough sample size
• a stable experiment window
• guardrail metrics
• minimal contamination between variants

How do teams run an A/B test?

1. Define the product problem.
2. Write the hypothesis.
3. Choose the primary outcome metric.
4. Create the variants.
5. Randomize eligible users.
6. Run long enough for reliable data.
7. Analyze the result and uncertainty.
8. Decide whether to ship, iterate, or reject the change.

Common mistakes

Stopping as soon as one version looks ahead

Early results can be noisy.

Testing too many changes at once

Then the result becomes harder to interpret.

Choosing the metric after seeing the result

Define success before the experiment.

Using A/B testing for tiny traffic

Qualitative methods may be more useful when the sample cannot support a reliable experiment.

The bottom line

A/B testing asks: when comparable users encounter these alternatives, which version creates the better measurable outcome?

Related terms

Iteration · Usability testing · Product analytics · Product outcome

Relevant Figr resource

Read User Research Methods for how A/B testing fits alongside qualitative research.