A/B testing in marketing: how to test so the result is real

An A/B test shows two versions of something to comparable groups of people and measures which one brings more of the result you care about. It sounds simple. Most tests still end with a “winner” that is really just noise.

What to test first

Test the things that change the decision to buy, not the colour of a button:

• The offer: free shipping vs a discount, a bundle vs a single product, a trial vs a first-order price.
• The main message in ads and in the first screen of the landing page.
• The creative format: video vs image, product in use vs product alone.
• The landing page: product page vs a dedicated landing page.
• The form or checkout: fewer fields, guest checkout, payment options.

How to run a fair test

1. One hypothesis, one change. “Showing delivery time in the ad will raise the conversion rate” is a hypothesis. Changing five things at once is not a test.
2. Choose the metric before you start: purchases or leads, not clicks. A version can win on clicks and lose on sales.
3. Split traffic randomly and at the same time. Comparing this week with last week is not an A/B test; seasons, prices and competitors change.
4. Run it long enough. At least one full week to cover weekdays and weekends, and until each version has a meaningful number of conversions. With a handful of conversions per version, any difference is chance.
5. Do not stop early because one version is ahead after two days. Early leads often disappear.

Tools

• Meta: the built-in A/B test (experiments) splits audiences properly.
• Google Ads: experiments for campaigns, and ad variations.
• Website: tools that split visitors between two page versions, or a simple server-side split if your developers can build it.

Mistakes that produce false winners

• Too little data. With small numbers, differences of 20–30 % can appear by pure chance.
• Peeking and stopping as soon as the result looks good.
• Testing on the wrong metric, such as click-through rate.
• Letting the platform send more budget to one version during the test, which is not a fair split.
• Not writing tests down. Keep a simple log: hypothesis, dates, result, decision. It stops the same test being repeated a year later.

What to do with the result

Keep the winner, write down why you think it won, and use that insight in the next test. A good test programme is a series of small, documented steps, each building on the last.

Tests are part of how we work with clients month to month; see Growth partner.