A/B test: from our 24 months experience

We had been working with our clients to improve their sites, some of them for more than 24 variants over 24 months period.

Here is some facts you may not know about A/B test:

Creativity is the major factor to find a good variant

The creativity involved in A/B test is highly underrated.

Knowledge to UI, design are crucial. Adjustment on button size, colors, text copies are common, but the most successful variants need creativity.

In one case, adding a “red dot” in hamburger menu help user discover the new feature in the menu by 50%. 10 pixel changes.

Not all experiments lead to posititive results.

Variants are “hypothesis”. Some of them lead to better results, sometimes all 2 varients leads to worse result and gains nothing.

These are un-avoidable, which can only be reduced but not removed, by experience.

The more creative the variant is, higher the risk, more likely they falls or, they have much better results.

It takes 3-6 weeks to select a winner

In general. It depends on varies factors, but better expect 3-6 week to find the winning variant.

And if there is a marketing campagin running, hold the experience even a bit longer. Visitors from a recent campaign might not best represent your ordinary traffic.

Each experiment provide 3-10% improvement

There are experiment that leads to 30% improvement, but very rare. High risk high return applies here.

In general, expect 3-7% improvement, and experiments needs to run months-over-months. Expect 50-60% improvement year-to-year for an monthly experiments.

AB test return will deminish on one page

The return will deminish over experiments, and it will be increasing difficult to get improvements.

If reach that point, further experiments will involves new key visuals, a better workflow, even a re-design, with data collected from past experiments.