An A/B test is a statistical tool, and statistics need numbers. With a few hundred visits a month and only a handful of conversions, any difference between variants will fall entirely within ordinary fluctuation. You can launch such a test, wait two weeks and declare a winner—but that winner will have been chosen by chance, not evidence.
What determines whether a test makes sense
The required number of visits depends on three things: your current conversion rate, the size of the change you want to detect and the confidence level you require. The rarer the conversion and the smaller the expected improvement, the more traffic you need—and the requirement grows much faster than intuition suggests.
- Baseline conversion rateAt 1%, you need many times more visits than at 10% to see the same relative difference.
- Detectable changeDetecting a 20% improvement is far cheaper than detecting a 3% improvement.
- Number of variantsEvery additional variant divides your traffic and lengthens the test, often beyond reason.
- Weekly cycleA test shorter than two full weeks confuses the effect of the change with day-of-week effects.
Instead of guessing, use a sample-size calculator—many are available online for free. Enter the actual conversion rate and traffic for the page you want to test. If the result says the test would need to run for eight months, you have your answer: this is not the right tool for that page at this time.
If the page being tested does not generate at least a few hundred conversions a month, an A/B test will rarely prove anything in a useful timeframe. Behaviour and conversations will reveal the truth faster—see how it works in Meetlead.
Mistakes that make even a good test lie
Even with enough traffic, it is easy to produce a result that looks convincing but has no substance. The following mistakes appear in almost every team that is new to testing.
Scroll the table sideways to see all columns.
| Mistake | What it looks like | Consequence |
|---|---|---|
| Peeking and stopping | Checking every day and stopping as soon as one variant leads | Declaring a winner based on a temporary fluctuation |
| Running the test for too little time | Testing for four working days | Measuring a day-of-week effect instead of the change |
| Testing several things at once | A new headline, colour and layout | You do not know what worked |
| Using the wrong metric | Measuring clicks instead of orders | The variant wins but sells less |
| Ignoring segments | One result for mobile and desktop | An improvement on one device masks a decline on the other |
The final row is particularly deceptive in Poland, where mobile traffic can account for a very large share. We explain how to separate this analysis in the article about mobile conversion rates.
What to do when traffic is too low
This is the most important part of this article, and the part most guides omit. Being unable to test does not mean you have to guess. It simply means your evidence will be qualitative rather than statistical—and with small numbers, qualitative evidence is simply more effective.
- Watch how people use the page.Ten session recordings from one page will reveal problems faster than any test.
- Look for patterns in maps.Clicks on non-interactive elements and scroll reach are facts, not hypotheses to test.
- Read customer questions.A recurring question indicates missing information. You do not need to test whether adding it is worthwhile.
- Fix things that are clearly broken.A validation error or an off-screen button does not require statistical proof.
- Test substantial changes, not shades.With low traffic, only clear differences are detectable. A button shade will never be one.
- Measure before and after.Less reliable than a concurrent test, but usually sufficient for a large change measured over a long window.
Before building a testing apparatus, remove the barriers that are obvious in a recording. Schedule implementation and we will help set up the first behaviour analysis on your website.
A note on before-and-after measurement: it is vulnerable to seasonality, campaign changes and fluctuations in traffic quality. Treat it as a clue, not proof. If the result rises after a change while the number of questions about the same issue falls, you have two independent signals pointing in the same direction—and that is usually enough to make a decision.
What to test once you have enough traffic
A test costs time and traffic, so it should concern something whose outcome is genuinely uncertain. Testing something that clearly needs fixing wastes both resources.
Scroll the table sideways to see all columns.
| Change | Test it? | Why |
|---|---|---|
| A different way of presenting the price | Yes | The reaction is hard to predict and the impact can be large |
| Removing three fields from a form | Worth measuring, but do not delay the decision | The direction is known; you are measuring the scale of the effect |
| Fixing a validation error | No | It is a defect, not a hypothesis |
| A new offer structure | Yes | A substantial change, a clear effect and a measurable goal |
| A button shade | Usually not | The effect is usually smaller than your ability to detect it |
The best sources of testing hypotheses are the same evidence you use for diagnosis: observations, maps, recordings and questions. We describe a structured process for collecting and prioritising them in the website conversion audit, and explain how to choose the events you measure in the article about website micro-conversions. How many visits a heatmap itself needs before it can feed a hypothesis is covered in our guide to heatmap sample size.
A short checklist before launching a test
- You know the current conversion rate of the page being tested, not the entire website.
- The calculated sample size fits within a reasonable test duration.
- The test will run for at least two full weeks and include weekends.
- You have chosen one primary metric linked to a real business outcome.
- You are changing one thing at a time or deliberately testing a complete page variant.
- You will review the result separately for mobile and desktop.
- The end date is set in advance and does not depend on the current result.
- You know what you will do if the result is inconclusive—because that is the most common outcome.
The last point prevents the most disappointment. Most tests end not in a spectacular victory but in no significant difference. That is still a result: it tells you the change does not matter and you can stop arguing about it.
A recording, a map and a customer question can give you an answer within a week, regardless of traffic volume. Explore behaviour analysis in Meetlead and see how many hypotheses you can eliminate without testing.
