A small company changes the headline on its main page. The next week there are more enquiries, so the change "worked". Or there are fewer, so the change "failed". In both cases the conclusion may be wrong. Enquiries move for many reasons: the season, a newsletter, a holiday, a competitor, plain chance.
An A/B test is the simple way to separate a change from the noise. Half of the visitors see the current page (A), the other half see a version with one change (B), at the same time. You compare one number at the end. You do not need an expensive tool to do this properly, but you do need rules.
When a test makes sense
Be honest about traffic first. If the page gets fifty visitors a month and three enquiries, no test will give a reliable answer in a reasonable time. In that case do not test; make the change you believe in, note the date, and watch the result with caution. If you want help writing the page itself, see a landing page for one small company.
A test starts to be useful when the page gets a few hundred visitors a month and the action you care about (an enquiry, a booking, a download) happens at least a few times a month. Even then, expect only large differences to show clearly.
Step 1: pick one page and one number
Choose the page where a result matters most, usually the page people reach before they contact you. Then choose one number:
- the share of visitors who send the enquiry form, or
- the share who click the phone number or the booking button.
One number is enough. If you track five, one of them will always look good by chance.
Step 2: change one thing
Change only one element: the headline, the text of the button, the position of the form, the first picture, or the price display. If you change the headline, the picture and the button together and the result improves, you will not know which change helped, and you cannot repeat it on other pages.
Write down in one sentence what you change and why you expect it to help. For example: "A headline that names the customer's problem will get more enquiries than a headline that names our service."
Step 3: write the decision rule before you start
This step is the one most often skipped, and it matters most. Before the test starts, write down:
- the start and end date (four weeks is a sensible default for a small site)
- the number you compare
- what difference makes you switch to B, for example "B has at least 20% more enquiries than A"
- what you do if the difference is smaller: keep A
Put the page on a single sheet with these four lines and give it to whoever is responsible. When the results arrive, you will be tempted to move the goalposts. The written rule protects you.
Step 4: run both versions at the same time
Do not show A for two weeks and then B for two weeks. A holiday, a campaign or a news item in one of those periods will distort the comparison. Both versions must run in parallel, each seeing roughly half of the visitors.
Many website platforms and testing tools can split visitors for you. If yours cannot, a simple alternative is to use two addresses and send half of your own campaign traffic to each. Keep the rest of the setup identical.
If you use a testing tool, read the search engine's guidance on testing so that the test does not harm how your page appears in search. Google's documentation covers the points to watch, such as showing the same content to people and to search crawlers and not leaving a test running longer than needed.
Step 5: do not look every day
Looking at the numbers daily and stopping when B is ahead is a classic mistake. Results swing in the first days. Check once to be sure the form and tracking work, then leave it until the end date.
During the test, do not change anything else on that page and do not launch other changes to the same audience if you can avoid it. Keep a short log of anything unusual, such as a mailing or a price change.
Step 6: decide and write it down
On the end date, take the number for A and the number for B and apply your rule.
- B is clearly better. Make B the page. Write down what you changed, the dates and the numbers.
- The difference is small or unclear. Keep A. A small difference with little traffic is very likely noise. You have still learned that this change does not matter much.
- A is better. Keep A. This is a useful result too.
Then choose the next single change. A few small tests in a year, each properly done, teach you more about your customers than many unrecorded changes. When enquiries do arrive, handle them quickly; a three-touch routine for following up on a sales lead helps, and handing leads from marketing to sales helps when more than one person is involved.
Common mistakes
- Changing several things at once. You cannot tell what worked.
- Stopping early when one version is ahead. Early leads often disappear.
- Running A and B one after the other. The calendar affects the result.
- Judging by feeling after the fact. Without a rule written in advance, almost any result can be explained.
- Testing a page with almost no visitors. Wait or change the page without a test.
- Ignoring consent rules. If your testing tool uses cookies or tracking, make sure your cookie notice and privacy information cover it.
A first step
This week, pick the one page that matters most and the one number you would like to improve. Write the four-line decision rule, choose one change and set an end date four weeks ahead. Then leave the page alone until that date.
Sources
- Google Search Central, Website testing and Google Search
- Nielsen Norman Group, A/B testing
- Wikipedia, A/B testing
- This article is a planning aid. Thresholds such as the visitor numbers and the 20% example are illustrations, not guarantees. No client results are claimed.
