A CRO process for sites that will never reach significance

Almost every CRO article assumes you can run A/B tests. Most businesses I work with can’t, and telling them otherwise wastes months. So first, here’s the arithmetic that decides whether testing is open to you, and then what to do when, as for most sites in this market, it isn’t.

The sample-size maths, honestly

To detect a difference between two versions of a page, each version needs enough conversions that the difference can’t plausibly be chance. Two things drive the number you need, and both work against small sites:

• The smaller the effect you want to detect, the more traffic you need, and the relationship is harsh: halving the effect size roughly quadruples the sample.
• The lower your conversion rate, the more traffic you need. A page converting at 1% needs far more visitors than one at 10% to reach the same confidence.

Take a realistic case. A regional B2B site with a few thousand sessions a month and a low single-digit conversion rate, trying to detect anything smaller than a big uplift, needs months for each test. That’s per test, one after another. Two or three tests a year, each possibly inconclusive, isn’t a CRO programme.

Worse, it fails silently. A test without enough traffic doesn’t tell you so. It returns a number, someone calls it a winner, it goes live, and the “uplift” disappears. Plenty of published test wins are exactly that.

Run the numbers for your own site before you plan anything. If one test would take more than six weeks, testing isn’t your method.

What to do instead

Qualitative methods have a property that makes up for low traffic: they don’t need volume, because you’re looking for problems, not measuring differences. Twenty sessions will show you a broken form. They won’t show you a 3% uplift, but the broken form is worth more than the 3% anyway.

1. Session recordings. Watch thirty recordings of people who reached the key page and didn’t convert. Treat it as an investigation, not a sample. You’re looking for hesitation, repeated scrolling, angry clicking and where people give up.
2. Form analytics. Which field loses people, which gets retyped, and where people drop off on mobile compared with desktop. On lead-generation sites this one method finds more money than everything else combined.
3. Exit surveys. One question on the page people leave from: “What stopped you today?” Even a few answers point the way, and the words people use become your next headline.
4. Ask the sales team. They hear the objections every day. The three questions they answer most on calls are almost never answered on the page, and adding them usually beats any layout test.
5. Five-second tests with real people. Show the page to five people in your target market for five seconds and ask what the company does. If they can’t say, nothing else on the page matters yet.
6. Read the search queries. Search Console shows what people wanted when they arrived. A gap between that and what the page says is a conversion problem dressed up as an SEO report.

Before-and-after testing and its real risks

Changing the page and comparing the next month with the last is what most small sites end up doing. It’s legitimate if you’re honest about its weaknesses, and dangerous if you’re not.

• Seasons affect everything. Ramadan, summer, school terms and the end of a financial quarter move conversion rates more than most page changes do.
• The mix of visitors changes under you. A change in which campaign is running or which keyword ranks changes who lands on the page, and that, not your change, may explain the whole effect.
• Change one thing at a time, or you learn nothing. Redesigning the whole page and calling it a test tells you whether the new page is better, not why.
• Use a longer window than feels necessary, and compare with the same period last year as well as last month, where you have the data.
• Write down what you expect first. A prediction made before the change is the only protection against explaining any result afterwards.

Fix the obvious before testing the subtle

None of these needs a test to justify it, and on most sites in this market at least half are still there:

• A form asking for information the business doesn’t need at this stage.
• No WhatsApp option on a site whose customers close deals on WhatsApp.
• No prices, or prices in the wrong currency for the visitor’s market.
• No phone number visible on mobile without scrolling.
• A page that takes six seconds to become usable on a mid-range Android phone.
• Arabic pages with a broken right-to-left layout, or English labels left in an otherwise Arabic form.
• No answer to the buyer’s real question: delivery time, warranty, who else uses this, what it costs.
• A button that says “Submit”.

This list is the whole first month of most of my CRO work, and it usually brings a bigger change than the following year of testing would have.

Deciding what to change next

With no statistics to settle arguments, you need a consistent way to choose. Score each proposed change out of five on four things:

• Evidence: did something specific show us this is a problem, or is it a hunch?
• Reach: how many of the people who matter will see it?
• Effort: how long until it’s live?
• Reversibility: how easily can we undo it if things get worse?

Strong evidence, wide reach, low effort and easy to undo go first. The point isn’t precision. It’s that the loudest person in the room stops deciding, which without tests is the real risk.

When you can finally test properly

Once traffic grows, come back to testing, but start with the tests that could have the biggest effect, because those are the only ones your traffic can settle. Test the offer, how prices are presented and the page structure. Don’t spend your first real testing capacity on button colours.

And keep the qualitative work going regardless. Tests tell you which version won; recordings and surveys tell you what to build next. Teams that stop watching sessions once they can test run out of ideas within two quarters.

For the page itself, see what an Arabic landing page needs, and for the tracking, a GA4 event setup you will actually use.

This process is what my CRO and landing pages work looks like for smaller sites, and it depends on getting the measurement right first, which is analytics and tracking.

Frequently asked

Can I lower the confidence level to speed things up?

You can, and you’ll ship more false winners. If you do, say so clearly in the report and treat the results as weak evidence, not decisions.

Are bandit or sequential testing tools the answer?

They share traffic out more efficiently, but they don’t create traffic. With truly low volume the limit is still there.

How many session recordings should I watch?

Twenty to thirty from the right group of visitors, watched properly with notes. Most people watch three and stop, which is why the method gets dismissed.

Can I test on smaller steps instead?

Sometimes. A click to the contact page happens far more often than a form submission, so tests finish faster. The risk is improving a step that doesn’t lead to more customers, so check that the two are linked before you rely on it.

Should I hire a CRO agency at this size?

Not one that sells a testing programme. At this traffic level you need someone who’ll watch recordings, talk to your sales team and fix the eight obvious things, which is a much shorter job.

Want to know which of the eight obvious problems your site has? Send me the link.

Muhammad Ammar

SEO and digital marketing consultant since 2015, working with businesses in Egypt, Saudi Arabia and the UAE.

About the author and editorial standards