Many B2B SaaS pages do not yet have the conditions for a credible A/B test. The Evidence Threshold tells you whether to diagnose, validate or run a controlled test first.
Anita Valentinova
Published
Last updated
Before A/B testing in B2B SaaS, find where qualified buyers are lost and why, using sales, support, customer and behavioural evidence. Then validate the proposed change in cheaper ways. Run a controlled test only when traffic is stable, the conversion event is clear and the volume can produce a credible result.
When a demo page underperforms, the first suggestion is often a test. Change the headline, split the traffic, let the data decide. It sounds disciplined, and in a product with a large and steady flow of signups it often is.
A smaller B2B SaaS team is usually in a different position: a demo page with a modest number of requests a month, a long sales cycle and traffic that changes every time a campaign starts or stops. A test on that page can run for a long time and still not tell the team anything it can act on.
This post is about the work that comes before a test, and how to tell when a test becomes worth running.
A/B testing answers one narrow question well: of two versions shown to comparable visitors over the same period, which produced more of a single event. Four things about B2B SaaS make that question hard to ask cleanly.
Volume. A test needs enough conversions in each variant to separate a real difference from noise. How many depends on your current conversion rate and on how small a change you want to be able to detect. I am deliberately not giving a traffic number here, because any single figure will be wrong for most readers. Put your own numbers into a sample-size calculator, such as Evan Miller's, and compare the sample it asks for with the traffic that step receives.
Distance from revenue. The event a test counts is usually a form fill or a signup. The outcome you care about is an opportunity sales accepts, or an account that activates and pays. A variant can win on the first and lose on the second, and with a long sales cycle you find out months later.
Mixed traffic. If the page receives paid search this month, a LinkedIn campaign next month and a newsletter spike in between, the audience changes from week to week. The test ends up comparing more than the two variants.
The size of the change. The decisions that matter most on a SaaS site are often structural: which buyer the page is for, what the offer is, whether to ask for a demo or offer a trial. Those are hard to isolate in a split test, and they are exactly the decisions teams most want evidence for.
I use a simple check before recommending any test. I call it the Evidence Threshold. A decision crosses the threshold when the evidence you hold is strong enough for the size of the commitment you are about to make. Four checks tell you how far away you are.
Constraint clarity
Constraint clarity is how sure you are about where the right people are being lost. If the team has opinions and no shared view, the next move is to diagnose. A test at this point optimises a step you have not confirmed is the constraint.
Confidence in the fix
Confidence in the fix is how much evidence supports the specific change proposed. If the constraint is clear but the fix is a matter of opinion, validate the change before anyone builds it.
Cost of being wrong
Cost of being wrong is what you lose if the change fails, and how easily you can undo it. A headline is cheap to reverse. A new offer or qualification model is not, and it deserves more evidence before it ships.
Testability
Testability is whether a controlled test on this step could give a credible answer in time to matter. It needs stable traffic, one clear conversion event and enough volume inside your decision window. Where it is high, run the experiment. Where it is low, ship the validated change with a baseline and measure what you can.
The same four checks, with a matrix that turns the scores into a next move, are in the full Evidence Threshold guide.
Diagnosis is about location. Before deciding what to change, you want to know which step is losing qualified buyers, and whether they were qualified when they arrived.
The evidence is usually already in the business:
Read together, they tend to point at one of a few places. The wrong people are arriving. The right people arrive and the page does not answer what they came for. The page works and the form or the next step loses them. Or the conversion happens and the handoff afterwards wastes it.
Validation is about reducing the risk of building the wrong thing. You have a constraint and a proposed change, and you want cheaper evidence than a full build and a long test.
Four methods cover most situations:
Message testing
Message testing checks a few versions of the core claim with people who match the ICP, then records what they understand, doubt and would do next.
Prototype or draft-page review
A prototype review walks a handful of buyers, customers or sales colleagues through a draft and notes where they hesitate. It is small, and it is often enough to remove the weakest option.
Sales-call evidence
Sales evidence means trying the new framing or proof in live conversations and seeing whether it shortens the explanation or brings up better questions.
Controlled rollout
A controlled rollout ships the change to one campaign, one segment or one region first, with a baseline recorded beforehand.
None of these gives you statistical certainty. They give you a reasoned basis for choosing what to build.
A test earns its place when the step has steady traffic from a consistent mix of sources, one conversion event that the business trusts, and enough volume to detect the change you care about within a period you can afford to wait. Signup and early onboarding steps in a self-serve product are the likeliest places to meet those conditions.
Even then, check quality downstream. Record which variant each lead or account saw, and look at sales acceptance or activation before declaring a winner.
When a test is not possible, teams often ship a change and compare this month with last month. That is a reasonable thing to do. It is also weaker evidence, because seasonality, campaigns and releases all move the number at the same time.
Report these as observations. Say what changed, what you measured before and after, what else changed in the same period, and how confident that leaves you.
Pick one path that matters commercially, for example paid search to demo request, and go through the four checks in order. The first check is often where it stops: there are opinions about the page and no shared view of where qualified buyers are being lost.
Controlled tests are valuable when the conditions are there. I have designed and run them in that situation. What I would avoid is treating a test as the default first move, then treating an inconclusive result as a reason to stop looking.
Yes. Conversion work is about finding and removing what stops qualified buyers from taking the next step. Sales and support evidence, customer research, analytics and behavioural data can locate the constraint, and message tests or prototype reviews can reduce the risk of a change. A/B testing is one method for the final step.
There is no single number. It depends on the current conversion rate of the step, the smallest change you want to detect and how long you can wait. Use a sample-size calculator with your own figures. If the answer is many months, a test is unlikely to be the best use of that page.
The event a test counts, usually a demo request or signup, happens weeks or months before revenue. A variant that increases form fills can lower lead quality, and you only see that once sales has worked the leads. Tracking which variant each lead saw helps, but it lengthens the time to a trustworthy answer.
Diagnose where qualified buyers are being lost, validate the proposed change with message testing, prototype reviews or sales conversations, and then ship it with a recorded baseline. Describe what you see afterwards as an observation, and note what else changed in the same period.
It is a check I use before recommending a test. It scores four things for one step: constraint clarity, confidence in the fix, cost of being wrong and testability. The scores point to one of three moves: diagnose, validate or experiment.
No. Some steps in a smaller product do have steady volume, often signup or an early onboarding step. Test there if the conversion event is clear. The point is to choose the method that fits the evidence available, step by step.
If you have a demo or signup path that underperforms and you are not sure whether to diagnose, validate or test, request an Opportunity Review. I will look at one priority path with you and tell you which of the three I would do first, and why.
If you need to decide what comes first on one underperforming demo or signup path, see how a SaaS conversion diagnostic works.
GrowthLens