Where AI Falls Short in CRO: What to Automate, What to Measure, and What Still Needs a Human

AI is good at reading a page and bad at knowing your customers. A large language model can tell you that a headline is vague, a CTA is buried, or a form asks for too much. It cannot tell you whether your visitors care, because it never sees them. That distinction decides what you should automate and what you should not.

We build an AI CRO audit tool, so this is not a neutral opinion. It is the one we have arrived at from watching where automated findings hold up and where they quietly mislead.

What AI is genuinely good at in a CRO audit

The honest case for automation is consistency, not intelligence. A model applies the same checks to page fifty that it applied to page one, at two in the morning, without getting bored or developing a favourite theory. Human reviewers do not do this. They get sharper on the pages they find interesting and vaguer on the rest.

That makes AI genuinely useful for the parts of CRO that are pattern recognition against known conventions:

• Clarity checks. Whether a value proposition is specific or could describe any competitor.
• Convention checks. Whether the page does what visitors expect — a visible primary action, proof near claims, a form that asks only what it needs.
• Comparison at scale. Scoring twenty landing pages against the same rubric so you can see which is weakest, rather than which you looked at most recently.
• Triage. Deciding which page deserves an expensive human hour.

Every one of those is a judgement about the page in isolation. That is exactly the boundary.

Where do LLMs fall short in CRO?

LLMs fall short in conversion optimisation wherever the answer depends on evidence the model cannot see: your analytics, your customers, your margins, and what actually happened when you changed something. Below are the six failures that matter most in practice.

1. It reads a page, not a session

A model sees a static document. It does not see the visitor who scrolled twice, opened your pricing in a second tab, came back four days later and converted from a branded search. Conversion is a behaviour spread across sessions and devices; a page audit is a snapshot of one surface within it.

This is why AI findings skew toward things visible on the page and away from things visible only in the funnel. If your real problem is that traffic arrives with the wrong intent, no amount of page-level review will name it — the page might be excellent and still convert badly.

The practical symptom is an audit that reads well and changes nothing. You fix the six things it found, the page is genuinely better, and the conversion rate does not move, because the loss was happening one step earlier. Page-level review answers "is this page good", which is a different question from "is this where we are losing people".

2. It has no access to your data

The model does not know your conversion rate, your traffic mix, your device split, or which page types already perform. Without that, everything it produces is a hypothesis of equal weight. It cannot tell you that a weak checkout matters more than a weak blog page, because it does not know that ninety percent of your revenue passes through the former.

Prioritisation is where CRO earns its money, and prioritisation requires volume data. An audit that ranks findings by severity rather than by exposure will confidently point you at the wrong fix.

The fix is not complicated, but it has to happen outside the tool. Pull your top landing pages by sessions before you audit anything, and read the audit in that order rather than the order it hands you. A severity-ranked list is a list of problems; a traffic-weighted list is a list of decisions. Only the second one tells you what to do on Monday.

3. It cannot establish cause

An LLM can say a CTA is likely to underperform. It cannot say that changing it will lift conversion, because that claim requires a controlled comparison against a baseline you have actually measured. Causal inference needs an experiment, not a reading.

This matters more than it sounds. The failure mode is not that AI recommendations are wrong — many are reasonable — it is that they arrive with the same confident tone whether they would move the number by four percent or by nothing at all. Treat every finding as a hypothesis with an unknown effect size until you have tested it.

It also means the honest unit of an audit is the question, not the answer. A finding that says "the CTA competes with three other actions" is useful because it tells you what to test. A finding that says "changing this will improve conversion" is claiming knowledge that cannot exist before the test runs.

4. It optimises toward convention, not toward your audience

Models are trained on what most pages do. That makes them a reliable guide to convention and a poor guide to differentiation. Convention is usually the right starting point — visitors carry expectations from every other site they use — but it is a floor, not a ceiling.

If your audience is unusual, the average is actively misleading. A model will flag a long, dense technical page as a wall of text. For an engineering buyer comparing implementation details, that density may be the reason they trust you. Nothing in the page tells the model which situation it is in.

So the rule of thumb is: accept convention findings by default, and override them where you have evidence about your specific buyers. "Shorten this" is usually right. "Shorten this" on the one page your most technical customers read before committing is a decision you should be making with data, not with a rubric built from everyone else's pages.

5. It cannot reach most of your funnel

Audits read publicly reachable pages. Everything behind authentication is invisible: onboarding, dashboards, upgrade prompts, post-purchase flows, the account settings screen where subscriptions get cancelled. For most subscription products, a large share of the revenue-relevant experience sits behind a login.

The same applies to anything gated by state — cart pages with items in them, checkout steps that require a real payment method, personalised views. If your conversion problem lives in one of those, an automated audit will report cheerfully on the pages either side of it.

This is worth checking before you buy any audit tool, because the gap is invisible in the output. Nothing in a report says "we could not see your checkout". You get a clean set of findings about your marketing pages and no indication that the half of the journey where people actually pay was never examined. Walk the logged-in path manually, with the same rubric, and treat that as part of the same audit.

6. It sounds equally sure when it is wrong

This is the most practically dangerous limitation. Language models produce fluent, structured, confident output regardless of whether the underlying judgement is well-founded. There is no visible difference between a finding drawn from a strong pattern and one produced because the format demanded another bullet point.

Human reviewers hedge when they are unsure, and that hedging carries information. Automated output flattens it. The practical defence is to treat structure as structure, not as certainty, and to check any finding you are about to spend real money on.

A useful habit is to read any automated audit looking for the finding you disagree with. There is almost always one, and locating it recalibrates how you read the other nine — you stop treating the list as a verdict and start treating it as a well-organised set of suggestions, which is what it is. The confident formatting is a property of the model, not evidence about your page.

What to automate, what to measure, and what needs a human

The useful question is not whether AI is good at CRO. It is which parts of CRO are page-shaped, which are data-shaped, and which are people-shaped.

A practical roadmap

The sequence matters more than the tools. Running these in the wrong order is how teams end up with a long list of improvements and no measurable change.

• 1. Establish exposure first. Before auditing anything, find out which pages carry your traffic and revenue. Optimising a page nobody reaches is the most common wasted week in CRO.
• 2. Automate the sweep. Run a consistent audit across those pages to rank them against each other. The output you want here is an ordering, not a to-do list.
• 3. Read the top of the list yourself. Manually review the two or three weakest high-traffic pages. This is where automated findings get confirmed, discarded, or reinterpreted against what you know about your buyers.
• 4. Ask people, if the answer is about motivation. If the audit and the analytics both say a page underperforms and neither explains why, that is a research question, not a copywriting one.
• 5. Change one thing and measure it. Attribution collapses when you ship four fixes at once, and the next audit cannot tell you which one earned the improvement.
• 6. Re-run the same audit. The value of a repeatable rubric is that score movement means something. A one-off review has no baseline to move against.

How do you tell whether an AI finding is worth acting on?

A simple test, applied before you spend anything: can you name the visitor this change helps, and the moment in their journey it helps them?

If you can, the finding is probably real and you have just described the hypothesis you are testing. If you cannot, you are about to make a page marginally more conventional without knowing whether conventionality was the problem. That is not harmful, but it is not conversion optimisation either — it is tidying.

The second test is exposure. Multiply the severity of the finding by the number of people who encounter it. A moderate problem on your highest-traffic page beats a severe problem three clicks deep, almost every time.

The honest summary

AI has genuinely changed the economics of the first pass. Reviewing thirty pages against a consistent rubric used to be a week of specialist time and is now cheap enough to do routinely. That is a real gain, and it is the gain we built our own tool around.

What has not changed is that conversion is a claim about human behaviour, and claims about behaviour are settled by measurement, not by reading. Use automation to decide where to look. Use your data to decide what matters. Use experiments to decide whether you were right. Anyone selling you a tool that claims to do all three is describing a product that does not exist yet.