SEO A/B testing

SEO A/B testing: split-test titles and measure them honestly

Change titles on a random half of similar pages, leave the other half alone, and compare. How to split, how long to run, and how to tell a real effect from noise.

Explore the live demo →

5 days free · no card needed · ends on its own

Updated

SEO A/B testing (or SEO split testing) means changing something, usually the title template, on a random half of a group of similar pages, leaving the other half alone, and comparing how the two halves’ organic visits change. Unlike a conversion A/B test, you split pages, not visitors, because Google sees only one version of each page. It works on any site with dozens of similar pages, and the control half is what makes the result trustworthy.

Page-split vs user-split testing

User-split (CRO) testPage-split (SEO) test
What is splitVisitors to the same pagePages of the same type
What Google seesOne version (Googlebot doesn’t keep cookies)Version A on some pages, version B on others
MeasuresWhat visitors do on the pageRankings, impressions and clicks from search
NeedsTraffic to one pageMany similar pages: listings, city or category pages, products
Typical toolsOptimizely, VWO, your own codeSearch Console data and a random split of URLs

A user-split test can’t tell you whether a title ranks or gets clicked better, because Google only ever indexes one title per page. To test SEO, the pages themselves have to differ, and you compare groups of pages.

Google’s rules for testing

Google allows testing and asks for a few things (Google’s guide to website testing):

A page-split title test fits all four: each page has one title that everyone sees, there are no alternate URLs, and you either keep the change or revert it.

How to run a title split test

  1. Pick one page type with many similar pages. City pages, category-in-city pages, listings. They must share a template, so the only difference is your change. DirectoryLens needs at least 40 pages and at least 20 pages with impressions in each half; below that, normal page-to-page differences swamp most effects.
  2. Split the pages at random, and freeze the split. Don’t split alphabetically or by ID ranges: alphabetical groups can differ in size of city, and older IDs are often older pages. In a spreadsheet, add a random number to each URL, sort, take the top half, and save it. Never move a page between halves.
  3. Check the halves match before you start. Compare their clicks per day over the last 28 days. They won’t be identical, which is why the method compares each half with its own past.
  4. Change the template on the test half only, usually with a condition in your CMS on the list of test URLs. Write down the day it went live.
  5. Wait for Google to recrawl. Titles in search results update only after Google has crawled the pages again. DirectoryLens starts measuring 7 days after the go-live date.
  6. Measure for at least 14 days, then compare (below). Don’t stop the day it looks good; early differences often fade.
  7. Roll out, or revert. Then test the next idea.

Measure it with difference in differences

A plain before-and-after can’t separate your change from everything else: a holiday, a Google update, a competitor dropping out. The unchanged half goes through all of that too, so compare each half’s change:

effect = (test after / test before) / (control after / control before) - 1

Use visits per day in each window, so windows of different lengths work. For example: the test half goes from 100 to 120 visits a day (+20%), the control half from 100 to 110 (+10%). The effect is 1.20 ÷ 1.10 − 1 = about +9%. The season added 10% to everything; your titles added about 9% on top.

Then ask how sure you can be. Pages differ a lot from each other, so the honest way is to resample pages: draw each half’s pages again with replacement a few hundred times, recompute the effect each time, and take the middle 95% of results as the range. If the whole range is above zero, the change likely helped; if it is all below zero, it likely hurt; if it spans zero, you haven’t shown a difference.

You can do this from Search Console without a testing tool. Filter the Pages dimension with a regex matching each half’s URLs (Custom (regex) filters), or pull page-level daily data through the API, and sum clicks per half per window.

What is the difference between A/A and A/B testing?

An A/A test splits pages into two halves and changes nothing. Any difference you see is noise, so it shows how big a difference your pages produce by chance, and catches mistakes in how you split or measure. If an A/A test on your city pages swings ±8% in a month, a title test showing +5% proves nothing. An A/B test is the real thing: one half gets the change.

Sample size and how long to run

The honest answer for most sites: you can detect big effects, not small ones. Two things decide what you can see:

Run until you have at least 14 days after the recrawl, longer for page types with few visits, and avoid ending a test during a known Google update or a holiday that hits the halves differently. Then stop: Google’s testing guidance asks you not to leave tests running longer than needed.

What you can and can’t test this way

ChangeTestable with a page split?Note
Title templateYesThe most common test; shows up after a recrawl.
Meta description templateYesGoogle may not use it for every search, which dilutes the effect.
H1 and intro textYesCan affect both ranking and the title Google shows.
Internal link blocks (nearby, related)Yes, with careLinks from test pages also reach control pages, which blurs the split.
Structured dataYesRich results can take weeks to appear.
Site-wide changes (speed, navigation, domain)NoEvery page changes at once, so there is no control half.
BacklinksNoYou don’t control which pages get them.

Common mistakes

Is SEO A/B testing worth it?

Yes, when you have many pages on one template and a change you would otherwise roll out blind. A title template change on 900 city pages can lose as many visits as it gains, and you won’t see that in a before-and-after. It is not worth it for a single page (there is no control), for a page type with a few dozen low-traffic pages, or for changes so small no realistic test could detect them. For one important page, change it, annotate the date, and watch it against similar pages.

What to test first on a directory

Pick ideas from your own data first. The click rate check shows which page types are clicked below their position, and striking distance keywords show which words their titles miss.

How DirectoryLens runs title tests

Quick wins > Title tests splits a page type of at least 40 pages into two halves with a stable random hash, so a page never switches half. You download the list (CSV), change the title template on the “test” rows in your CMS, and enter the day the new titles went live. DirectoryLens doesn’t edit your site.

It compares the 28 days before go-live with the days from 7 days after it, using difference in differences on visits per day and on click rate, with a 95% range from resampling pages. The result is one sentence: waiting for Google, too early (under 14 days), too few pages (under 20 with impressions per half), likely helped, likely hurt, or no clear effect. One test runs per page type at a time, so two tests on the same pages can’t be confused. You can see the report on sample data in the live demo.

Questions

What is SEO A/B testing?

Changing something, such as the title template, on a random half of similar pages and comparing their organic traffic with the unchanged half. It splits pages, not visitors, because Google sees one version of each page.

What is the difference between A/A and A/B testing?

An A/A test splits pages and changes nothing, to measure normal noise and check the setup. An A/B test changes one half, and its result means something only if it is bigger than that noise.

Is AB testing worth it for SEO?

Yes when you have many pages on one template and a change you would otherwise roll out blind. Not for single pages or small, low-traffic page types, where no test can tell the difference.

Does SEO split testing break Google’s rules?

No, if you follow Google’s testing guidance: no cloaking, rel=canonical for alternate URLs, 302 rather than 301 redirects, and end the test once you have an answer.

How long should an SEO test run?

Until Google has recrawled the changed pages, plus at least two weeks of data after that; longer for page types with few visits. Don’t stop early because it looks good.

How many pages do I need for SEO split testing?

At least a few dozen with impressions in each half to see large effects, and hundreds to see small ones. Pages with almost no traffic add little.

Test your next title change on half the pages.

Connect Search Console, pick a page type, and DirectoryLens splits it, tracks both halves and tells you whether the change helped.

Explore the live demo →

5 days free · no card needed · ends on its own