Updated
SEO A/B testing (or SEO split testing) means changing something, usually the title template, on a random half of a group of similar pages, leaving the other half alone, and comparing how the two halves’ organic visits change. Unlike a conversion A/B test, you split pages, not visitors, because Google sees only one version of each page. It works on any site with dozens of similar pages, and the control half is what makes the result trustworthy.
Page-split vs user-split testing
| User-split (CRO) test | Page-split (SEO) test | |
|---|---|---|
| What is split | Visitors to the same page | Pages of the same type |
| What Google sees | One version (Googlebot doesn’t keep cookies) | Version A on some pages, version B on others |
| Measures | What visitors do on the page | Rankings, impressions and clicks from search |
| Needs | Traffic to one page | Many similar pages: listings, city or category pages, products |
| Typical tools | Optimizely, VWO, your own code | Search Console data and a random split of URLs |
A user-split test can’t tell you whether a title ranks or gets clicked better, because Google only ever indexes one title per page. To test SEO, the pages themselves have to differ, and you compare groups of pages.
Google’s rules for testing
Google allows testing and asks for a few things (Google’s guide to website testing):
- No cloaking. Don’t show Googlebot different content than visitors. Every visitor and Googlebot should see the same version of a given page.
- For tests on separate URLs, use rel=canonical pointing to the original, rather than noindex.
- For redirect-based tests, use a 302 (temporary), not a 301, so the original URL stays indexed.
- Run it only as long as needed, then remove the test and its alternate URLs.
A page-split title test fits all four: each page has one title that everyone sees, there are no alternate URLs, and you either keep the change or revert it.
How to run a title split test
- Pick one page type with many similar pages. City pages, category-in-city pages, listings. They must share a template, so the only difference is your change. DirectoryLens needs at least 40 pages and at least 20 pages with impressions in each half; below that, normal page-to-page differences swamp most effects.
- Split the pages at random, and freeze the split. Don’t split alphabetically or by ID ranges: alphabetical groups can differ in size of city, and older IDs are often older pages. In a spreadsheet, add a random number to each URL, sort, take the top half, and save it. Never move a page between halves.
- Check the halves match before you start. Compare their clicks per day over the last 28 days. They won’t be identical, which is why the method compares each half with its own past.
- Change the template on the test half only, usually with a condition in your CMS on the list of test URLs. Write down the day it went live.
- Wait for Google to recrawl. Titles in search results update only after Google has crawled the pages again. DirectoryLens starts measuring 7 days after the go-live date.
- Measure for at least 14 days, then compare (below). Don’t stop the day it looks good; early differences often fade.
- Roll out, or revert. Then test the next idea.
Measure it with difference in differences
A plain before-and-after can’t separate your change from everything else: a holiday, a Google update, a competitor dropping out. The unchanged half goes through all of that too, so compare each half’s change:
effect = (test after / test before) / (control after / control before) - 1Use visits per day in each window, so windows of different lengths work. For example: the test half goes from 100 to 120 visits a day (+20%), the control half from 100 to 110 (+10%). The effect is 1.20 ÷ 1.10 − 1 = about +9%. The season added 10% to everything; your titles added about 9% on top.
Then ask how sure you can be. Pages differ a lot from each other, so the honest way is to resample pages: draw each half’s pages again with replacement a few hundred times, recompute the effect each time, and take the middle 95% of results as the range. If the whole range is above zero, the change likely helped; if it is all below zero, it likely hurt; if it spans zero, you haven’t shown a difference.
You can do this from Search Console without a testing tool. Filter the Pages dimension with a regex matching each half’s URLs (Custom (regex) filters), or pull page-level daily data through the API, and sum clicks per half per window.
What is the difference between A/A and A/B testing?
An A/A test splits pages into two halves and changes nothing. Any difference you see is noise, so it shows how big a difference your pages produce by chance, and catches mistakes in how you split or measure. If an A/A test on your city pages swings ±8% in a month, a title test showing +5% proves nothing. An A/B test is the real thing: one half gets the change.
Sample size and how long to run
The honest answer for most sites: you can detect big effects, not small ones. Two things decide what you can see:
- Number of pages with impressions. More pages make the range narrower. With 20 pages a half you will usually only see large changes; with several hundred you can see smaller ones.
- Visits per page. Pages with a handful of visits a month add little information. A page type with 500 pages but 30 visits a month in total can’t settle a test.
Run until you have at least 14 days after the recrawl, longer for page types with few visits, and avoid ending a test during a known Google update or a holiday that hits the halves differently. Then stop: Google’s testing guidance asks you not to leave tests running longer than needed.
What you can and can’t test this way
| Change | Testable with a page split? | Note |
|---|---|---|
| Title template | Yes | The most common test; shows up after a recrawl. |
| Meta description template | Yes | Google may not use it for every search, which dilutes the effect. |
| H1 and intro text | Yes | Can affect both ranking and the title Google shows. |
| Internal link blocks (nearby, related) | Yes, with care | Links from test pages also reach control pages, which blurs the split. |
| Structured data | Yes | Rich results can take weeks to appear. |
| Site-wide changes (speed, navigation, domain) | No | Every page changes at once, so there is no control half. |
| Backlinks | No | You don’t control which pages get them. |
Common mistakes
- Non-random halves. Testing on “the big cities” and comparing with “the rest” measures city size, not your title.
- Changing the control half too. A site-wide release during the test (new navigation, a URL change) hits both halves unevenly.
- Two tests on the same pages. You can’t tell which change did what.
- Peeking and stopping. Checking daily and stopping on the first good day picks out noise.
- Counting the recrawl days. Until Google has recrawled the pages, searchers still see the old titles.
- Reading “no clear effect” as “no effect”. It means the test couldn’t tell. A bigger page type or a longer run might.
Is SEO A/B testing worth it?
Yes, when you have many pages on one template and a change you would otherwise roll out blind. A title template change on 900 city pages can lose as many visits as it gains, and you won’t see that in a before-and-after. It is not worth it for a single page (there is no control), for a page type with a few dozen low-traffic pages, or for changes so small no realistic test could detect them. For one important page, change it, annotate the date, and watch it against similar pages.
What to test first on a directory
- Listing counts in titles: “Plumbers in Austin, TX” vs “14 Plumbers in Austin, TX”.
- The state: “Springfield” exists in many states; does adding “IL” help or just lengthen the title?
- Order: service first or city first.
- Year or freshness words: some searches reward them, some don’t.
- Listing titles: business name only vs name plus main service and city.
Pick ideas from your own data first. The click rate check shows which page types are clicked below their position, and striking distance keywords show which words their titles miss.
How DirectoryLens runs title tests
Quick wins > Title tests splits a page type of at least 40 pages into two halves with a stable random hash, so a page never switches half. You download the list (CSV), change the title template on the “test” rows in your CMS, and enter the day the new titles went live. DirectoryLens doesn’t edit your site.
It compares the 28 days before go-live with the days from 7 days after it, using difference in differences on visits per day and on click rate, with a 95% range from resampling pages. The result is one sentence: waiting for Google, too early (under 14 days), too few pages (under 20 with impressions per half), likely helped, likely hurt, or no clear effect. One test runs per page type at a time, so two tests on the same pages can’t be confused. You can see the report on sample data in the live demo.