Search Console guide

“Blocked by robots.txt” and “Indexed, though blocked”: what each means and how to fix it

Robots.txt stops crawling, not indexing. That one fact explains both statuses, and why the fix for each is different.

Explore the live demo →

5 days free · no card needed · ends on its own

Updated

“Blocked by robots.txt” means a rule in your robots.txt file stops Googlebot from crawling the URL, so Google can’t read the page and normally doesn’t index it. “Indexed, though blocked by robots.txt” is the opposite surprise: Google indexed the URL anyway, from links, without reading it. If you want the page in Google, remove the rule. If you want it out, allow crawling and use noindex instead.

The one thing to remember: robots.txt controls crawling, not indexing. Google’s robots.txt introduction says it is “not a mechanism for keeping a web page out of Google.” And a blocked page can’t show Google its noindex tag, because Google never fetches it.

Does robots.txt prevent indexing?

Not reliably. Robots.txt tells crawlers which URLs they may fetch. Google won’t crawl a disallowed URL, but if other pages link to it, Google can still index the URL itself, using the link text, and show it in results without a description. To keep a page out of Google, Google recommends a noindex rule or password protection (robots.txt introduction).

That gives two separate statuses in Search Console, with opposite fixes:

Blocked by robots.txtIndexed, though blocked by robots.txt
Where it appearsWhy pages aren’t indexedImprove page experience (a warning)
Crawled?NoNo
Indexed?NoYes, from links, without the content
What searchers seeNothingYour URL with a very limited snippet
If you want it in GoogleRemove the blocking ruleRemove the blocking rule
If you want it out of GoogleUsually nothing; it is already outRemove the block and add noindex

Blocked by robots.txt: what it means

In the Page indexing report, Blocked by robots.txt lists URLs Google knows about but won’t crawl because of a rule in your robots.txt. Google’s help page calls it “URL blocked by robots.txt” and notes that the block doesn’t guarantee the page stays out of the index.

Often this is exactly what you want. Site search, account pages, cart-style actions and endless filter combinations don’t belong in Google, and blocking them saves crawling for pages that do. Google’s help itself says URLs blocked by robots.txt are probably blocked on purpose, so there may be nothing to validate.

How to fix the “Blocked by robots.txt” error

When the list includes pages you want indexed, follow Google’s unblock a page steps:

  1. Confirm the block. Inspect the URL in Search Console. Under Page indexing, Crawl allowed? shows whether robots.txt blocks it, and the page shows “Blocked by robots.txt”.
  2. Find the rule. Open https://yourdomain.com/robots.txt and look for the Disallow line that matches the URL path. Rules match from the start of the path, so Disallow: /plumbers also blocks /plumbers-supply and /plumbers/austin-tx. * matches any characters and $ marks the end of the URL.
  3. Edit or remove the rule in the file, or in your CMS or host’s settings if they generate it. Re-test the URL with a robots.txt validator or URL Inspection’s live test.
  4. Tell Google. In the robots.txt report (Settings), open the file and choose Request a recrawl. Google normally caches robots.txt for up to 24 hours (robots.txt specification), so this helps when it’s urgent.
  5. Request indexing for the most important unblocked URLs, then Validate fix in the report.

Common accidental blocks:

Indexed, though blocked by robots.txt

This warning means Google indexed URLs it is not allowed to crawl. Google’s help explains that it always respects robots.txt, but can still index a blocked page using information from pages that link to it; the snippet in results will probably be very limited. It usually turns up for pages blocked on purpose that something links to: filter URLs shared in emails, old URLs that external sites still link to, or private pages someone linked publicly.

How to fix “Indexed, though blocked by robots.txt”

Why isn’t my website indexed in Google Search Console?

When almost nothing is indexed, check these in order:

  1. A site-wide block. Disallow: / under User-agent: * stops all crawling. Look at your robots.txt first; it takes ten seconds.
  2. A site-wide noindex, often a CMS setting left on from the build. See excluded by noindex tag.
  3. A login or password on the whole site, which shows up as 401 or 403.
  4. The wrong property. If you verified http://example.com but the site runs on https://www.example.com, the report covers the wrong URLs. A Domain property covers every protocol and subdomain.
  5. A new site. Google’s help says it can take a week or so to start crawling and indexing a new site. Submit a sitemap and request indexing for the home page, then check the URL Inspection tool for the home page’s status.
  6. No way in. Google finds pages through links and sitemaps. A site no other site links to, with no sitemap, can wait a long time. If pages are known but waiting, see discovered, currently not indexed.

How Google reads conflicting rules

Two rules from Google’s robots.txt specification explain most surprises:

How to find your robots.txt file

Robots.txt lives at the root of each protocol and host: take any URL, cut everything after the domain, and add /robots.txt. The file for https://www.example.com/plumbers/austin-tx is https://www.example.com/robots.txt. Subdomains don’t inherit it, and each page is governed by exactly one file (robots.txt report help). Google reads up to 500 KiB of it and ignores the rest.

On hosted platforms such as Wix, Shopify or Squarespace you may not be able to edit the file directly, because the platform generates it, and some of its blocks are deliberate. Google’s help suggests searching your host’s documentation for “robots.txt” plus the platform name before changing anything.

How to validate a robots.txt file

Search Console’s old robots.txt Tester was retired in late 2023 (Search Engine Land). What replaced it:

Blocking on purpose: directory examples

On a directory, robots.txt is a crawl-budget tool. Search results, sort orders and filter combinations can multiply into millions of URLs that add nothing for searchers. Google’s faceted navigation guide suggests disallowing crawling of filtered URLs when you don’t need them in search. A typical file:

User-agent: *
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*view=map
Disallow: /*?*radius=
Disallow: /account/
Disallow: /claim-listing/

Sitemap: https://example.com/sitemap.xml

These blocks belong under “Blocked by robots.txt” and you can leave them there. Four rules keep them from causing harm:

How DirectoryLens fits in

Google’s URL Inspection API reports whether robots.txt allows crawling for each URL, and DirectoryLens stores that with every inspection. In the Google indexing report, the Individual pages tab shows Google’s status for each inspected URL, so a template you blocked by mistake shows up as a page type full of “Blocked by robots.txt”. The URL changes check, which you run after changing URLs, reports old URLs that robots.txt blocks, since Google can’t follow a redirect it isn’t allowed to fetch. For the full list of statuses, see the page indexing issues guide.

Questions

Does robots.txt prevent indexing?

No. It stops Google crawling a URL, but Google can still index the URL from links to it, without its content. To keep a page out of Google, use noindex or password protection.

How do I fix a “Blocked by robots.txt” error?

Inspect the URL to confirm the block, find the Disallow rule in your robots.txt that matches its path, edit or remove it, request a recrawl in the robots.txt report, then request indexing for the key pages.

Why isn’t my website indexed in Google Search Console?

Check the Page indexing report for the reason. A site-wide Disallow: / in robots.txt, a noindex left on from staging, or a login wall are the usual causes when nothing at all is indexed; a new site may simply not have been crawled yet.

How do I find my robots.txt file?

It is at the root of the host: https://yourdomain.com/robots.txt. Each subdomain and protocol has its own file.

How do I validate a robots.txt file?

Use the robots.txt report in Search Console settings for fetch status and parsing errors, URL Inspection to test a single URL, or Google’s open-source robots.txt parser to test rules before you deploy them.

How long does Google take to notice a robots.txt change?

Google generally caches robots.txt for up to 24 hours. You can ask for a faster recrawl from the robots.txt report.

Should I block pages in robots.txt that I have noindexed?

No. If Google can’t crawl a page, it never sees the noindex, and the URL can stay indexed from links.

Catch the templates you blocked by mistake.

Connect Search Console and DirectoryLens shows Google’s status for every inspected page, grouped by page type.

Explore the live demo →

5 days free · no card needed · ends on its own