Updated
“Blocked by robots.txt” means a rule in your robots.txt file stops Googlebot from crawling the URL, so Google can’t read the page and normally doesn’t index it. “Indexed, though blocked by robots.txt” is the opposite surprise: Google indexed the URL anyway, from links, without reading it. If you want the page in Google, remove the rule. If you want it out, allow crawling and use noindex instead.
Does robots.txt prevent indexing?
Not reliably. Robots.txt tells crawlers which URLs they may fetch. Google won’t crawl a disallowed URL, but if other pages link to it, Google can still index the URL itself, using the link text, and show it in results without a description. To keep a page out of Google, Google recommends a noindex rule or password protection (robots.txt introduction).
That gives two separate statuses in Search Console, with opposite fixes:
| Blocked by robots.txt | Indexed, though blocked by robots.txt | |
|---|---|---|
| Where it appears | Why pages aren’t indexed | Improve page experience (a warning) |
| Crawled? | No | No |
| Indexed? | No | Yes, from links, without the content |
| What searchers see | Nothing | Your URL with a very limited snippet |
| If you want it in Google | Remove the blocking rule | Remove the blocking rule |
| If you want it out of Google | Usually nothing; it is already out | Remove the block and add noindex |
Blocked by robots.txt: what it means
In the Page indexing report, Blocked by robots.txt lists URLs Google knows about but won’t crawl because of a rule in your robots.txt. Google’s help page calls it “URL blocked by robots.txt” and notes that the block doesn’t guarantee the page stays out of the index.
Often this is exactly what you want. Site search, account pages, cart-style actions and endless filter combinations don’t belong in Google, and blocking them saves crawling for pages that do. Google’s help itself says URLs blocked by robots.txt are probably blocked on purpose, so there may be nothing to validate.
How to fix the “Blocked by robots.txt” error
When the list includes pages you want indexed, follow Google’s unblock a page steps:
- Confirm the block. Inspect the URL in Search Console. Under Page indexing, Crawl allowed? shows whether robots.txt blocks it, and the page shows “Blocked by robots.txt”.
- Find the rule. Open
https://yourdomain.com/robots.txtand look for theDisallowline that matches the URL path. Rules match from the start of the path, soDisallow: /plumbersalso blocks/plumbers-supplyand/plumbers/austin-tx.*matches any characters and$marks the end of the URL. - Edit or remove the rule in the file, or in your CMS or host’s settings if they generate it. Re-test the URL with a robots.txt validator or URL Inspection’s live test.
- Tell Google. In the robots.txt report (Settings), open the file and choose Request a recrawl. Google normally caches robots.txt for up to 24 hours (robots.txt specification), so this helps when it’s urgent.
- Request indexing for the most important unblocked URLs, then Validate fix in the report.
Common accidental blocks:
Disallow: /left over from a staging site or a launch checklist.- A broad parameter rule such as
Disallow: /*?that also blocks?page=2and every paginated listing page. - A short prefix that catches more than intended, as in the
/plumbersexample above. - Blocked CSS, JavaScript or API paths. The page itself isn’t reported as blocked, but Google can’t render it, which can lead to soft 404s and missing content.
- Separate subdomains:
m.example.comandblog.example.comeach read their own robots.txt.
Indexed, though blocked by robots.txt
This warning means Google indexed URLs it is not allowed to crawl. Google’s help explains that it always respects robots.txt, but can still index a blocked page using information from pages that link to it; the snippet in results will probably be very limited. It usually turns up for pages blocked on purpose that something links to: filter URLs shared in emails, old URLs that external sites still link to, or private pages someone linked publicly.
How to fix “Indexed, though blocked by robots.txt”
- You want the page in Google: remove the robots.txt rule, request a recrawl of robots.txt, then request indexing for the URL.
- You want the page out of Google: remove the robots.txt rule and add
<meta name="robots" content="noindex">(or anX-Robots-Tag: noindexheader). Google has to crawl the page to see the noindex, which is why the block must go. Once Google has dropped the URLs you can decide whether to block them again to save crawling; leaving them crawlable with noindex is the safer default. - You need it gone fast: the Removals tool hides a URL from results for about six months (Removals help). Use that time to put the noindex in place.
- Truly private pages: put them behind a login. Google’s noindex documentation and robots.txt guide both point to password protection for content that must not be found.
Why isn’t my website indexed in Google Search Console?
When almost nothing is indexed, check these in order:
- A site-wide block.
Disallow: /underUser-agent: *stops all crawling. Look at your robots.txt first; it takes ten seconds. - A site-wide noindex, often a CMS setting left on from the build. See excluded by noindex tag.
- A login or password on the whole site, which shows up as 401 or 403.
- The wrong property. If you verified
http://example.combut the site runs onhttps://www.example.com, the report covers the wrong URLs. A Domain property covers every protocol and subdomain. - A new site. Google’s help says it can take a week or so to start crawling and indexing a new site. Submit a sitemap and request indexing for the home page, then check the URL Inspection tool for the home page’s status.
- No way in. Google finds pages through links and sitemaps. A site no other site links to, with no sitemap, can wait a long time. If pages are known but waiting, see discovered, currently not indexed.
How Google reads conflicting rules
Two rules from Google’s robots.txt specification explain most surprises:
- Only one group applies to a crawler: the group with the most specific matching user agent. If you have a
User-agent: Googlebotgroup, Googlebot ignores everything underUser-agent: *, so rules you meant for all crawlers must be repeated there. - The most specific rule wins, measured by the length of the path.
Allow: /plumbers/beatsDisallow: /plumbersfor/plumbers/austin-tx. When an Allow and a Disallow are equally specific, Google uses the less restrictive one.
How to find your robots.txt file
Robots.txt lives at the root of each protocol and host: take any URL, cut everything after the domain, and add /robots.txt. The file for https://www.example.com/plumbers/austin-tx is https://www.example.com/robots.txt. Subdomains don’t inherit it, and each page is governed by exactly one file (robots.txt report help). Google reads up to 500 KiB of it and ignores the rest.
On hosted platforms such as Wix, Shopify or Squarespace you may not be able to edit the file directly, because the platform generates it, and some of its blocks are deliberate. Google’s help suggests searching your host’s documentation for “robots.txt” plus the platform name before changing anything.
How to validate a robots.txt file
Search Console’s old robots.txt Tester was retired in late 2023 (Search Engine Land). What replaced it:
- The robots.txt report (Settings > robots.txt). It shows the robots.txt files Google found for the top 20 hosts of your property, when each was last fetched, its fetch status, size, and any parsing errors or warnings, plus previous versions from the last 30 days. It only works for Domain properties and URL-prefix properties at the root of a host (Google’s help).
- URL Inspection for a single URL: the live test tells you whether crawling is allowed.
- Google’s open-source robots.txt parser on GitHub, the same library Google Search uses, for testing rules locally before you deploy them.
- A third-party robots.txt validator, which Google’s help also suggests for finding the rule that blocks a URL.
Blocking on purpose: directory examples
On a directory, robots.txt is a crawl-budget tool. Search results, sort orders and filter combinations can multiply into millions of URLs that add nothing for searchers. Google’s faceted navigation guide suggests disallowing crawling of filtered URLs when you don’t need them in search. A typical file:
User-agent: *
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*view=map
Disallow: /*?*radius=
Disallow: /account/
Disallow: /claim-listing/
Sitemap: https://example.com/sitemap.xmlThese blocks belong under “Blocked by robots.txt” and you can leave them there. Four rules keep them from causing harm:
- Never block listing, city or category pages, or the paginated lists that lead to listings (
?page=). Google needs to crawl them to find your listings. See pagination and faceted navigation. - Don’t block URLs that are already indexed to get them out. Noindex them or canonicalise them first, wait until Google drops them, and only then block if you still need to save crawling.
- Don’t block a page and noindex it at the same time. The block hides the noindex. See excluded by noindex tag.
- List your sitemap in robots.txt, as above. The sitemap in robots.txt guide covers the syntax.
How DirectoryLens fits in
Google’s URL Inspection API reports whether robots.txt allows crawling for each URL, and DirectoryLens stores that with every inspection. In the Google indexing report, the Individual pages tab shows Google’s status for each inspected URL, so a template you blocked by mistake shows up as a page type full of “Blocked by robots.txt”. The URL changes check, which you run after changing URLs, reports old URLs that robots.txt blocks, since Google can’t follow a redirect it isn’t allowed to fetch. For the full list of statuses, see the page indexing issues guide.