Updated
To add your sitemap to robots.txt, add a line with Sitemap: followed by the sitemap’s full URL, for example Sitemap: https://www.example.com/sitemap.xml, anywhere in the robots.txt file at the root of your site. Add one line per sitemap, or a single line for your sitemap index. It helps Google, Bing and other crawlers find the sitemap, but it doesn’t replace submitting the sitemap in Search Console, which is where you see whether Google could read it.
User-agent: *
Disallow: /search/
Sitemap: https://www.example.com/sitemap_index.xmlThe exact syntax
Google’s robots.txt specification defines the sitemap line as sitemap: [absoluteURL]. The rules:
| Rule | Detail |
|---|---|
| Full URL only | Include the protocol and host: https://www.example.com/sitemap.xml, never /sitemap.xml. |
| Anywhere in the file | The sitemap line isn’t part of any user-agent group. Top, bottom or between groups all work; the bottom is the convention. |
| Applies to every crawler | Putting it under User-agent: Googlebot doesn’t limit it to Google. Every crawler that supports sitemaps reads it. |
| As many as you need | Use one Sitemap: line per sitemap or sitemap index. There is no set limit, though the whole file must stay under Google’s 500 KiB limit. |
| Other hosts allowed | The sitemap can sit on a different host from the robots.txt file, such as a CDN subdomain (see below). |
| Capitalization | Google’s own examples use both Sitemap: and sitemap:. Keep the URL exactly as it is served; URLs are case-sensitive. |
robots.txt itself must be a plain-text UTF-8 file named exactly robots.txt at the root of the host, such as https://www.example.com/robots.txt (how to create one). Its rules apply only to that host, protocol and port, so https://shop.example.com needs its own file.
Examples
One sitemap
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlSeveral sitemaps
Sitemap: https://www.example.com/sitemap-pages.xml
Sitemap: https://www.example.com/sitemap-posts.xml
Sitemap: https://www.example.com/sitemap-listings.xmlA sitemap index (recommended for larger sites)
List the index, not every child sitemap. Crawlers read the index and follow it to each child.
Sitemap: https://www.example.com/sitemap_index.xmlWordPress with Yoast SEO or Rank Math
Both plugins publish a sitemap index at /sitemap_index.xml and serve a virtual robots.txt. Rank Math adds the sitemap line by default; Yoast adds it in the robots.txt it creates. Edit robots.txt in Yoast SEO > Tools > File editor or in Rank Math > General Settings > Edit robots.txt. A real robots.txt file uploaded to the site root overrides the virtual one, so if your edits don’t show up, look for that file.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap_index.xmlWordPress without an SEO plugin
WordPress’s built-in sitemap lives at /wp-sitemap.xml and is listed in WordPress’s virtual robots.txt automatically. When an SEO plugin takes over sitemaps, it usually turns this one off; make sure robots.txt points to the sitemap that actually exists.
A directory with many sitemaps
User-agent: *
Disallow: /search
Disallow: /*?sort=
Disallow: /account/
# One index that lists every sitemap below
Sitemap: https://www.example.com/sitemap_index.xml
# The index lists, for example:
# /sitemap-listings-1.xml ... /sitemap-listings-12.xml
# /sitemap-cities.xml
# /sitemap-categories.xml
# /sitemap-city-categories.xmlThe disallow rules keep crawlers out of internal search and sort URLs, while the sitemap points them at the canonical listing, city and category pages. Make sure the sitemaps never list a URL that robots.txt blocks; Google reports that as an error.
Sitemaps on another host or a CDN
If your sitemaps are generated on a separate host, such as https://sitemaps.example.com or a storage bucket behind a CDN, reference them from the robots.txt of the site whose URLs they list. Google’s sitemap guide gives exactly this setup:
# robots.txt of https://www.example.com/
Sitemap: https://sitemaps.example.com/sitemap-www-example-com.xmlEach sitemap should list only URLs from the site whose robots.txt points to it. The sitemaps protocol treats that robots.txt reference as proof that you control the site. If you submit a cross-host sitemap in Search Console instead, Google asks you to verify ownership of every site whose URLs it lists.
Sitemap vs robots.txt: what each file does
The two files are often confused because they sit side by side, but they answer different questions:
| robots.txt | XML sitemap | |
|---|---|---|
| Question it answers | Which URLs may crawlers request? | Which URLs do you want crawled and indexed, and when did they change? |
| Location | Exactly /robots.txt at the root of each host | Anywhere; listed in robots.txt and submitted in Search Console |
| Format | Plain text rules | XML (also plain text or RSS) |
| Size limit | 500 KiB read by Google | 50,000 URLs or 50MB uncompressed per file |
| Keeps a page out of Google’s index? | No: a blocked URL can still be indexed from links | No: leaving a URL out doesn’t stop indexing |
To keep a page out of search results, use a noindex tag and let Google crawl it (see excluded by noindex). robots.txt controls crawling; the sitemap suggests what to crawl; neither controls indexing on its own.
Does it replace submitting in Search Console?
No. Do both. They do different jobs:
| Sitemap line in robots.txt | Submitted in Search Console | |
|---|---|---|
| Helps Google find the sitemap | Yes | Yes |
| Read by Bing and other crawlers | Yes | No (submit in Bing Webmaster Tools too) |
| Shows read status and errors | No | Yes, in the Sitemaps report |
| Shows how many URLs Google found | No | Yes, as discovered pages |
| Filter Page indexing by sitemap | No | Yes |
| Listed in URL Inspection under Sitemaps | No | Yes |
| Needs Search Console owner access | No | Yes |
Google retired its sitemap ping endpoint in 2023 and named robots.txt and Search Console as the ways to submit sitemaps. Bing recommends listing the sitemap in robots.txt and submitting it in Bing Webmaster Tools, and checks submitted sitemaps typically at least once a day. See Bing indexing for IndexNow.
Common mistakes
- A relative URL:
Sitemap: /sitemap.xmlisn’t valid. Use the full URL. - The old protocol or host after a migration:
Sitemap: http://example.com/sitemap.xmlon a site that moved tohttps://www.example.com. It redirects at best, and points at the wrong property in Search Console. - Pointing at a sitemap that no longer exists, such as
/sitemap.xmlafter an SEO plugin moved it to/sitemap_index.xml. Open every sitemap URL in robots.txt and check it loads. - Blocking the sitemap itself with a rule like
Disallow: /*.xml. That can lead to “Couldn’t fetch”. - Thinking the line is scoped to a user-agent. It isn’t; it applies to every crawler.
- Putting robots.txt in a subfolder such as
/blog/robots.txt. Crawlers only read the file at the root of the host. - Leaving a staging rule live:
Disallow: /copied from a staging site blocks everything, sitemap included. See blocked by robots.txt. - Listing URLs in the sitemap that robots.txt blocks. Pick one: either the URL should be crawled and listed, or blocked and left out.
Test it after every change
- Open
https://www.example.com/robots.txtin a private window and confirm theSitemap:line is there. On WordPress, this also tells you whether a physical file is overriding the plugin’s virtual one. - Open each sitemap URL from that line and check it loads as XML, not a 404 or an HTML page. The free sitemap checker does this for you from just the domain: it reads robots.txt, follows the sitemap line, validates the XML and tests a sample of the URLs listed.
- In Search Console, open the robots.txt report and check Google fetched the new version. If not, request a recrawl from the report.
- Submit the same sitemap in Indexing > Sitemaps and, after Google reads it, check its status and discovered pages. If it says Couldn’t fetch, follow the sitemap read checklist.
- Inspect one important URL with the URL Inspection tool and check that Crawl allowed? says Yes.
Is robots.txt still used?
Yes. Google reads it before crawling a site and generally caches it for up to 24 hours, so changes can take a day to apply. Google supports four fields: user-agent, allow, disallow and sitemap; others such as crawl-delay are ignored by Google. If robots.txt returns a 4xx error, Google treats the site as having no restrictions. If it returns a 5xx error, Google stops crawling for a while and then falls back to the last version it fetched.
How to find a site’s robots.txt and sitemap
- robots.txt: add
/robots.txtto the domain, for examplehttps://www.example.com/robots.txt. For your own site, Search Console’s robots.txt report (in Settings) shows the version Google fetched, when, and any parsing problems, and lets you request a recrawl after a fix. - The sitemap URL: look for the
Sitemap:line in robots.txt first. If there isn’t one, try/sitemap.xml,/sitemap_index.xml(Yoast, Rank Math) and/wp-sitemap.xml(WordPress). For your own site, the Sitemaps report in Search Console lists what has been submitted.
On directory websites
A directory’s robots.txt and sitemaps should tell the same story: robots.txt keeps crawlers out of the URLs that multiply without adding value (internal search results, sort orders, account pages), and the sitemap index lists the pages you want indexed, split by page type. Splitting matters: with listings, city pages and category-in-city pages in separate sitemaps, the Page indexing report filtered by sitemap shows which template Google is leaving out. Keep each listing sitemap well under 50,000 URLs, and update lastmod only when a listing really changes. A sitemap that lists every listing doesn’t get them crawled on its own; listings Google knows about but hasn’t reached show as discovered, not indexed, and internal links are the fix.
How DirectoryLens helps
DirectoryLens reads the sitemaps you submitted in Search Console, not ones found only through robots.txt, which is one more reason to submit them. The Sitemaps tab in Google indexing re-reads each one daily, follows sitemap indexes, shows URL counts per sitemap over time, flags any it couldn’t load, and alerts you when a sitemap or the total loses a set share of its URLs within a week. It also counts pages that get visits from Google but aren’t in any sitemap, which are worth adding. Those sitemap URLs then feed the daily URL inspections, so you see how much of each page type Google has indexed. For every other indexing status, see the page indexing issues guide.