Sitemap in robots.txt

How to add your sitemap to robots.txt

The exact syntax, examples for WordPress and large directories, sitemaps on other hosts, and why the robots.txt line doesn’t replace Search Console.

Explore the live demo →

5 days free · no card needed · ends on its own

Updated

To add your sitemap to robots.txt, add a line with Sitemap: followed by the sitemap’s full URL, for example Sitemap: https://www.example.com/sitemap.xml, anywhere in the robots.txt file at the root of your site. Add one line per sitemap, or a single line for your sitemap index. It helps Google, Bing and other crawlers find the sitemap, but it doesn’t replace submitting the sitemap in Search Console, which is where you see whether Google could read it.

User-agent: *
Disallow: /search/

Sitemap: https://www.example.com/sitemap_index.xml

The exact syntax

Google’s robots.txt specification defines the sitemap line as sitemap: [absoluteURL]. The rules:

RuleDetail
Full URL onlyInclude the protocol and host: https://www.example.com/sitemap.xml, never /sitemap.xml.
Anywhere in the fileThe sitemap line isn’t part of any user-agent group. Top, bottom or between groups all work; the bottom is the convention.
Applies to every crawlerPutting it under User-agent: Googlebot doesn’t limit it to Google. Every crawler that supports sitemaps reads it.
As many as you needUse one Sitemap: line per sitemap or sitemap index. There is no set limit, though the whole file must stay under Google’s 500 KiB limit.
Other hosts allowedThe sitemap can sit on a different host from the robots.txt file, such as a CDN subdomain (see below).
CapitalizationGoogle’s own examples use both Sitemap: and sitemap:. Keep the URL exactly as it is served; URLs are case-sensitive.

robots.txt itself must be a plain-text UTF-8 file named exactly robots.txt at the root of the host, such as https://www.example.com/robots.txt (how to create one). Its rules apply only to that host, protocol and port, so https://shop.example.com needs its own file.

Examples

One sitemap

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Several sitemaps

Sitemap: https://www.example.com/sitemap-pages.xml
Sitemap: https://www.example.com/sitemap-posts.xml
Sitemap: https://www.example.com/sitemap-listings.xml

A sitemap index (recommended for larger sites)

List the index, not every child sitemap. Crawlers read the index and follow it to each child.

Sitemap: https://www.example.com/sitemap_index.xml

WordPress with Yoast SEO or Rank Math

Both plugins publish a sitemap index at /sitemap_index.xml and serve a virtual robots.txt. Rank Math adds the sitemap line by default; Yoast adds it in the robots.txt it creates. Edit robots.txt in Yoast SEO > Tools > File editor or in Rank Math > General Settings > Edit robots.txt. A real robots.txt file uploaded to the site root overrides the virtual one, so if your edits don’t show up, look for that file.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap_index.xml

WordPress without an SEO plugin

WordPress’s built-in sitemap lives at /wp-sitemap.xml and is listed in WordPress’s virtual robots.txt automatically. When an SEO plugin takes over sitemaps, it usually turns this one off; make sure robots.txt points to the sitemap that actually exists.

A directory with many sitemaps

User-agent: *
Disallow: /search
Disallow: /*?sort=
Disallow: /account/

# One index that lists every sitemap below
Sitemap: https://www.example.com/sitemap_index.xml

# The index lists, for example:
#   /sitemap-listings-1.xml ... /sitemap-listings-12.xml
#   /sitemap-cities.xml
#   /sitemap-categories.xml
#   /sitemap-city-categories.xml

The disallow rules keep crawlers out of internal search and sort URLs, while the sitemap points them at the canonical listing, city and category pages. Make sure the sitemaps never list a URL that robots.txt blocks; Google reports that as an error.

Sitemaps on another host or a CDN

If your sitemaps are generated on a separate host, such as https://sitemaps.example.com or a storage bucket behind a CDN, reference them from the robots.txt of the site whose URLs they list. Google’s sitemap guide gives exactly this setup:

# robots.txt of https://www.example.com/
Sitemap: https://sitemaps.example.com/sitemap-www-example-com.xml

Each sitemap should list only URLs from the site whose robots.txt points to it. The sitemaps protocol treats that robots.txt reference as proof that you control the site. If you submit a cross-host sitemap in Search Console instead, Google asks you to verify ownership of every site whose URLs it lists.

Sitemap vs robots.txt: what each file does

The two files are often confused because they sit side by side, but they answer different questions:

robots.txtXML sitemap
Question it answersWhich URLs may crawlers request?Which URLs do you want crawled and indexed, and when did they change?
LocationExactly /robots.txt at the root of each hostAnywhere; listed in robots.txt and submitted in Search Console
FormatPlain text rulesXML (also plain text or RSS)
Size limit500 KiB read by Google50,000 URLs or 50MB uncompressed per file
Keeps a page out of Google’s index?No: a blocked URL can still be indexed from linksNo: leaving a URL out doesn’t stop indexing

To keep a page out of search results, use a noindex tag and let Google crawl it (see excluded by noindex). robots.txt controls crawling; the sitemap suggests what to crawl; neither controls indexing on its own.

Does it replace submitting in Search Console?

No. Do both. They do different jobs:

Sitemap line in robots.txtSubmitted in Search Console
Helps Google find the sitemapYesYes
Read by Bing and other crawlersYesNo (submit in Bing Webmaster Tools too)
Shows read status and errorsNoYes, in the Sitemaps report
Shows how many URLs Google foundNoYes, as discovered pages
Filter Page indexing by sitemapNoYes
Listed in URL Inspection under SitemapsNoYes
Needs Search Console owner accessNoYes

Google retired its sitemap ping endpoint in 2023 and named robots.txt and Search Console as the ways to submit sitemaps. Bing recommends listing the sitemap in robots.txt and submitting it in Bing Webmaster Tools, and checks submitted sitemaps typically at least once a day. See Bing indexing for IndexNow.

Common mistakes

Test it after every change

  1. Open https://www.example.com/robots.txt in a private window and confirm the Sitemap: line is there. On WordPress, this also tells you whether a physical file is overriding the plugin’s virtual one.
  2. Open each sitemap URL from that line and check it loads as XML, not a 404 or an HTML page. The free sitemap checker does this for you from just the domain: it reads robots.txt, follows the sitemap line, validates the XML and tests a sample of the URLs listed.
  3. In Search Console, open the robots.txt report and check Google fetched the new version. If not, request a recrawl from the report.
  4. Submit the same sitemap in Indexing > Sitemaps and, after Google reads it, check its status and discovered pages. If it says Couldn’t fetch, follow the sitemap read checklist.
  5. Inspect one important URL with the URL Inspection tool and check that Crawl allowed? says Yes.

Is robots.txt still used?

Yes. Google reads it before crawling a site and generally caches it for up to 24 hours, so changes can take a day to apply. Google supports four fields: user-agent, allow, disallow and sitemap; others such as crawl-delay are ignored by Google. If robots.txt returns a 4xx error, Google treats the site as having no restrictions. If it returns a 5xx error, Google stops crawling for a while and then falls back to the last version it fetched.

How to find a site’s robots.txt and sitemap

On directory websites

A directory’s robots.txt and sitemaps should tell the same story: robots.txt keeps crawlers out of the URLs that multiply without adding value (internal search results, sort orders, account pages), and the sitemap index lists the pages you want indexed, split by page type. Splitting matters: with listings, city pages and category-in-city pages in separate sitemaps, the Page indexing report filtered by sitemap shows which template Google is leaving out. Keep each listing sitemap well under 50,000 URLs, and update lastmod only when a listing really changes. A sitemap that lists every listing doesn’t get them crawled on its own; listings Google knows about but hasn’t reached show as discovered, not indexed, and internal links are the fix.

How DirectoryLens helps

DirectoryLens reads the sitemaps you submitted in Search Console, not ones found only through robots.txt, which is one more reason to submit them. The Sitemaps tab in Google indexing re-reads each one daily, follows sitemap indexes, shows URL counts per sitemap over time, flags any it couldn’t load, and alerts you when a sitemap or the total loses a set share of its URLs within a week. It also counts pages that get visits from Google but aren’t in any sitemap, which are worth adding. Those sitemap URLs then feed the daily URL inspections, so you see how much of each page type Google has indexed. For every other indexing status, see the page indexing issues guide.

Questions

How do I add my sitemap to robots.txt?

Add a line with Sitemap: followed by the full sitemap URL, such as Sitemap: https://www.example.com/sitemap.xml, to the robots.txt file at the root of your site. Use the full URL, including https://.

Where should the sitemap line go in robots.txt?

Anywhere. It isn’t tied to any user-agent group, so crawlers read it wherever it is. Most sites put it at the end of the file.

Can I list more than one sitemap in robots.txt?

Yes. Add one Sitemap: line per sitemap, or list a single sitemap index that points to all of them. Google sets no limit on the number of lines.

Do I still need to submit my sitemap in Search Console?

Yes. The robots.txt line helps crawlers find it, but only Search Console shows whether Google could read the sitemap, how many URLs it found, and any errors.

How do I get my sitemap URL?

Check the Sitemap: line in your robots.txt, then try /sitemap.xml, /sitemap_index.xml (Yoast and Rank Math) or /wp-sitemap.xml (WordPress). Search Console’s Sitemaps report lists what has been submitted.

How do I add a robots.txt file?

Create a plain-text UTF-8 file named robots.txt and upload it to the root of your site so it loads at /robots.txt. On WordPress, edit the virtual file through Yoast’s File editor or Rank Math’s Edit robots.txt setting.

Is robots.txt still used?

Yes. Google, Bing and other crawlers read it before crawling. Google supports the user-agent, allow, disallow and sitemap fields and generally caches the file for up to 24 hours.

How do I get the robots.txt file of a website?

Add /robots.txt to the site’s domain, for example https://www.example.com/robots.txt. For your own site, Search Console’s robots.txt report shows the version Google last fetched.

See every sitemap Google should be reading.

Connect Search Console and DirectoryLens re-reads your submitted sitemaps daily and shows how much of each page type Google has indexed.

Explore the live demo →

5 days free · no card needed · ends on its own