Question: How can I use robots.txt to reference a sitemap?

Robots.txt is a file that webmasters create to instruct web robots (typically search engine robots) how to crawl and index pages on their website. It's possible to specify the location of your website's XML Sitemap within the robots.txt file, enabling search engines to easily locate it.

To reference your Sitemap in robots.txt, add the following line at the end of your robots.txt file:

Sitemap: http://www.example.com/sitemap_location.xml

Replace http://www.example.com/sitemap_location.xml with the actual URL of your XML Sitemap. This simple instruction tells any compliant web robot where your sitemap is located.

Please note, the 'Sitemap' directive is independent of the user-agent line, so it doesn't matter where you place it in your robots.txt file. Also, if you have multiple sitemaps, you can add multiple 'Sitemap' directives.

Here is an example of a robots.txt file:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
Sitemap: http://www.example.com/sitemap.xml

In this example, several directories are excluded from being crawled by all bots ('User-agent: *'), and the location of the sitemap is specified.