--- title: Google's New Robots.txt Sitemap description: Via the Google Blog, more stats and analysis of robots.txt files were released in Google's Sitem... image: https://www.websitemagazine.com/hubfs/Imported_Blog_Media/gradient-regular-logo-mark-Jan-23-2023-12-06-32-2984-AM-Apr-10-2023-01-17-22-0561-PM-Apr-25-2023-03-25-59-9070-AM-2.png --- [Log In](https://www.websitemagazine.com/_hcms/mem/login) [Subscribe](https://websitemagazine.tradepub.com/free/w_webk05/prgm.cgi) [![Website Magazine Logo](https://www.websitemagazine.com/hs-fs/hubfs/gradient-regular@3x-1.png?width=586&height=145&name=gradient-regular@3x-1.png "Website Magazine Logo")](http://Website%20Services,%20Inc.) - [About](https://www.websitemagazine.com/about) - [FAQs](https://www.websitemagazine.com/about/faq) - [Advertise](https://www.websitemagazine.com/about/advertise) - [Editorial Staff](https://www.websitemagazine.com/about/editorial-publishing-staff) - [Resources](https://websitemagazine.tradepub.com) - [Newsletter](https://www.websitemagazine.com/newsletter-sign-up) - More - [Contact Us](https://www.websitemagazine.com/contact) [Log In](https://www.websitemagazine.com/_hcms/mem/login) [Subscribe](https://websitemagazine.tradepub.com/free/w_webk05/prgm.cgi) Search - [Latest](https://www.websitemagazine.com) - [Design](https://www.websitemagazine.com/web-design) - [Development](https://www.websitemagazine.com/web-development) - [SEO](https://www.websitemagazine.com/seo) - [Ecommerce](https://www.websitemagazine.com/ecommerce) - [Content](https://www.websitemagazine.com/content) - [Marketing](https://www.websitemagazine.com/marketing) - [Advertising](https://www.websitemagazine.com/advertising) - [Email](https://www.websitemagazine.com/email) - [Infographics](https://www.websitemagazine.com/infographics) - [View All Topics](https://www.websitemagazine.com/channels) # Google's New Robots.txt Sitemap ![Peter Devereaux](https://www.websitemagazine.com/hubfs/Peter_Devereaux.png) [by Peter Devereaux](https://www.websitemagazine.com/marketing/author/peter-devereaux) 07 Feb, 2006 - - - [mailto:?Body=https://www.websitemagazine.com/marketing/google-sitemaps-robots](mailto:?Body=https://www.websitemagazine.com/marketing/google-sitemaps-robots) - Link Copied! ![Google's New Robots.txt Sitemap](https://www.websitemagazine.com/hubfs/Imported_Blog_Media/gradient-regular-logo-mark-Jan-23-2023-12-06-32-2984-AM-Apr-10-2023-01-17-22-0561-PM-Apr-25-2023-03-25-59-9070-AM-2.png) Via the [Google Blog](https://sitemaps.blogspot.com/2006/02/more-stats-and-analysis-of-robotstxt.html), more stats and analysis of robots.txt files were released in Google's Sitemaps feature. If your site has a robots.txt file, the new *robots.txt tab* provides Googlebot's view of that file, including when the spider last accessed it, the status it returns ([404 error if it does not exist](https://www.websitemagazine.com/definition/404-error)), and if it blocks access to your home page. It also shows any syntax errors in the file. ![analysis of cached robots.txt](https://www.websitemagazine.com/hs-fs/hubfs/Imported_Blog_Media/google-sitemap1-3-1-1-1.jpg?width=400&height=136&name=google-sitemap1-3-1-1-1.jpg) You can enter a list of URLs to see if the robots.txt file allows or blocks them. ![Blocked and Allowed URLs in Robots.txt file](https://www.websitemagazine.com/hs-fs/hubfs/Imported_Blog_Media/google-sitemap2-3-1-1-1.jpg?width=400&height=288&name=google-sitemap2-3-1-1-1.jpg) You can also test changes to your robots.txt file by entering and then testing them against the Googlebot user-agent, other Google user agents, or the Robots Standard. This lets Google Sitemap users experiment with changes to see how they would impact the crawl of your site, as well as make sure there are no errors in the file, before making changes to the file on your site. If you don't have a robots.txt file, you can use this page to test a potential robots.txt file before you add it to your site. **More stats** Crawl stats now include the page on your site that had the highest PageRank, by month, for the last three months. ![Page of your site with the highest PageRank](https://www.websitemagazine.com/hs-fs/hubfs/Imported_Blog_Media/google-sitemap3-3-1-1-1.gif?width=360&height=165&name=google-sitemap3-3-1-1-1.gif) Page analysis now includes a list of the most common words in your site's content and in external links to your site. This gives you additional information about why your site might come up for particular search queries. ![Common Search Words and Queries](https://www.websitemagazine.com/hs-fs/hubfs/Imported_Blog_Media/google-sitemap4-3-1-1-1.jpg?width=400&height=102&name=google-sitemap4-3-1-1-1.jpg) ![Peter Devereaux](https://www.websitemagazine.com/hubfs/Peter_Devereaux.png) [Peter Devereaux](https://www.websitemagazine.com/marketing/author/peter-devereaux) As the Editor-in-Chief of Website Magazine and President of Website Services, Peter has established himself as a prominent figure in the digital marketing industry. With a wealth of experience and knowledge, Peter has been a driving force in shaping the landscape of digital marketing. His leadership in creating innovative and targeted marketing campaigns has helped numerous businesses achieve their revenue growth goals. Under his direction, Website Magazine has become a trusted source of information and insights for digital marketers worldwide. As President of Website Services, Peter oversees a team of talented professionals who specialize in SEO/SEM, email marketing, social media, and digital advertising. Through his hands-on approach, he ensures that his team delivers exceptional results to their clients. With a passion for digital marketing, Peter is committed to staying up-to-date with the latest industry trends and technologies, making him a sought-after thought leader in the field. - - - - - [mailto:peter.devereaux@websitemagazine.com](mailto:peter.devereaux@websitemagazine.com) #### Popular Post ### [31 Back-to-School Keyword Targets ![31 Back-to-School Keyword Targets](https://www.websitemagazine.com/hubfs/Imported_Blog_Media/schoolbus-1-1-1-1.jpg)](https://www.websitemagazine.com/marketing/31-back-to-school-keyword-targets) ### [GoDaddy Goes .XXX ![GoDaddy Goes .XXX](https://www.websitemagazine.com/hubfs/Imported_Blog_Media/gradient-regular-logo-mark-Jan-23-2023-12-15-39-1451-AM-Apr-10-2023-01-36-09-0977-PM-3-1.png)](https://www.websitemagazine.com/marketing/godaddy-goes-xxx) ### [23 Cool Tech Facts ![23 Cool Tech Facts](https://www.websitemagazine.com/hubfs/Imported_Blog_Media/fact_100x100-2-1-1-1.png)](https://www.websitemagazine.com/marketing/23-cool-tech-facts) #### Subscribe to Our Newsletter! [![](https://a.impactradius-go.com/display-ad/13053-1438301)](https://semrush.sjv.io/c/3976891/1438301/13053)![](https://imp.pxf.io/i/3976891/1438301/13053) #### Latest in Marketing [![3 Tips to Make Your Website More User-Friendly in 2023](https://www.websitemagazine.com/hubfs/Imported_Blog_Media/pexels-photo-2284350-1-1.jpg) 3 Tips to Make Your Website More User-Friendly in 2023](https://www.websitemagazine.com/marketing/3-tips-to-make-your-website-more-user-friendly) [![Web.com Launches New Professional QuickStart Website Service](https://www.websitemagazine.com/hubfs/Imported_Blog_Media/pexels-photo-4348404-1.jpg) Web.com Launches New Professional QuickStart Website Service](https://www.websitemagazine.com/marketing/web-com-launches-new-professional-quickstart-website-service) [![6 Ways to Stretch Your Marketing Budget](https://www.websitemagazine.com/hubfs/Imported_Blog_Media/gradient-regular-logo-mark-Feb-07-2023-03-32-33-0828-PM-Apr-25-2023-03-10-37-2214-AM-1.png) 6 Ways to Stretch Your Marketing Budget](https://www.websitemagazine.com/marketing/6-ways-to-stretch-your-marketing-budget) [![5 Reasons Visitors Leave Your Website and Website Tracking Tools to Track Website Visitors](https://www.websitemagazine.com/hubfs/pexels-photo-1008155.jpg) 5 Reasons Visitors Leave Your Website and Website Tracking Tools to Track Website Visitors](https://www.websitemagazine.com/marketing/5-reasons-visitors-leave-your-website) [![5 Lessons About Workplace Energy](https://www.websitemagazine.com/hubfs/Imported_Blog_Media/distress-1-1-1-1.png) 5 Lessons About Workplace Energy](https://www.websitemagazine.com/marketing/5-lessons-about-workplace-energy) [![Y Combinator Startups to #WebTechWatch](https://www.websitemagazine.com/hubfs/Imported_Blog_Media/startup-funding-4-1-1-1.jpg) Y Combinator Startups to #WebTechWatch](https://www.websitemagazine.com/marketing/y-combinator-startups-to-webtechwatch) [![gradient-regular@3x](https://www.websitemagazine.com/hs-fs/hubfs/gradient-regular@3x.png?width=128&height=128&name=gradient-regular@3x.png)](https://www.websitemagazine.com/) 999 E Touhy Ave, Des Plaines, IL 60018   [info@websitemagazine.com](mailto:info@websitemagazine.com) [(888) 7WEBMAG](tel:18887932624) [(312) 313-6576](tel:13123136576) About - [Advertise](https://www.websitemagazine.com/about/advertise) - [Editorial](https://www.websitemagazine.com/about/editorial-publishing-staff) - [Contact Us](https://www.websitemagazine.com/contact) Resources Membership - Login - Subscribe Signup for our newsletter © 2026 Website Magazine. All Rights Reserved - [Privacy Policy](https://www.websitemagazine.com/about/privacy) ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Peter Devereaux", "url" : "https://www.websitemagazine.com/marketing/author/peter-devereaux" }, "dateModified" : "2023-05-01T21:29:14.983Z", "datePublished" : "2006-02-07T06:00:00.000Z", "headline" : "Google's New Robots.txt Sitemap", "image" : [ "https://www.websitemagazine.com/hubfs/Imported_Blog_Media/gradient-regular-logo-mark-Jan-23-2023-12-06-32-2984-AM-Apr-10-2023-01-17-22-0561-PM-Apr-25-2023-03-25-59-9070-AM-2.png" ], "mainEntityOfPage" : { "@id" : "https://www.websitemagazine.com/marketing/google-sitemaps-robots", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://www.websitemagazine.com/hubfs/gradient-regular@3x-1.png" }, "name" : "Website Services, Inc." } } ``` ```json { "@context" : "https://schema.org", "@type" : "NewsArticle", "author" : [ { "@type" : "Person", "name" : "Peter Devereaux" } ], "headline" : "Google's New Robots.txt Sitemap", "publisher" : { "name" : "Website Magazine", "url" : "https://www.websitemagazine.com/" } } ```