more stats and analysis of robots.txt files were released in Google's Sitemaps
feature. Some of the details and features now provided include analysis of the
cached robots.txt file, the ability to enter a list to see if the robots.txt
file allows or blocks spider agents, the page of your site with the highest
PageRank a list of the most common words in your site's content and in external
links to your site.
Read more about Google's new robots.txt sitemap now.
google sitemaps, google robots, google agent