SEO Tools, Tips & Trends | SEO Blog at Website Magazine

Current Search Engine Spider Information

Written by Pete Prestipino | Jan 12, 2009 6:00:00 AM

This is the latest data on identifying the search engine spiders. You can get more information by viewing your site's visitor log reports. Information on blocking the Search Engine Spiders with a Robots.txt file can be found at robotstxt.org.

   
 Search Engine: Ask.com
User Agent Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +https://about.ask.com/en/docs/about/webmasters.shtml)
Robots.txt Identifier User-agent: Teoma
Details: https://about.ask.com/en/docs/about/webmasters.shtml
   
 Search Engine: Google
User Agent Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)
Robots.txt Identifier User-agent: Googlebot
Details: https://www.google.com/bot.html
   
 Search Engine: Google Mobile
User Agent Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +https://www.google.com/bot.html)
Robots.txt Identifier User-agent: Googlebot-Mobile
Details: https://www.google.com/support/webmasters/bin/answer.py?answer=35308
   
 Search Engine: Google Images
User Agent Googlebot-Image/1.0
Robots.txt Identifier User-agent: Googlebot-Image
Details: https://www.google.com/support/webmasters/bin/answer.py?answer=35308
   
 Search Engine: Google Adsense
User Agent Mediapartners-Google/2.1
Robots.txt Identifier User-agent: Mediapartners-Google
Details: https://www.google.com/bot.html Note: This and other Googlebots share crawled pages with each other to reduce the amount of crawling Google is required to do for their different services.
   
 Search Engine: LookSmart
User Agent Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; https://www.WISEnutbot.com)
Robots.txt Identifier User-agent: ZyBorg
Details: Powers LookSmart's Wisenut.com - https://www.WISEnutbot.com
   
 Search Engine: Microsoft / MSN / Live
User Agent(s) msnbot/1.0 (+https://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+https://search.msn.com/msnbot.htm)
msnbot-news/1.0 (+https://search.msn.com/msnbot.htm)
msnbot-products/1.0 (+https://search.msn.com/msnbot.htm)
Robots.txt Identifier(s) User-agent: msnbot
User-agent: msnbot-media
User-agent: msnbot-news
User-agent: msnbot-products
Details: MSN has not updated their page https://search.msn.com/msnbot.htm yet to officially address the specific bots they are using. If you wish to block a specific bot in your robots.txt file use the associated bot specific name as the User-agent
   
 Search Engine: Yahoo
User Agent Mozilla/5.0 (compatible; Yahoo! Slurp; https://help.yahoo.com/help/us/ysearch/slurp)
Robots.txt Identifier User-agent: Slurp
Details: https://help.yahoo.com/help/us/ysearch/slurp/
   
 Search Engine: Yahoo Mobile
User Agent Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; https://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
Robots.txt Identifier User-agent: YahooSeeker/M1A1-R2D2
Details: https://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html