Current Search Engine Spider Information

Posted on

  • email
  • twitter
  • facebook
  • share this

share this


This is the latest data on identifying the search engine spiders. You can get more information by viewing your site's visitor log reports. Information on blocking the Search Engine Spiders with a Robots.txt file can be found at robotstxt.org.


   
 Search Engine: Ask.com
User Agent Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
Robots.txt Identifier User-agent: Teoma
Details: http://about.ask.com/en/docs/about/webmasters.shtml
   
 Search Engine: Google
User Agent Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Robots.txt Identifier User-agent: Googlebot
Details: http://www.google.com/bot.html
   
 Search Engine: Google Mobile
User Agent Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Robots.txt Identifier User-agent: Googlebot-Mobile
Details: http://www.google.com/support/webmasters/bin/answer.py?answer=35308
   
 Search Engine: Google Images
User Agent Googlebot-Image/1.0
Robots.txt Identifier User-agent: Googlebot-Image
Details: http://www.google.com/support/webmasters/bin/answer.py?answer=35308
   
 Search Engine: Google Adsense
User Agent Mediapartners-Google/2.1
Robots.txt Identifier User-agent: Mediapartners-Google
Details: http://www.google.com/bot.html Note: This and other Googlebots share crawled pages with each other to reduce the amount of crawling Google is required to do for their different services.
   
 Search Engine: LookSmart
User Agent Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com)
Robots.txt Identifier User-agent: ZyBorg
Details: Powers LookSmart's Wisenut.com - http://www.WISEnutbot.com
   
 Search Engine: Microsoft / MSN / Live
User Agent(s) msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-news/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-products/1.0 (+http://search.msn.com/msnbot.htm)
Robots.txt Identifier(s) User-agent: msnbot
User-agent: msnbot-media
User-agent: msnbot-news
User-agent: msnbot-products
Details: MSN has not updated their page http://search.msn.com/msnbot.htm yet to officially address the specific bots they are using. If you wish to block a specific bot in your robots.txt file use the associated bot specific name as the User-agent
   
 Search Engine: Yahoo
User Agent Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Robots.txt Identifier User-agent: Slurp
Details: http://help.yahoo.com/help/us/ysearch/slurp/
   
 Search Engine: Yahoo Mobile
User Agent Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
Robots.txt Identifier User-agent: YahooSeeker/M1A1-R2D2
Details: http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html

 
     :: Try iContact for FREE for 30 Days! Sign up now at iContact.com ::


Login To Comment


Become a Member

Not already a part of our community? Sign up to participate in the discussion. It's free and quick.

Sign Up

1 comment

04-17-2009 2:41 PM

: by Dante Monteverde : Are you looking at your website from the perspective of a search engine spider

Add to the discussion!

999 E Touhy Ave
Des Plaines, IL 60018

Toll Free: 1.800.817.1518
International: 1.773.628.2779
Fax: 1.773.272.0920
Email: info@websitemagazine.com

Facebook


Twitter