Current Search Engine Spider Information

Posted on

  • email
  • twitter
  • facebook
  • share this

share this

This is the latest data on identifying the search engine spiders. You can get more information by viewing your site's visitor log reports. Information on blocking the Search Engine Spiders with a Robots.txt file can be found at

 Search Engine:
User Agent Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +
Robots.txt Identifier User-agent: Teoma
 Search Engine: Google
User Agent Mozilla/5.0 (compatible; Googlebot/2.1; +
Robots.txt Identifier User-agent: Googlebot
 Search Engine: Google Mobile
User Agent Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +
Robots.txt Identifier User-agent: Googlebot-Mobile
 Search Engine: Google Images
User Agent Googlebot-Image/1.0
Robots.txt Identifier User-agent: Googlebot-Image
 Search Engine: Google Adsense
User Agent Mediapartners-Google/2.1
Robots.txt Identifier User-agent: Mediapartners-Google
Details: Note: This and other Googlebots share crawled pages with each other to reduce the amount of crawling Google is required to do for their different services.
 Search Engine: LookSmart
User Agent Mozilla/4.0 compatible ZyBorg/1.0 (;
Robots.txt Identifier User-agent: ZyBorg
Details: Powers LookSmart's -
 Search Engine: Microsoft / MSN / Live
User Agent(s) msnbot/1.0 (+
msnbot-media/1.0 (+
msnbot-news/1.0 (+
msnbot-products/1.0 (+
Robots.txt Identifier(s) User-agent: msnbot
User-agent: msnbot-media
User-agent: msnbot-news
User-agent: msnbot-products
Details: MSN has not updated their page yet to officially address the specific bots they are using. If you wish to block a specific bot in your robots.txt file use the associated bot specific name as the User-agent
 Search Engine: Yahoo
User Agent Mozilla/5.0 (compatible; Yahoo! Slurp;
Robots.txt Identifier User-agent: Slurp
 Search Engine: Yahoo Mobile
User Agent Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/ (compatible;YahooSeeker/M1A1-R2D2;
Robots.txt Identifier User-agent: YahooSeeker/M1A1-R2D2

Login To Comment

Become a Member

Not already a part of our community? Sign up to participate in the discussion. It's free and quick.

Sign Up

1 comment

04-17-2009 2:41 PM

: by Dante Monteverde : Are you looking at your website from the perspective of a search engine spider

Add to the discussion!

999 E Touhy Ave
Des Plaines, IL 60018

Toll Free: 1.800.817.1518
International: 1.773.628.2779
Fax: 1.773.272.0920