Current Search Engine Spider Information
by Pete Prestipino
Posted on 01.12.2009
- email
- facebook
- share this
share this
advertisement
This is the latest data on identifying the search engine spiders. You can get more information by viewing your site's visitor log reports. Information on blocking the Search Engine Spiders with a Robots.txt file can be found at robotstxt.org.
| |
|
|
Search Engine: |
Google Adsense |
| User Agent |
Mediapartners-Google/2.1 |
|
Robots.txt Identifier |
User-agent: Mediapartners-Google |
|
Details: |
http://www.google.com/bot.html
Note: This and other Googlebots share crawled pages with each other to
reduce the amount of crawling Google is required to do for their
different services. |
| |
|
|
Search Engine: |
LookSmart |
| User Agent |
Mozilla/4.0 compatible ZyBorg/1.0
(wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com)
|
|
Robots.txt Identifier |
User-agent: ZyBorg |
|
Details: |
Powers LookSmart's Wisenut.com -
http://www.WISEnutbot.com |
| |
|
|
Search Engine: |
Microsoft / MSN / Live |
| User Agent(s) |
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-news/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-products/1.0 (+http://search.msn.com/msnbot.htm)
|
|
Robots.txt Identifier(s) |
User-agent: msnbot
User-agent: msnbot-media
User-agent: msnbot-news
User-agent: msnbot-products |
|
Details: |
MSN has not updated their page
http://search.msn.com/msnbot.htm yet to officially address the
specific bots they are using. If you wish to block a specific bot in
your robots.txt file use the associated bot specific name as the
User-agent |
| |
|
|
Search Engine: |
Yahoo Mobile |
| User Agent |
Nokia6682/2.0 (3.01.1) SymbianOS/8.0
Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;
http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
|
|
Robots.txt Identifier |
User-agent: YahooSeeker/M1A1-R2D2 |
|
Details: |
http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html |
Discover the Tactics and Techniques of the Top Digital Enterprises
today in Website Magazine's Special Spring Issue - Web 100.
Become a Member
Not already a part of our community? Sign up to participate in the discussion. It’s free and quick.
Sign Up