Skip to Main Content

Googlebot Pays Attention to the Little Guy [Infographic]

Posted on 8.05.2014

Considering Google’s almost endless reach, it is important to understand how the search engine operates.

In a study posted on the company’s official blog, Incapsula observed the behavior of bots (robots used to crawl and index websites) on the Web, specifically Googlebot and its imitators.

One of the more surprising findings has to do with with the correlation, or lack there of, between the number of page crawls and a website's popularity. Incapsula stated that before the study a number of their clients had expressed the belief that a website’s popularity directly affected the number of crawls it would receive but that with just a 1.4 percent correlation there was no evidence to suggest any significant connection between popularity and page crawls.


 SUBSCRIBE FREE to Website Magazine - 12 Issues 


On a more serious note, the study found that for every 24 visits to a website by a Googlebot there will be one visit by a Fake Googlebot. These imposters are used in a variety of ways including in DDoS attacks (23 percent) and as malicious Scrapers, Spammers and Scanners (account for 10.8 percent combined). In fact, the study found that Fake Googlebot is the third most common bot used for DDoS attacks.

The study detailed 400 million search engine visits to 10,000 websites and 2.19 billion page crawls over 30 days.

 

Leave Your Comment

Login to Comment

Become a Member

Not already a part of our community?
Sign up to participate in the discussion. It's free and quick.

Sign Up

 

Leave a comment
    Load more comments
    New code
  •    
      

    The Ultimate Guide to Personalization

    Kibo