Googlebot Pays Attention to the Little Guy [Infographic]
Considering Google’s almost endless reach, it is important to understand how the search engine operates.
One of the more surprising findings has to do with with the correlation, or lack there of, between the number of page crawls and a website's popularity. Incapsula stated that before the study a number of their clients had expressed the belief that a website’s popularity directly affected the number of crawls it would receive but that with just a 1.4 percent correlation there was no evidence to suggest any significant connection between popularity and page crawls.
On a more serious note, the study found that for every 24 visits to a website by a Googlebot there will be one visit by a Fake Googlebot. These imposters are used in a variety of ways including in DDoS attacks (23 percent) and as malicious Scrapers, Spammers and Scanners (account for 10.8 percent combined). In fact, the study found that Fake Googlebot is the third most common bot used for DDoS attacks.
The study detailed 400 million search engine visits to 10,000 websites and 2.19 billion page crawls over 30 days.