Google Now Able to Crawl Locale-Adaptive Pages
With Google’s reach spanning the globe, many companies have designed their websites to be adaptable to the user’s local language, which, until now, had caused problems for the search engines crawling bot, Googlebot.
For those who do not know, locale-adaptive pages are able to change their content based on the user’s language as well as their perceived geographic location. Prior to the update, there was a chance that the search engine juggernaut might not crawl, index or rank off of sites locale-adaptive content. There are two reasons for this. The first is the default IP addresses for Googlebot appear to be based in the USA. The second, is that Googlebot would send HTTP requests without placing Accept-Language in the request header.
Now, with the introduction of the new locale-aware crawl configurations, Googlebot will be able to sense what content may adapt and change based on the visitors language and location. To make locale-aware crawling possible, the search engine enabled Googlebot with two new capabilities: geo-distributed and language-dependent crawling.
Through the new geo-distributed crawling capabilities Googlebot will start to use IP address that are coming from outside the USA in addition to the current IP addresses which Googlebot currently uses which come from inside the USA. For example, if a website from India is blocking content to visitors from the USA,Googlebot will be able to use an IP address from India to gain access to the content.
What’s more, for sites that alter their content based on any Accept-Language field set by browsers’ HTTP headers, the new language-dependent crawling capabilities Google will be more likely to discover, index and rank content in the different languages your site supports. The search engine is able do this by leveraging a variety of different Accept-Language HTTP headers while crawling locale-adaptive sites.