Skip to Main Content

Google Now Able to Crawl Locale-Adaptive Pages

Posted on 1.29.2015

While Yahoo and Bing have recently gained ground on Google’s search traffic, Google is still the be all and end all when it comes to Internet success.

With Google’s reach spanning the globe, many companies have designed their websites to be adaptable to the user’s local language, which, until now, had caused problems for the search engines crawling bot, Googlebot.

Google has recently announced that it has updated its website crawlers to allow them to gather information from locale-adaptive pages.

For those who do not know, locale-adaptive pages are able to change their content based on the user’s language as well as their perceived geographic location. Prior to the update, there was a chance that the search engine juggernaut might not crawl, index or rank off of sites locale-adaptive content. There are two reasons for this. The first is the default IP addresses for Googlebot appear to be based in the USA. The second, is that Googlebot would send HTTP requests without placing Accept-Language in the request header.

 Request Website Magazine's Free Weekly Newsletters 

Now, with the introduction of the new locale-aware crawl configurations, Googlebot will be able to sense what content may adapt and change based on the visitors language and location. To make locale-aware crawling possible, the search engine enabled Googlebot with two new capabilities: geo-distributed and language-dependent crawling.

Through the new geo-distributed crawling capabilities Googlebot will start to use IP address that are coming from outside the USA in addition to the current IP addresses which Googlebot currently uses which come from inside the USA. For example, if a website from India is blocking content to visitors from the USA,Googlebot will be able to use an IP address from India to gain access to the content.

What’s more, for sites that alter their content based on any Accept-Language field set by browsers’ HTTP headers, the new language-dependent crawling capabilities Google will be more likely to discover, index and rank content in the different languages your site supports. The search engine is able do this by leveraging a variety of different Accept-Language HTTP headers while crawling locale-adaptive sites.

Leave Your Comment

Login to Comment

Become a Member

Not already a part of our community?
Sign up to participate in the discussion. It's free and quick.

Sign Up


Leave a comment
    Load more comments
    New code