Skip to Main Content

Google Using RSS/Atom Feeds To Discover New URLs

Posted on 11.01.2009

Google announced a new feature last week that uses RSS and Atom feeds for the discovery of new webpages.

Using feeds for discovery allows Google to index new pages more quickly than traditional crawling methods. According to the announcement, Google may use “many potential sources to access updates from feeds including Reader, notifications services, or direct crawls of feeds.

If you’re publishing an RSS/Atom feed, make sure that those files are not disallowed by your robots.txt file. If you’re unsure whether your feeds are crawlable, use the robots.txt tester in Google Webmaster Tools.


Stay up to date on the latest Internet trends:
Request a professional subscription to Website Magazine,
the most popular print publication on Web success.

Today's Top Picks for Our Readers:
Recommended by Recommended by NetLine

Leave Your Comment

Login to Comment

Become a Member

Not already a part of our community?
Sign up to participate in the discussion. It's free and quick.

Sign Up


Leave a comment
    Load more comments
    New code