November 03, 2009
Google in it's official blog recently announced that, it has added a new functionality to search/discover the new URLs quickly.
Google now uses RSS and Atom feeds in order to discover the new URLs. RSS/Atom feeds have been like hot cakes providing the latest news and articles to the users. As per Google, RSS feeds allows them to get the new pages into the search index more quickly than traditional crawling methods.
Google says -
"we may also use the potential sources in order to access updates from feeds including Reader, notification services, or direct crawls of feeds. Going forward, we might also explore mechanisms such as PubSubHubbub to identify updated items."
So in order to Google crawl your websites RSS/Atom feeds, make a check to find out if Googlebot can crawl the feeds to find your pages as quickly as possible, by reviewing your robots.txt file. If you have a Google Webmasters Tools account then you can test your feed URLs with the robots.txt tester.
Get going and get your content be found on the web more easily and quickly.