By on October 30, 2009

Google Feeds Its Spiders

8



Halloween Spider and WebJust in time for Halloween, Google has given us a chance to put together a very timely and pithy headline (although I have to give credit where credit is due – hat tip to Andy). So how exactly is Google doing this and what the heck does it mean? In a nutshell, it’s a way to move toward a kind of ‘simulated’ real time search because the idea is to use its RSS/Atom feeds to identify and index new content more quickly. Is this replacing the traditional crawling technique of forever? There is no consensus on this aspect but it is obvious that Google is fully on board the real-time search train that is leaving the station as we speak.

ReadWriteWeb reports

According to a post on Google’s Webmaster Central blog, Google is now discovering web sites by automatically scanning RSS and Atom feeds. This new process will help Google more quickly identify web pages and will allow users to find new content in search results as soon as it goes live. While not exactly “real-time,” using feeds to identify updates to websites is an arguably faster method than the traditional crawling techniques Google has used in the past. And Google may get even faster in the near future – the post also notes that the company may soon explore using mechanisms like the real-time protocol PubSubHubbub to identify updated items going forward.

This is pretty nifty. Of course, the obvious question is how do you rank these new entries into whatever keyword clustered group the page belongs in? Just because they are the newest or freshest entries into a space by no means determines their relevancy and quality.

In fact, one could argue that real time could be a real pain in the butt because it could simply end up meaning that whoever is first is best. That’s not a good result. It’s kinda like saying that the kid in school who runs the 100 yard dash the fastest get the prize for Best Science Project. There is no correlation between speed and quality. It happens on occasion but it is more rare than one might think. Real time may be more suited for news and not for research. It’s too early to tell but these are questions that will be cropping up regularly moving forward.

The bottom line is that Google is going to be using all of its considerable resources to try to harness the new push to real time results. Once everything is gathered however then the fun really begins.

If I could be so bold as to make a suggestion I would like to see a “real time” search option and “traditional” one. I’m not sure there will be a real clean way to present real time results with those that are actually the best result without making the SERP’s a complete usability train wreck.

What are your thoughts on this one?


Similar Stories in: Search News | Forward: Email This Post

8 comments on “Google Feeds Its Spiders”

  1. Chris Countey Says:

    October 30th, 2009 at 11:10 am

    I’ve noticed in the past few weeks that our blog updates are spidered almost immediately upon completion. Not only that, but our regular website, [link removed] is also being spidered more frequently. Do you think the future of Google SEO will almost require a regular site to piggyback on a supporting blog?
    .-= Chris Countey´s last blog ..Article on Finding Promotional Products Online =-.

  2. Chewie Says:

    October 30th, 2009 at 12:01 pm

    I am really surprised that Google haven’t been doing this for some time, it would seem a pretty obvious way to spider fresh content.

    Way did it take them so long i wonder?
    .-= Chewie´s last blog ..Facebook users are loyal, just like your pet dog =-.

  3. Jaan Kanellis Says:

    October 30th, 2009 at 12:08 pm

    Right Chewie i think they have been doing this for a while now.
    .-= Jaan Kanellis´s last blog ..Google Analytics More Powerful, Flexible And Intelligent…BUT WHEN? =-.

  4. Google Feeds Its Spiders - Marketing Pilgrim | ByteBooth Says:

    October 30th, 2009 at 12:14 pm

    [...] hat tip to Andy). So how exactly is Google doing this and what the heck does it mean? In … … read more (No Ratings Yet)  Loading … [...]

  5. Newer Entry » - Marketing Pilgrim | ByteBooth Says:

    October 30th, 2009 at 4:17 pm

    [...] hat tip to Andy). So how exactly is Google doing this and what the heck does it mean? In … … read more (No Ratings Yet)  Loading … [...]

  6. Paul L'Acosta Says:

    October 30th, 2009 at 10:34 pm

    Frank, I think Google has been too busy lately for the last couple of years worrying about their stock price while others (even including Bing!) has been harnessing the power of real-time search. Of course, there’s no need to mention Twitter’s valuable platform and step-ahead position. Catching up is always a bummer and I think that’s what they’re trying to do now. –Paul
    .-= Paul L’Acosta´s last blog ..marketingfails: So glad I don’t have to endure through this anymore. RT @OwenGreaves: Wait For Windows 7 SP1, Support Firm Warns Users http://ff.im/-aLTG8 =-.

  7. ben 10 oyun Says:

    November 11th, 2009 at 3:03 pm

    thanks for all admin
    owe you gratitude..

  8. Mark Mc Says:

    November 17th, 2009 at 6:52 am

    I like your suggestion of offering the searcher a real time or a traditional option. it is not yet clear whether Google is doing this because of a need it is finding among users or if they are trying to keep up with the Twitters.