Friday, September 22nd, 2006 by Andy Beal

5

Publishers to Spend Half Million Dollars on a Robots.txt File

Good grief, Reuters is reporting a group of publishers are planning to spend $583,700 to find a way to prevent Google from indexing their content!

“Since search engine operators rely on robotic ’spiders’ to manage their automated processes, publishers’ Web sites need to start speaking a language which the operators can teach their robots to understand,” according to a document seen by Reuters that outlines the publishers’ plans.

What the..?

Google already offers a way for a publisher to have their content removed. In addition, modify your Robots.txt file to disallow the search engines and you can donate that money to a charity that helps paranoid publishers! ;-)

Want more marketing news & views? Follow Marketing Pilgrim's Andy Beal on Twitter!

Similar Stories in: General | Forward:

Share this post

Share on TwitterStumble This!Bookmark on DeliciousShare on FriendFeedDigg This!Share on Facebook

 

5 comments on “Publishers to Spend Half Million Dollars on a Robots.txt File”

  1. Mike Says:

    September 22nd, 2006 at 7:24 pm

    Wow, someone talked them into that deal?
    If people want to throw away money I’ll take some.

  2. Brian Gilley Says:

    September 24th, 2006 at 2:49 pm

    I could’ve hooked them up for about $20 since it only takes a whopping 5 seconds to do.

    That’s ridiculous!

  3. Paul Drago Says:

    September 24th, 2006 at 9:32 pm

    I guess the World Newspaper Association can’t even ask their own webguys since they haven’t bothered with a robots.txt file either.
    Yet, amazingly our local paper has one. New York Times has one.
    Hell even the Clarion Ledger in mississippi has one! (though # be nice. isn’t a command I am familiar with :) )

  4. I’m Going to Shoot Myself in the Foot | inter:digital strategies Says:

    November 1st, 2006 at 2:25 pm

    [...] Andy Beal describes this as publishers building a new robots.txt file, but I think there’s really a bit more to it. The publishers still see that search engines are a great way to get users to their information. They don’t want their content removed, they simply want more control over how it’s used. [...]

  5. Abstroose Says:

    August 15th, 2007 at 6:17 am

    $500,000?! Wow, I possibly don’t see what you could want in a Robots.txt file worth that much?