Publishers to Spend Half Million Dollars on a Robots.txt File

Friday, September 22nd, 2006 by Andy Beal

Publishers to Spend Half Million Dollars on a Robots.txt File

Good grief, Reuters is reporting a group of publishers are planning to spend $583,700 to find a way to prevent Google from indexing their content!

“Since search engine operators rely on robotic ’spiders’ to manage their automated processes, publishers’ Web sites need to start speaking a language which the operators can teach their robots to understand,” according to a document seen by Reuters that outlines the publishers’ plans.

What the..?

Google already offers a way for a publisher to have their content removed. In addition, modify your Robots.txt file to disallow the search engines and you can donate that money to a charity that helps paranoid publishers!

Want more marketing news & views? Follow Marketing Pilgrim's Andy Beal on Twitter!

Share this post

5 comments on “Publishers to Spend Half Million Dollars on a Robots.txt File”

Mike Says:

September 22nd, 2006 at 7:24 pm

Wow, someone talked them into that deal?
If people want to throw away money I’ll take some.
Brian Gilley Says:

September 24th, 2006 at 2:49 pm

I could’ve hooked them up for about $20 since it only takes a whopping 5 seconds to do.

That’s ridiculous!
Paul Drago Says:

September 24th, 2006 at 9:32 pm

I guess the World Newspaper Association can’t even ask their own webguys since they haven’t bothered with a robots.txt file either.
Yet, amazingly our local paper has one. New York Times has one.
Hell even the Clarion Ledger in mississippi has one! (though # be nice. isn’t a command I am familiar with )
I’m Going to Shoot Myself in the Foot | inter:digital strategies Says:

November 1st, 2006 at 2:25 pm

[...] Andy Beal describes this as publishers building a new robots.txt file, but I think there’s really a bit more to it. The publishers still see that search engines are a great way to get users to their information. They don’t want their content removed, they simply want more control over how it’s used. [...]
Abstroose Says:

August 15th, 2007 at 6:17 am

$500,000?! Wow, I possibly don’t see what you could want in a Robots.txt file worth that much?

Marketing Pilgrim