Hello, you there! Person reading this post. How did you find it? If you’re a loyal MarketingPilgrim follower then maybe you came in here via our daily email update. Perhaps you came in off a social media link or you clicked a bookmark. If you’ve never been here before, then chances are you came in via a search engine. Welcome. We’re happy to have you and we hope you come back often.
Search engines are vital to the success of any site with written content. So why would you want to stop them from indexing your site?
The UK newspaper The Times stopped them because they felt it ran counter to their paid content business model. Back in 2010, the newspaper blocked search engine spiders as part of their conversion to digital subscriptions. Apparently, they believed that their good name alone would drive traffic to the site.