Yahoo! has accepted webmaster feedback that Yahoo! Slurp (their User Agent nickname), has been crawling too heavily – particularly during recent testing. Yahoo! comments that they’ve now improved their crawl process to reduce the impact on your server and will hit your site with fewer machine IP addresses.
The new crawler continues to improve comprehensiveness and freshness of coverage while also enhancing crawl efficiency, which reduces spurious load on websites.
On a more important note… if Pilgrim readers notice any impact on the index itself, please comment and share your observations. Thanks in advance.