Internet Marketing Monitor
August 22, 2007
Filed Under (Yahoo) by Derick on 08-22-2007

If you’re running an operation as big as Yahoo!’s, efficiency is a must. To that end, Yahoo! has been hard at work making sure its crawler is as efficient as possible:

In response to feedback from the community, we’re reducing our crawling machines by a significant amount with this update, so you might have observed a significant reduction in crawl load from us along with fewer machine IP addresses hitting you over the last few days.

In addition, the company also says it’s rolled out “significant” updates to it’s index and ranking systems. The new, more efficient crawler should improve “comprehensiveness and freshness” in the index while at the same time reduce the load it puts on servers.

Related Posts & Pages Recent Posts



Comments:
2 Comments posted on "Yahoo! Crawler Gets Efficiency Boost; Index Gets “Significant” Update"
RustyS on August 22nd, 2007 at 3:39 pm #

It’s about time. If you aren’t careful with your filtering, or not filtering at all, Yahoo’s spider can easily rack up a disproportionate amount of pageviews (and visit sessions due to all the IPs) on a site’s log reports.


[…] week Yahoo! announced updates to the way it’s crawler scoured the web (which not everyone is happy with, by the way). This […]


Post a comment
Name: 
Email: 
URL: 
Comments: