« Orkut and Social Networking Websites | Main | 'Miserable Failure' update. The 'Empire' Strikes Back »
February 17, 2004
Have You Been Crawled by Yahoo!?
Yahoo!'s new crawler is out spidering the web. Check your logfiles and look for a footprint of the Yahoo! Slurp crawler (basically the Inktomi cralwer with some appropriate modifications). The referrer left by the crawler points us to a Help page that has some basic information about this crawler. Yahoo, plans to do a 'cached pages' service, just like Google has, and comments on the fact that although your site may be crawled it may take some time before your pages appear.
Interesting, very interesting... The game has began.
Posted by Basileios at February 17, 2004 03:14 AM
Trackback Pings
TrackBack URL for this entry:
http://www.rugles.com/mty/mt-tb.cgi/284