Yahoo不遵循robots.txt?

查看最近的蜘蛛抓取日志，我发现Yahoo! Slurp抓走了两个css文件，css是放在我用robots.txt禁止抓取的文件夹中，不知道Slurp什么时候开始也这么流氓了？

虽然不是见不得人的文件，可之前我一直相信大型搜索引擎除了百度都能很守规矩地遵循robots.txt，现在看来不见得。

从支持的http版本来看，Googlebot和Baiduspider不愧为搜索引擎市场领头羊手下的蜘蛛，支持http 1.1 Gzip的特性有效地减少了中间的传输流量。

»»3条评论

1. How I Lost Thirty Pounds in Thirty Days posted on 09-05-04 at 14:24Reply

   Hi, good post. I have been thinking about this issue,so thanks for writing. I will certainly be subscribing to your blog.

2. Yeeseo Team posted on 09-05-04 at 14:53Reply

   @How I Lost Thirty Pounds in Thirty Days:

   hi, sob guy!

   You don't have to say this, your mom told me you've always been a liar.

3. God Father posted on 09-05-04 at 15:02Reply

   @How I Lost Thirty Pounds in Thirty Days:

   I subscribed a blog on mars~