Comments on: Google Reader ignores robots.txt, so are feed readers ‘bots’? http://www.petercooper.co.uk/google-reader-ignores-robottxt-rules-51.html Tue, 06 Jan 2009 08:43:49 +0000 http://wordpress.org/?v=2.0.5 by: Ed http://www.petercooper.co.uk/google-reader-ignores-robottxt-rules-51.html#comment-9109 Thu, 26 Apr 2007 09:33:32 +0000 http://www.petercooper.co.uk/google-reader-ignores-robottxt-rules-51.html#comment-9109 Feed Digest isn't a Bot it's a Service. ;-) Feed Digest isn’t a Bot it’s a Service. ;-)

]]> by: peter http://www.petercooper.co.uk/google-reader-ignores-robottxt-rules-51.html#comment-9068 Thu, 26 Apr 2007 04:27:38 +0000 http://www.petercooper.co.uk/google-reader-ignores-robottxt-rules-51.html#comment-9068 Thanks for clearing that up Mihai. That's exactly the sort of answer I was looking for. Thanks again! I do wonder if, perhaps, some reform of robots.txt might come in the future to cover some of these new technologies and uses (since putting usage terms inside feeds themselves hasn't worked out so great). Thanks for clearing that up Mihai. That’s exactly the sort of answer I was looking for. Thanks again!

I do wonder if, perhaps, some reform of robots.txt might come in the future to cover some of these new technologies and uses (since putting usage terms inside feeds themselves hasn’t worked out so great).

]]> by: Mihai Parparita http://www.petercooper.co.uk/google-reader-ignores-robottxt-rules-51.html#comment-9065 Thu, 26 Apr 2007 03:40:47 +0000 http://www.petercooper.co.uk/google-reader-ignores-robottxt-rules-51.html#comment-9065 Our take on things is that feed readers are user agents (like browsers), in that they fetch URLs on behalf of a user (and at his/her request). Therefore ignoring robots.txt is OK. This is explained here: http://www.google.com/support/webmasters/bin/answer.py?answer=33545&topic=8461 For what it's worth, Google sites like Google News, Blog Search, etc. are are pretty forbidding as far as robots.txt goes, but their feeds are OK to fetch by any aggregator. Mihai Parparita Google Reader Engineer Our take on things is that feed readers are user agents (like browsers), in that they fetch URLs on behalf of a user (and at his/her request). Therefore ignoring robots.txt is OK. This is explained here:

http://www.google.com/support/webmasters/bin/answer.py?answer=33545&topic=8461

For what it’s worth, Google sites like Google News, Blog Search, etc. are are pretty forbidding as far as robots.txt goes, but their feeds are OK to fetch by any aggregator.

Mihai Parparita
Google Reader Engineer

]]> by: peter http://www.petercooper.co.uk/google-reader-ignores-robottxt-rules-51.html#comment-9055 Thu, 26 Apr 2007 00:23:40 +0000 http://www.petercooper.co.uk/google-reader-ignores-robottxt-rules-51.html#comment-9055 Blake from IceRocket has very nicely agreed to whitelist us. However, this issue still stands. robots.txt was designed long before the advent of RSS readers, yet it does not appear to be taken into account nowadays as it's such an old standard. Perhaps robots.txt needs reinventing or, at least, redefining? Blake from IceRocket has very nicely agreed to whitelist us. However, this issue still stands. robots.txt was designed long before the advent of RSS readers, yet it does not appear to be taken into account nowadays as it’s such an old standard.

Perhaps robots.txt needs reinventing or, at least, redefining?

]]>