It's not only the chief

by Volker Weber

Do they pay the webmaster by LOC?

Comments

The competitor is more concise :-)

Klaus Pohlmann, 2006-02-20

Looks like that way would be more efficient for the White House in Washington... but actually, maybe some of the directories would be interesting to crawl and he gives away there names on purpose so some smart journalist could get to the information more easily?

Ragnar Schierholz, 2006-02-20

Did you notice that it's recursive? 5th line of "Disallow"...

Disallow: /robots.txt

Wouldn't that mean that a well behaved robot would bail out at that point and not follow the remaining directives? ;-)

Chris Linfoot, 2006-02-20

Interesting observation... is that actually covered by the standard? I mean, those guys who designed the robots.txt mechanism, would they have thought about what happens if robots.txt denies access to itself? Somehow, I have my doubts...

Ragnar Schierholz, 2006-02-20

Ragnar,

"robots.txt" is interpreted by many search engines. How this recursive instruction is interpreted depends on the implementation of the crawler. Formally, the behaviour is undefined, so anything could happen :-)

In practice, the instruction will probably have no effect. The resource "robots.txt" is already loaded when the parser hits the instruction "Disallow /robots.txt" and the admission to load the resource can not be denied retroactively.

Timo Stamm, 2006-02-20

Maybe it was just a misunderstanding of the webmaster ... maybe he had "sitemap.xml" and "robots.txt" mixed up?

Steffen Gutermann, 2006-02-21

Recent comments

Jochen Kattoll on The Neighbor’s Window :: Oscar Winning Short Film at 22:21
Harald Gärttner on Microsoft Office app on Android and iOS at 15:09
Volker Weber on App-Store-Interna: Apple geht gegen Buchveröffentlichung vor at 14:55
Oliver Stör on App-Store-Interna: Apple geht gegen Buchveröffentlichung vor at 13:42
Volker Weber on Microsoft Office app on Android and iOS at 13:02
Harald Gärttner on Microsoft Office app on Android and iOS at 12:56
Volker Weber on Marshall Monitor II A.N.C. :: Erste Eindrücke at 08:07
Thomas Cloer on Marshall Monitor II A.N.C. :: Erste Eindrücke at 07:57
Matthias Lorz on The Neighbor’s Window :: Oscar Winning Short Film at 12:47
Volker Weber on The Neighbor’s Window :: Oscar Winning Short Film at 11:54
Fabio Peruzzi on The Neighbor’s Window :: Oscar Winning Short Film at 11:48
Fabio Peruzzi on The Neighbor’s Window :: Oscar Winning Short Film at 11:45
Horia Stanescu on The Neighbor’s Window :: Oscar Winning Short Film at 07:04
Martin Funk on The Neighbor’s Window :: Oscar Winning Short Film at 00:00
Sven Bühler on I am not ready for a foldable phone at 22:03
Andreas Imnitzer on The Neighbor’s Window :: Oscar Winning Short Film at 21:48
Roland Dressler on I am not ready for a foldable phone at 15:02
Daniel Seiler on I am not ready for a foldable phone at 13:51
Roland Dressler on I am not ready for a foldable phone at 12:55
Hubert Stettner on I am not ready for a foldable phone at 10:51
Matthias Welling on Tools and Weapons #nowreading at 09:05
Ingo Harpel on You may secretly be a Bing user at 20:01
Amy Blumenfield on Tools and Weapons #nowreading at 19:44
Horia Stanescu on You are famous on Botnet at 12:33
Volker Weber on You are famous on Botnet at 20:17

Ceci n'est pas un blog

I explain difficult concepts in simple ways. For free, and for money. Clue procurement and bullshit detection.

vowe

Contact
Publications
Stuff that works
Amazon Wish List
Frequently Asked Questions

rss feed  twitter  amazon

Local time is 23:06

visitors.gif

Paypal vowe