It's not only the chief

by Volker Weber

Do they pay the webmaster by LOC?

Comments

The competitor is more concise :-)

Klaus Pohlmann, 2006-02-20

Looks like that way would be more efficient for the White House in Washington... but actually, maybe some of the directories would be interesting to crawl and he gives away there names on purpose so some smart journalist could get to the information more easily?

Ragnar Schierholz, 2006-02-20

Did you notice that it's recursive? 5th line of "Disallow"...

Disallow: /robots.txt

Wouldn't that mean that a well behaved robot would bail out at that point and not follow the remaining directives? ;-)

Chris Linfoot, 2006-02-20

Interesting observation... is that actually covered by the standard? I mean, those guys who designed the robots.txt mechanism, would they have thought about what happens if robots.txt denies access to itself? Somehow, I have my doubts...

Ragnar Schierholz, 2006-02-20

Ragnar,

"robots.txt" is interpreted by many search engines. How this recursive instruction is interpreted depends on the implementation of the crawler. Formally, the behaviour is undefined, so anything could happen :-)

In practice, the instruction will probably have no effect. The resource "robots.txt" is already loaded when the parser hits the instruction "Disallow /robots.txt" and the admission to load the resource can not be denied retroactively.

Timo Stamm, 2006-02-20

Maybe it was just a misunderstanding of the webmaster ... maybe he had "sitemap.xml" and "robots.txt" mixed up?

Steffen Gutermann, 2006-02-21

Recent comments

Ben Uris on Klopapier gibt's :: Nur anderswo at 08:50
Thomas Mendorf on Klopapier gibt's :: Nur anderswo at 22:53
Volker Weber on Software Support for the Microsoft Surface Pro X at 22:00
Mariano Kamp on Logitech Zubehör für das Homeoffice at 21:58
Nina Wittich on Klopapier gibt's :: Nur anderswo at 21:26
Amy Blumenfield on Jitsi Meet Desktop Clients at 19:09
Jochen Schug on Logitech Zubehör für das Homeoffice at 18:43
Hermann Behrens on Klopapier gibt's :: Nur anderswo at 16:01
Friedrich Holstein on Klopapier gibt's :: Nur anderswo at 14:56
Mariano Kamp on Logitech Zubehör für das Homeoffice at 14:33
Jochen Kattoll on Software Support for the Microsoft Surface Pro X at 14:03
Volker Weber on Jitsi Meet Desktop Clients at 13:40
Uwe Papenfuss on Klopapier gibt's :: Nur anderswo at 13:19
Kieren Johnson on Jitsi Meet Desktop Clients at 13:08
Volker Weber on Klopapier gibt's :: Nur anderswo at 12:40
Moritz Petersen on Klopapier gibt's :: Nur anderswo at 12:39
Ingo Seifert on Klopapier gibt's :: Nur anderswo at 12:26
Jens Nullmeyer on Klopapier gibt's :: Nur anderswo at 12:20
Volker Weber on Klopapier gibt's :: Nur anderswo at 12:01
Markus Mews on Klopapier gibt's :: Nur anderswo at 11:58
Henning Heinz on Zoom Alternativen at 11:44
Jens Wagner on Software Support for the Microsoft Surface Pro X at 10:28
Volker Weber on Zoom Alternativen at 10:27
Robert Dahlem on Zoom Alternativen at 09:45
Markus Mews on Software Support for the Microsoft Surface Pro X at 08:44

Ceci n'est pas un blog

I explain difficult concepts in simple ways. For free, and for money. Clue procurement and bullshit detection.

vowe

Contact
Publications
Stuff that works
Amazon Wish List
Frequently Asked Questions

rss feed  twitter  amazon

Local time is 10:45

visitors.gif

Paypal vowe