It's not only the chief
by Volker Weber
Do they pay the webmaster by LOC?
Comments
The competitor is more concise :-)
Looks like that way would be more efficient for the White House in Washington... but actually, maybe some of the directories would be interesting to crawl and he gives away there names on purpose so some smart journalist could get to the information more easily?
Did you notice that it's recursive? 5th line of "Disallow"...
Disallow: /robots.txt
Wouldn't that mean that a well behaved robot would bail out at that point and not follow the remaining directives? ;-)
Interesting observation... is that actually covered by the standard? I mean, those guys who designed the robots.txt mechanism, would they have thought about what happens if robots.txt denies access to itself? Somehow, I have my doubts...
Ragnar,
"robots.txt" is interpreted by many search engines. How this recursive instruction is interpreted depends on the implementation of the crawler. Formally, the behaviour is undefined, so anything could happen :-)
In practice, the instruction will probably have no effect. The resource "robots.txt" is already loaded when the parser hits the instruction "Disallow /robots.txt" and the admission to load the resource can not be denied retroactively.
Maybe it was just a misunderstanding of the webmaster ... maybe he had "sitemap.xml" and "robots.txt" mixed up?
Post a comment
Recent comments
Frank Koehntopp
on Updates from vowe's Apple zoo at 17:27
Tobias Mueller
on Updates from vowe's Apple zoo at 14:11
Lukas Praml
on Updates from vowe's Apple zoo at 14:01
Paul Farris
on Google syncs address book to BlackBerrys at 12:21
Alan Bell
on Amazing Head Tracking at 16:45
Oswald Prucker
on Google syncs address book to BlackBerrys at 14:36
Kai Schmalenbach
on Apple TV: First impressions at 11:13
Armin Roth
on Google syncs address book to BlackBerrys at 10:47
Gonzague Dambricourt
on Google syncs address book to BlackBerrys at 23:57
Nick Daisley
on Foldershare, R.I.P. at 20:20
peter mojica
on Notes or Exchange? Let's put an end to those creative statistics at 19:42
Armin Roth
on Weihnachtsmüll von Microsoft at 10:33
Michael Sedlaczek
on Weihnachtsmüll von Microsoft at 10:26
Julian Buss
on Use scripts to improve install process for Lotus Notes at 08:36
Philipp Sury
on Apple TV: First impressions at 07:04
Frank Paolino
on New poll: which BlackBerry could be your next? at 05:10
marco foellmer
on BlackBerry Curve 8900 at 00:20
Uwe Brahm
on Foldershare, R.I.P. at 19:03
Nick Shelness
on Foldershare, R.I.P. at 16:55
Nick Shelness
on Foldershare, R.I.P. at 16:43
Ralf M Petter
on Foldershare, R.I.P. at 15:32
Jan Tietze
on Foldershare, R.I.P. at 14:12
Ole Saalmann
on Foldershare, R.I.P. at 13:33
Stefan Weigand
on Doc Scrubber cleans your Word document at 13:21
Doug Petrosky
on Apple TV: First impressions at 13:00
Click here to subscribe
