The googlebot is hammering me again today. It's looking for stuff like this:

/Chizumatic/cref.shtml/Chizumatic/translation/translation/tmw/else/tmw/reviews/Shingu.shtml
/Chizumatic/cref.shtml/else/tmw/translation/translation/else/reviews/NinjaNonsense.shtml
/Chizumatic/cref.shtml/translation/nitpicks/reviews/tmw/else/tmw/ALDVD06.shtml
/Chizumatic/cref.shtml/Chizumatic/translation/nitpicks/else/translation/nitpicks/reviews/UFOPrincessValkyrie.shtml

None of those is a real path here; but the way my server was set up before they'd all return the "cref.shtml" entry (the second item in the path) and it's loaded with sub-paths. (It's the index of all my reviews.)

Last time this happened I had to block the googlebot in my firewall for a while. Then one of my readers showed me how to set it up so that my server would return a 503 for all those paths.

What I'm hoping is this: that all those idiotic paths are ones the googlebot has in its history and it thinks it is supposed to visit again, and that when it does this time and gets a 503, that it'll tag the path as a bad one and never try to visit it again. hope hope hope

hope hope hope because it's been visiting bogus paths in that non-existent directory tree for the last four hours and it doesn't show any sign of stopping any time soon. I sure hope it doesn't treat the 503 as a temporary thing, and revisit all of them again and again over the upcoming weeks in hopes of finding out if the 503 has been cleared... (Maybe I should have set it up to return a 504, eh?)

It's getting kind of lonely around this part of the blogosphere these days. Shamus has carpal tunnel and may have to curtail his blogging. Wonderduck has medical problems in his family. Ubu has gotten infected with politics. And I'm not feeling so good, myself.

Posted by: Steven Den Beste in Site Stuff at 06:13 PM | No Comments | Add Comment
Post contains 278 words, total size 2 kb.