Seemed like a good idea at the time
ScriptyGoddess posted a link to tip from NSLog(); suggesting a cool way of automatically searching from 404 pages, which I implemented a couple of days ago.
Unfortunately I didn't take Googlebot into account, and when it visited today it generated rather alot of 404 errors in rather a short space of time. The server didn't like that (all of a sudden alot of cgi scripts running doing alot of big database queries).
So the automatic search feature disappeared quite quickly. :-)
Comments
Comments are now on the LiveJournal version of this entry.
Hey, no worries. I should have read the comments rather than jumping headlong into it :-)
(Oh, and sorry about the double-trackback - didn't mean to do that!)
A thought: Perhaps a check before it executes any other code (like those big database queries) it checks the useragent string to see if GoogleBot is listed in it and simply dump out a standard 404 response? Simple regex, right?
That would work for Google, but what about the countless other search engines?
Why not have it not work for any robot accessing the page, that would eliminate most search engines?
Woah dude. Sorry about that. Some of the concerns other people brought up made me not implement it, which is all discussed in the comments, but based on your experience, maybe I should move it up and edit the post itself. Thanks for letting me know!