Rui CarmoSince the MSN indexer doesn't know the first thing about robots.txt and persists in attempting to index stuff it shouldn't (and usually in batches of something like 30 requests at once, which causes far too much CPU load for me to consider it "harmless"), I've just banned msnbot/0.9 permanently from this site.
I don't care if Windows Live users can't find me - let them use a decent search engine instead.
Another notorious offender is hl_ftien_spider, which comes from IP address 211.100.29.23 and which is about to get permanently firewalled off. Judging from the amount of complaints I find by searching for it in Google, you'd expect net263 to have fixed this by now.
However, looking at the logs I find that Seekbot isn't too smart either. I've seen it try to have a go at a couple of URLs that it should have no business trying to get at and requesting the same URL twice in a row.
So it's on probation for now. Any funny business, and I'll shoot.