Today when I checked the logs and MRTG graph, I saw a huge spike on the load of the system at around 7:00am.
Yeah - the process load went all the way to 40, making a huge spike in my MRTG graph. And the kernel log reports:
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) __alloc_pages: 0-order allocation failed (gfp=0xf0/0) __alloc_pages: 0-order allocation failed (gfp=0xf0/0) __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) VM: killing process mt-comments.cgi
Looks like MovableType's commenting script has gong wacko and got killed by the kernel. Looking at the Apache log, I was amazed - within a minute or two, 217 POST requests, pretending to be Windows 2003 Server, hit the
mt-comments.cgi script on the server, originated from 17 different IP addresses all over the world, all trying to place casino advertisement on Sui's weblog (what have you done Sui!). I can't imagine all these Perl processes fired up to process the requests. Every thing eventually died down after 15 minutes. Can't imagine if they keep on hitting it.
I found it is very difficult to defend against spambot of this sort. You cannot null-route the IP address, as the requests are coming from everywhere. You cannot block on the user-agent, as it masqueraded itself behind some creditable user-agent. The only way would be using content based black list - but they always can find a way to bypass it. Like in Sui's case, the spammer changed every second letter to numeric entity - which makes keyword detection impossible. Moreover, running things like
mt-blacklist is very costly, and running hundreds of those would only make DDoS even worse.
Hopefully it would not come back tonight.