Somehow during the day I felt the Internet was a bit slow at work today. Try to do a ping to another box on the net, and the return trip time has been fairly irregular. Open up the MRTG page for the router in a web browser, and then BANG! This is what I have seen:
Aarrgghh!! Data is coming from the Internet via our ADSL link at 975kbit/sec since Saturday night, as you can see from the flat line. Suddenly, I thought of an image of a dark room only illuminated by the flashing light coming off an old CRT monitor. Wire is all over the place, and there's a box of left over pizza sitting next to a mid-tower computer case, with its skin stripped off. A black hat hacker hammers a few keys with this evil grin on his face, and in his Russian accent he spelt out "die!" before clicking on the button that labels "Denial of Service". Muwahahahah!!!
No. I gotta stand up and combat the "evil". I quickly logged onto our router box, which is an old Pentium 166 box running Linux 2.4, and tried to use the logging facility in iptables to track down the source and destination of these packets. Hey! How come it comes from our proxy server trying to pull stuff from our co-location box on the Internet? Check out the proxy server logs, and then found several of our production sites kept on pulling a zipped XML file from our co-location box at an amazing rate. It turns out to be a piece of code that I've written a while ago to periodically pull some configuration file from a centralised place. The timer has somehow stuffed up, and the cron'ed task just kept on looping and fetching...
D'oh. It turns out that there is no black hat hacker. There is one just stupid coder who with his own code, DoS'ed the Internet connection between the office and the co-location box.