At 22:11 Central time, someone attempted to exploit a bug somewhere in the system - my guess is that they hit the PHPNuke or PHPBB code on Nightstar.
The hack spawned off two perl processes, one called ' sik ' and the other called ' [EntropyClient] '.
Between them, they maxed out the CPU, and somehow maxed out the bandwidth as well.
Once connected, the process used wget to attempt to find this file.
http://myshellz.com/xpl/backdoors/bash
(which does exist, I haven't had a chance to look at it - anyone that wants a copy, let me know - I downloaded it.)
Luckily, because I'm reasonably paranoid about apache, the process they were running as was 'nobody', which has no rights to be able to escalate itself and replace /bin/bash.
I was actually aware that this was going on within approximately 60 seconds, and had logged into the box to check it. Unfortunately, right as I was killing the processes, they maxed out the bandwidth and my command couldn't go through. I had to manually go to the box.
By 23:30, I'd reached the box, killed the processes, and gotten everything restarted.
However - as part of trying to make sure they couldn't repeat that, I put in updates for apache, php, and several other packages. The PHP package upgrade conflicted with the version of MySQL I put in, so I didn't realize that the zoo wasn't functioning.
This morning, I identified the problem and reverted the php.
However, the code in the zoo will probably have to be looked at.
BW