Russell Beattie

How was your weekend? Mine sucked.

Okay, so the hacker got in to my machine sometime early Friday morning Pacific time. He left some screed in Portugese that I didn't bother translating. Thankfully Mike called me and woke me up after Elle noticed the hacked page. The Mobitopia guys were also contacting me, though via IM and SMS, which were a little less effective at that time of the morning in getting me up ;-).

I jumped on the server immediately and the guy was still there doing a bunch of stuff - or at least he had processes connected. He was in the middle of deleting *all* the logs on the server, so I have no idea where he came from (probably another hacked machine somewhere, so no biggie). I started shutting down services but there were all these scripts that would re-start stuff, so I finally just rebooted the server. Only when it came back up, I didn't have access to the machine, ssh wouldn't log me in. I then waited an impatient hour for someone at ServerMatrix to go check out what was going on. After some back and forth, he sent a message back to me (via a message board no less) telling me that someone had logged in (was it me? No.) so I called and told them to get out to the farm and rip the fucking machine out of the fucking wall if they needed to. (I was a bit tense at this point).

Finally, the box was down and I could recover whatever was left. Which was actually a lot. All the hacker did was change my home pages on all my sites. This is quite the achievement considering I have six sites in my own home-grown setup running on Tomcat. He was obviously on the site long enough to figure out what was running, and figure out how to change all the home pages. A script kiddie probably couldn't have done that - not that I have some obtuse setup, but it's not standard either.

This is where the paranoia sets in. All my email files were intact, MySQL, images, etc. but what did he *do* to them? Did he copy off all the private emails I read/sent in the past year from my IMAP store? Did he leave a back-door in one of my websites? Once the co-lo turned the machine on again, and restricted access to my home machine (over DSl, ugh), I copied everything I could off the server and they did a complete format/OS reload. So by Friday night I was starting to get things up and running again. Of course where I was downloading at 120Kps, I was uploading at only 40Kps. Urgh.

Thanks to Matt who hand-held me while Debian's email was giving me fits for the past 36 hours, and thanks *a ton* to Diego who went out to Bloglines and recovered the past month's posts and comments, and formatted them in an XML file I could just import into my DB. Both guys spent more than a few hours on the other end of IM listening to me bitch and whine. You guys *rock* seriously.

And Debian still rocks for easy-maintainability, but for some reason I could *not* get email to work correctly this weekend. It's still sort of limping along (though at least I'm getting it). Tomcat 5.5 and Java 5 also gave me fits. It was nuts, it seemed nothing went well during my install. Except that Mac OSX is a killer Unix terminal. It *is* Unix, I know... a really nice one. With several terminal windows running, Expose, Transmit, and TextWrangler, I didn't miss my Windoze box at all. Expose is a *killer* feature, especially if you have a bunch of terminal windows up. It just rules. This was actually the big test for my new Mac... I learned how to *work* on it under fire, do some development, etc. Very nice.

Okay, so how did the guy get in? No idea. The logs were gone. My best guess is a PHP CLI script I had running which allowed a Flash IRC app to re-route through my server to the freenode IRC servers. It was probably running as root and hackable as hell. I've also been playing with Apache and PHP 5 lately, so that was running on port 8080, and I really hadn't made any effort to secure it. Or it could have been any number of exploits out there that I never bothered to patch, or it could've been a bad password. We'll never know. Whatever it was, it was my fault for not maintaining my site better. Hopefully this new setup is more secure, enough to deter another attack for a while at least.

Okay, lessons: back up your data, NOW. I backed up my server last month, but the files were incomplete and a freakin' mess. So don't just back up, do it cleanly and in an organized, easy-to-find manner. Secondly, re-check your security. I've got a few more things to clean up and harden myself and I've been banging at the server all weekend.

Fuck, is tomorrow Monday?

-Russ

Update: Looks like it was an Awstats.pl exploit. Jeremy got nailed as well, but keeps better backups than myself (and logs). Using Awstats? Upgraded lately?