HoneyNet Scan of the Month #29 - Sept. 2003

by oldami@cotse.net

Question #1
Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

The first signs that something is amiss is the line "eth0: Promiscuous mode enabled." This indicates that a sniffer program has probably been run.

Just to get my bearings on this new system I do the following:
id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)

pwd
/root

ls -al
total 52
drwxr-x---   5 root     root         4096 Aug 10 20:29 .
drwxr-xr-x  18 root     root         4096 Aug 10 15:54 ..
-rw-r--r--   1 root     root         1126 Aug 23  1995 .Xresources
lrwxrwxrwx   1 root     root            9 Aug 10 15:30 .bash_history -> /dev/null
-rw-r--r--   1 root     root           24 Jun 10  2000 .bash_logout
-rw-r--r--   1 root     root          234 Jul  5  2001 .bash_profile
-rw-r--r--   1 root     root          176 Aug 23  1995 .bashrc
-rw-r--r--   1 root     root          210 Jun 10  2000 .cshrc
drwx------   2 root     root         4096 Aug  6 11:13 .links
drwx------   2 root     root         4096 Aug  6 11:51 .ssh
-rw-r--r--   1 root     root          196 Jul 11  2000 .tcshrc
-rw-r--r--   1 root     root         1257 Jul 14 13:55 anaconda-ks.cfg
drwxrwxr-x   2 500      500          4096 Aug 10 15:54 sslstop
-rw-r--r--   1 root     root         1627 Jul  4 14:09 sslstop.tar.gz

I see another sign that something is afoul. The .bash_history file has been changed to a symlink to /dev/null. This will cause bash to stop saving the command history to a file and stop us from getting the very useful information that it would provide. This symlink was created at Aug 10 15:30. The sslstop directory was created 24 minutes later. This appears to be a program to change some config options for apache.

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      845/smbd            
tcp        0      0 0.0.0.0:79              0.0.0.0:*               LISTEN      732/xinetd          
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      3137/smbd -D        
tcp        0      0 0.0.0.0:113             0.0.0.0:*               LISTEN      677/identd          
tcp        0      0 0.0.0.0:2003            0.0.0.0:*               LISTEN      3137/smbd -D        
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      732/xinetd          
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      699/sshd            
tcp        0      0 0.0.0.0:23              0.0.0.0:*               LISTEN      732/xinetd          
tcp        0      0 0.0.0.0:65336           0.0.0.0:*               LISTEN      15119/initd         
tcp        0      0 0.0.0.0:3128            0.0.0.0:*               LISTEN      25241/xopen         
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      759/sendmail: accep 
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      3137/smbd -D        
tcp        0      0 0.0.0.0:65436           0.0.0.0:*               LISTEN      15119/initd         
tcp        0   2528 192.168.1.79:65336      213.154.118.200:1188    ESTABLISHED 15119/initd         
udp        0      0 192.168.1.79:137        0.0.0.0:*                           850/nmbd            
udp        0      0 0.0.0.0:137             0.0.0.0:*                           850/nmbd            
udp        0      0 192.168.1.79:138        0.0.0.0:*                           850/nmbd            
udp        0      0 0.0.0.0:138             0.0.0.0:*                           850/nmbd            
udp        0      0 0.0.0.0:3049            0.0.0.0:*                           25239/xopen         
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     943    778/gpm             /dev/gpmctl
unix  4      [ ]         DGRAM                    7984   3247/syslogd        /dev/log
unix  2      [ ]         DGRAM                    15679  732/xinetd          
unix  2      [ ]         DGRAM                    7993   3252/klogd          
unix  2      [ ]         DGRAM                    1078   893/login -- root   
unix  2      [ ]         DGRAM                    990    820/crond           
unix  2      [ ]         DGRAM                    924    759/sendmail: accep 
unix  2      [ ]         DGRAM                    834    677/identd          
unix  2      [ ]         DGRAM                    804    657/apmd            
unix  2      [ ]         STREAM     CONNECTED     417    1/init              
Active IPX sockets
Proto Recv-Q Send-Q Local Address              Foreign Address            State

There are several suspicious items in this output, which I will discuss in the answer to question #3 below.

To see which files have been modified, I need to generate a new set of checksums:
for i in `cat /mnt/files`
do
md5sum $i
done >/mnt/newsums.md5

Unmount the floppy:
umount/mnt

On the trusted the Linux system, I perform a diff of the checksum files to see which files have changed.
mount /dev/fd0 /mnt
diff host79-2003-08-06.md5 /mnt/newsums.md5 >sums.diff

             Checksum diff output     Suspicious items are marked in red.

Many files will have changed during the normal operation of the system. I will ignore these. The changed files that do concern me are:

These are files that are frequently trojaned by a rootkit.  (Now I am glad that I have trusted versions on the floppy.)  I will discuss these further in the answer to question 8 below.

This process has shown me which files have been modified, but it does not show me which files have been added. To get this information I need a list of files in a format to compare to the list I created above. I create this list and remove the files from the /proc, /dev, and /mnt directories: mount /dev/fd0 /mnt
find / | grep -v ^/proc/ | grep -v ^/dev/ | grep -v ^/mnt/ >files2

To see which files have been added:
diff /mnt/files files2 >files.diff

             Files diff output     Suspicious items are marked in red.

Again, many files will have been added during the normal operation of the system. I look for suspicious additions and find several. The one that gets my attention first is /lib/.x, so I take a closer look at that:

/mnt/ls -alR /lib/.x
/lib/.x:
total 172
drwxr-xr-x    3 root     root         4096 Aug 10 15:32 .
drwxr-xr-x    8 root     root         4096 Aug 10 15:32 ..
-rwxr-xr-x    1 apache   apache       1223 Mar 20 15:53 .boot
-rwxr-xr-x    1 apache   apache      17931 Jan  8  2003 cl
-rwxr-xr-x    1 apache   apache        303 Dec 23  2002 hide
-rw-r--r--    1 root     root          222 Aug 10 15:32 hide.log
-rwxr-xr-x    1 apache   apache      59137 Mar 22 09:00 inst
-rw-r--r--    1 root     root         2442 Aug 10 15:32 install.log
-rw-r--r--    1 root     root            1 Aug 10 15:32 ip
-rwxr-xr-x    1 apache   apache      25795 Jan  8  2003 log
drwxrwxrwx    2 root     root         4096 Aug 10 15:32 s
-rwxr-xr-x    1 root     root        28632 Aug 10 15:32 sk

/lib/.x/s:
total 264
drwxrwxrwx    2 root     root         4096 Aug 10 15:32 .
drwxr-xr-x    3 root     root         4096 Aug 10 15:32 ..
-rwxrwxrwx    1 root     root         5192 Nov  4  2000 lsn
-rw-r--r--    1 root     root         1224 Aug 10 16:04 mfs
-rw-r--r--    1 root     root            6 Aug 10 15:32 pid
-r--r--r--    1 root     root            5 Aug 10 15:32 port
-rw-------    1 root     root          512 Aug 10 16:32 r_s
-rwxrwxrwx    1 root     root          536 Dec 28  2002 s_h_k
-rwxrwxrwx    1 root     root          340 Dec 28  2002 s_h_k.pub
-rwxrwxrwx    1 root     root          669 Dec 28  2002 sshd_config
-rwxrwxrwx    1 root     root       217667 Dec 28  2002 xopen

Notice that several of the files are owned by the apache user. This is unusual and leads me to believe that they were placed there by an apache exploit. I investigate these files using the commands file, less, and strings to discover the following.
             Output from these commands

============================================================
Time: Sun Aug 10 15:40:47     Size: 100
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 15:40:50     Size: 80
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 15:40:56     Size: 60
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 15:41:08     Size: 40
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 15:41:32     Size: 20
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 16:04:13     Size: 44
Path: proxyscan.undernet.org => 192.168.1.79 [23]
------------------------------------------------------------
k

I continue looking through the list of added files. The directory /etc/opt/psybnc has been added. Lets have a look at it:
/mnt/ls -alR /etc/opt/psybnc

             psybnc directory listing

This is the psyBNC program, which is used to hold open an IRC connection and to proxy IRC in order to hide the user's IP address. There are three instances of this program currently running, and one has an active connection. This is discussed in the answers below.

The only language files included with this version are English, Italian, and German. The intruder probably speaks one or more of these languages.

Now we know that the intruder likes to use IRC, so lets search memory for any clues we might find. The file /proc/kcore is mapped to the kernel core memory.
strings /proc/kcore | less

I use the search function of less to look for MSG, JOIN, and other common IRC strings. I find that the intruder has connected to fairfax.va.us.undernet.org and joined the channels #RedCode, #Ef~#Wurst, and #AiaBuni (which has a topic of "a bunch of hacked comps and the hackers")

Two snippets of conversation were found. One looks like the admin of this system talking to the intruder:
PRIVMSG decedatu :<NeThUs`> If you want a shell all you needed to was ask. If you don't mind please
PRIVMSG decedatu :<NeThUs`> let me know what you changed the root password to and which exploit you
PRIVMSG decedatu :<NeThUs`> used. I don't keep that box very secure as you now know. If you wish to
PRIVMSG decedatu :<NeThUs`> have a shell let me know those two pieces of information and I will
PRIVMSG decedatu :<NeThUs`> hook
PRIVMSG decedatu :<NeThUs`> it up for you. 
PRIVMSG decedatu :<NeThUs`> -Mike

Another is the intruder talking to a cohort.
:decedatu!~decedatu@193.231.34.1 PRIVMSG [[[kgb]]] :<BotikMik> uai cnd intra sick ii spunetzi ca am spus io ca il fuck...daca vretzi
:decedatu!~decedatu@193.231.34.1 PRIVMSG [[[kgb]]] :<killme> ...totzi murim...dar nu azi...zbye

Since many users of IRC talk in "chat shorthand" style, it can sometimes be difficult to identify the actual language. However, I do not believe that this is German or Italian. I am guessing that it is Romanian. To test this theory, I do Google searches for spunetzi, vretzi, and totzi. The results for all three searches were overwhelmingly Romanian. This guess is further backed up by the discovery (from the search of kcore, see above) of several systems involved in this incident, which are Romanian. Using the whois command I find the following.

Since the apache logs have been deleted, I search the hard drive for any remnants left in the free space.
strings /dev/sda1 | grep HTTP

All I find are the following entries.
[Sun Aug 10 13:16:27 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:16:37 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:40:51 2003] [error] System: No such fil213.154.118.219 - - [10/Aug/2003:13:16:27 -0700] "GET / HTTP/1.1" 400 385 "-" "-"
213.154.118.219 - - [10/Aug/2003:13:16:37 -0700] "GET / HTTP/1.1" 400 385 "-" "-"
213.154.118.219 - - [10/Aug/2003:13:23:17 -0700] "GET /sumthin HTTP/1.0" 404 279 "-" "-"

The time stamp is about two hours before the /lib/.x files were created, so I will assume that this is the intruder poking around on the web server. The IP address 213.154.118.219 (extreme-service-11.is.pcnet.ro) is another system in Romania. This is consistent with the IP addresses discovered earlier.

Question #2
Explain the impact that your actions had on the running system.

The actions taken to analyze the system will have a small impact on the system. Various log files will be modified and some swap space and disk space will be overwritten. Since this could destroy some evidence of the compromise, best practices recommend that minimal investigation be performed on a live system. The normal practice would be to create an image of the disk using dd or some other imaging tool and perform the analysis on that image. Since this is a virtual machine and we already have a suspended image, I will ignore this step. In fact, since I can easily restore the system back to the state it was in when suspended, I am free to analyze the live system with little concern about the impact of my actions (aside from isolating it from the network.) I can always return to the original suspended image for a more thorough forensic examination.

Question #3
List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

The trusted copy of the netstat command (see above) showed several suspicious ports open. I used the /proc directory to get the full path for the suspicious processes.
The /proc directory shows many details for each active process in a directory with the PID as the name. For example, the file /proc/15119/exe will be a symbolic link to the actual executable file. See this example

What I found:

Question #4
Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

The trusted netstat command (see output above) shows an active connection on port 65336 from a system with IP address 213.154.118.200 (sanido-08.is.pcnet.ro), which is an ISP in Romania. The process used for this connection is psybnc, which is a tool for holding open IRC connections and for proxying IRC connections to hide your IP address. We can deduce from this that the service is most likely IRC or IRC DCC protocol.

Question #5
How many instances of an SSH server were installed and at what times?

Question #6
Which instances of the SSH servers from question 5 were run?

By examining the /proc directory I discovered:

Question #7
Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

Once again, I look at the /proc directory to get some further information on a suspicious process.   (PID=25247    See the output)
This process has the same files open as the other instances of the xopen program. Strangely, though, its exe link does not show up. I think this may be caused by the program overlaying itself with the lsn program, which is compressed. It has one additional file open: /lib/.x/s/mfs, which I found earlier to contain password sniffer output.
Therefore, I conclude that this program has been modified to capture passwords. The file shows several FTP connections and one Telnet connection, however, no passwords were recorded. This was most likely caused by none being entered.

Question #8
Which system executables (if any) were trojaned and what configuration files did they use?

The diff of the checksums showed that several files were replaced with trojan versions. Examining these files with the strings command hinted at which configuration files they used.

This behavior is consistent with the operation of several common root kits.

Question #9
How and from where was the system likely compromised?

The running apache version is 1.3.20, which has a well-known vulnerability in the SSLv2 handshake. This is the vulnerability exploited by the "Slapper" worm and may be the exploit used to compromise this system. I believe that this or some other apache exploit was used to compromise this system. The files created in the /lib/.x directory which are owned by the apache user lead me to this conclusion.

The search for apache log file remnants showed which system was accessing the web server shortly before the compromise occurred. This leads me to believe that the compromise was most likely executed from the same address:    213.154.118.219 (extreme-service-11.is.pcnet.ro).

Bonus Question:
What nationality do you believe the attacker(s) to be, and why?

Romanian. The IP address seen in the apache log remnant was Romanian. The active network connection was from Romania. Several other Romanian IP addresses were encountered in this investigation. But, most convincingly, the IRC chat conversation discovered appeared to be in Romanian.