Honeynet Scan 15 (May) Challenge

John Edward Scott <jes@daedalus-soln.demon.co.uk>


On 15 March 2001, A Linux honeypot was successfully compromised, a rootkit was downloaded to the / partition and then deleted from the system. The purpose of this paper is to show step by step how the rootkit was recovered and identified from the the system and which files make up the rootkit. If possible, it will be shown whether the rootkit was actually installed on the system.

Tools used for the investigation

A variety of tools were used in this investigation ranging from commonly used Unix commands to custom tools

Accessing the Partition

In order to view the / partition we need to mount it onto a "known" "clean" machine, after successfully downloading, untarring and confirming that the image is the one that was provided by Honeynet (via the MD5 checksum), the following command was used to mount the partition

  mount -o ro,loop,nodev,noexec honeypot.hda8.dd /mnt/honeynet

We now have the / partition mounted at /mnt/honeynet. Note that the options used with the mount command were VERY important, since we don't accidently want to be able to execute any of the commands from the compromised partition and accidently infect our "clean" system (since at this point we don't know what we're dealing with).

Resurrecting the Dead

Using tools from the Coroners Toolkit, we can reconstruct any free but non-empty inodes from the disk image. Whilst in the directory containing the honeypot.hda8.dd image, the following commands were executed

We now have (in the hda8 subdirectory) the free but non-empty inodes resurrected for us to look through.

Using the Unix file command we can determine the filetype of each of the recovered files, just for ease of use (and permanent record), the following command was used whilst in the hda8 subdirectory

  file * > /tmp/recovered.filetypes

Looking through this file we can see that there are a variety of recovered files, including empty files, files containing ASCII text, shell scripts and data. There are also a number of recovered binary files.

The recovered files

I won't discuss all of the recovered files here, only the more "interesting" ones.

What was contained in the Rootkit?

By using the Coroners Toolkit we were able to reconstruct the deleted files. This allowed us to recover the original tarball containing the Rootkit, the contents of which are shown below

Was the Rootkit installed on the system?

Looking around the filesystem we have already discovered that the directories "/dev/ida/.drag-on" and "/dev/ida/.. " exist. This would correspond with the Rootkit Installation script having been executed. Both directories also contain the files that we would expect the Installation script to have copied there. However we need to check that all of the rootkit was installed (there's always a chance that the install script could have failed for some reason). To do this, we can compare the Rootkit binaries with the "in-situ" versions of the binaries. As a double precaution we use both the Unix diff command and MD5 checksums. The following command was executed from the directory containing the Rootkit binaries (after untarring the tarball).

    diff ps /mnt/honeynet/bin/ps

and it produced no output (indicating the files were the same). Comparing the MD5 checksums gives us

    md5 ps - gives 7728c15d89f27e376950f96a7510bf0f

    md5 /mnt/honeynet/bin/ps - gives 7728c15d89f27e376950f96a7510bf0f

Therefore the MD5 checksums are also equal. To further prove the point we can compare the size of both binaries and they are indeed equal. Performing these actions on all the binaries contained in the Rootkit confirms that all the trojaned binaries have been copied into the relevant directories. We can also see if we examine the /etc/rc.d/rc.sysinit file that the very last line is the line that has been added by the Rootkit install script (namely /usr/bin/lsattr -t1 -X53 -p

There is another trick we can perform using the incredibly useful Coroners Toolkit, by executing the following command

    unrm honeypot.hda8.dd > unrm.txt

Now we can search through this file looking for evidence that the emails that should have been produced by the install script were actually created. Using a pager (since the file is huge) we can search for the string roote (this was one of the email address recipients specified, we do indeed find that the email has been created (and presumably sent).

We have now confirmed in a number of different ways that the Rootkit was indeed installed.