1. Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

After resuming the virtual machine, we find the following situation:

That "Promiscuos mode enabled" is already not nice.

We are not given any information about the presence of a sniffer, so that gave me the first bad feeling.

 

Well, let's take a look at the machine's connection and open ports:


netstat -an

Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 192.168.1.79:65336 213.154.118.200:1188 ESTABLISHED
tcp 0 0 0.0.0.0:65436 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN <--let's think that this is squid. We'll see that it is not.
tcp 0 0 0.0.0.0:65336 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:113 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:79 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN
udp 0 0 0.0.0.0:3049 0.0.0.0:*
udp 0 0 0.0.0.0:138 0.0.0.0:*
udp 0 0 192.168.1.79:138 0.0.0.0:*
udp 0 0 0.0.0.0:137 0.0.0.0:*
udp 0 0 192.168.1.79:137 0.0.0.0:*
Active UNIX domain sockets (including servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ] STREAM CONNECTED 417
unix 2 [ ] DGRAM 804
unix 2 [ ] DGRAM 834
unix 2 [ ] DGRAM 924
unix 2 [ ] DGRAM 990
unix 2 [ ] DGRAM 1078
unix 2 [ ] DGRAM 7993
unix 2 [ ] DGRAM 15679
unix 4 [ ] DGRAM 7984 /dev/log
unix 2 [ ACC ] STREAM LISTENING 943 /dev/gpmctl
Active IPX sockets
Proto Recv-Q Send-Q Local Address Foreign Address State

 

A couple of ports are listening (coloured) which one wouldn't expect to find.

One connection is established on a high port.
tcp 0 0 192.168.1.79:65336 213.154.118.200:1188 ESTABLISHED

we move to another computer, connected to the internet, and we take a look with samspade (or other web-based tool that provider whois queries - NOT from our own IP address: if it's an attacker who's connected with us he might see his own logs and spot we are "tracing" him.

 

whois 213.154.118.200 then ??

 

inetnum: 213.154.96.0 - 213.154.127.255
netname: PCNET
descr: PCNET Data Network S.A.
descr: PROVIDER ADSL Network
country: RO
admin-c: BT17-RIPE
tech-c: PDNN1-RIPE
status: ASSIGNED PA
notify: tudor@pcnet.ro
mnt-by: AS8503-MNT
changed: tudor@pcnet.ro 20030704
source: RIPE
[..]
person: Bogdan Tudor
remarks: Technical Manager
remarks: PCNET Data Network S.A.
address: Bucharest, Romania
phone: +40 9 325 18 84
phone: +40 1 330 86 61
phone: +40 1 330 35 23
fax-no: +40 1 675 49 99
nic-hdl: BT17-RIPE
mnt-by: BT17-RIPE-MNT
notify: tudor@pcnet.ro
e-mail: tudor@pcnet.ro
changed: tudor@pcnet.ro 20011009
source: RIPE

 

Romania.... interesting..... and why is my host has an established session with a node in Romania on an unknown - to me -port ? You'll see more about this in Answer N.4 even if I actually investigated it at this time.

 

 

Well, then my 'classic' command would be a 'ps -Af', to see the runing processes.
This is what I get:

what have I misspelled? I am on linux, not BSD...

have you noticed the weird thing? It does not work like a linux version..... more like a bsd variant....
My favourite syntax of '-Af' is returning the help page as not being understood...
Then I try a
'ps -aux' which works as follows:

USER PID %CPU %MEM SIZE RSS TTY STAT START TIME COMMAND
ident 677 0.0 0.9 26932 944 ? S Aug 9 0:00 identd -e -o
ident 685 0.0 0.9 26932 944 ? S Aug 9 0:00 identd -e -o
ident 686 0.0 0.9 26932 944 ? S Aug 9 0:00 identd -e -o
ident 695 0.0 0.9 26932 944 ? S Aug 9 0:00 identd -e -o
ident 696 0.0 0.9 26932 944 ? S Aug 9 0:00 identd -e -o

root 1 0.0 0.5 1412 520 ? S Aug 9 0:05 init
root 2 0.0 0.0 0 0 ? SW Aug 9 0:00 [keventd]
root 3 0.0 0.0 0 0 ? SW Aug 9 0:00 [kapm-idled]
root 4 0.0 0.0 0 0 ? SWNAug 9 0:00 [ksoftirqd_CPU0]
root 6 0.0 0.0 0 0 ? SW Aug 9 0:00 [kreclaimd]
root 7 0.0 0.0 0 0 ? SW Aug 9 0:00 [bdflush]
root 8 0.0 0.0 0 0 ? SW Aug 9 0:00 [kupdated]
root 9 0.0 0.0 0 0 ? SW<Aug 9 0:00 [mdrecoveryd]
root 17 0.0 0.0 0 0 ? SW Aug 9 0:04 [kjournald]
root 92 0.0 0.0 0 0 ? SW Aug 9 0:00 [khubd]
root 657 0.0 0.5 1396 524 ? S Aug 9 0:00 /usr/sbin/apmd -p 10
root 699 0.0 1.3 2676 1272 ? S Aug 9 0:00 /usr/sbin/sshd
root 732 0.0 1.0 2264 956 ? S Aug 9 0:00 xinetd -stayalive -re
root 759 0.0 2.1 5296 1984 ? S Aug 9 0:00 sendmail: accepting c
root 778 0.0 0.5 1440 496 ? S Aug 9 0:00 gpm -t ps/2 -m /dev/m
root 820 0.0 0.6 1584 660 ? S Aug 9 0:00 crond
root 850 0.0 1.1 2416 1084 ? S Aug 9 0:00 nmbd -D
root 893 0.0 1.1 2320 1076 1 S Aug 9 0:00 login -- root
root 894 0.0 0.4 1384 448 2 S Aug 9 0:00 /sbin/mingetty tty2
root 895 0.0 0.4 1384 448 3 S Aug 9 0:00 /sbin/mingetty tty3
root 896 0.0 0.4 1384 448 4 S Aug 9 0:00 /sbin/mingetty tty4
root 899 0.0 0.4 1384 448 5 S Aug 9 0:00 /sbin/mingetty tty5
root 900 0.0 0.4 1384 448 6 S Aug 9 0:00 /sbin/mingetty tty6
root 901 0.0 1.3 2444 1276 1 S Aug 9 0:00 -bash
root 3247 0.0 0.6 1472 592 ? S 13:33 0:00 syslogd -m 0
root 3252 0.0 1.1 1984 1096 ? S 13:33 0:00 klogd -2
root 15119 0.0 1.3 2296 1240 ? S 16:02 0:01 initd
root 15372 0.0 0.6 1484 624 1 R 20:34 0:00 ps -aux
root 25239 0.0 0.3 1880 336 ? S 15:32 0:00 /lib/.x/s/xopen -q -p
root 25241 0.0 0.7 1888 672 ? S 15:32 0:00 /lib/.x/s/xopen -q -p
root 25247 0.0 0.7 1668 732 ? S 15:32 0:00 /lib/.x/s/lsn

 

After this discovery... the 'ps' executable "stinks" as being a trojaned - we'll see later that I'm right in this feeling.

But even if it is a trojaned one it... it DOES show some akward stuff: (in particular)
a. the last three lines...(/lib/.x/s/xopen ???)
b. nmbd IS there....where is its 'buddy': 'smbd' ? they always go together...

SINCE NOW ON, I decided that indeed the machine WAS hacked (or VERY lkely to be) so I could not trust the system and its outpus (see question.2).

The machine HAS to be disconnected.
This way we don't minimize the impact on the system (if it was a production server - where people need to connect - but it is not), but I consider it necessary as the hacker can find we spotted him and wipe our disk or similar nasty reaction...

Then I mounted a CD with trusted - statically compiled - binaries (FIRE - Forensic and Incident Response Environment - http://fire.dmzs.com/), and used those for EVERY command that follows.
Moreover I decided NOT TO write on the disk since I might overwrite important, possibly deleted but maybe still recoverable information/evidence.
For this purpose I redirected every output to files on a floppy disk, analysed then on another machine.
A couple of boxes of floppies are always handy in these situations :)

 

OK, let's return to the 'ps -aux' output previously presented: Do you see the last three preocesses ?? they smell even less nice than the whole lot so far. What is /lib/.x supposed to be ?
What on earth is that directory now ??
NOTE: Some files in the directory listing are linked to the actual files, or excerpt of them. Only those who were not harmful, or not completely investigated binaries, or not too big. Basically you'll read some scripts and pieces of logs. For other I linked some descritive short pages or pictures of their output when launched - NONE OF THIS OPERATION (launching not trusted programs to capture the output for this writeup) HAS BEEN EXECUTED ON THE ANALYSED MACHIHE, BUT ON ANOTHER, DISPONSABLE, ALSO NOT CONNECTED ONE.

 

/mnt/cdrom/statbins/linux2.2_x86/ls -la /lib/.x >/mnt/floppy/ls01.out

total 172
drwxr-xr-x 3 root root 4096 Aug 10 15:32 .
drwxr-xr-x 8 root root 4096 Aug 10 15:32 ..
-rwxr-xr-x 1 apache apache 1223 Mar 20 15:53 .boot
-rwxr-xr-x 1 apache apache 17931 Jan 8 2003 cl
-rwxr-xr-x 1 apache apache 303 Dec 23 2002 hide
-rw-r--r-- 1 root root 222 Aug 10 15:32 hide.log
-rwxr-xr-x 1 apache apache 59137 Mar 22 09:00 inst
-rw-r--r-- 1 root root 2442 Aug 10 15:32 install.log
-rw-r--r-- 1 root root 1 Aug 10 15:32 ip
-rwxr-xr-x 1 apache apache 25795 Jan 8 2003 log
drwxrwxrwx 2 root root 4096 Aug 10 15:32 s
-rwxr-xr-x 1 root root 28632 Aug 10 15:32 sk

 

 

it looks like a rootkit or something of that kind.....

especially when I see that 'hide.log' (copied to floppy and viewed on another machine) contains:

 

 

#####################################################
# SucKIT version 1.3b by Unseen <unseen@broken.org> #
#####################################################

RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]

 

and that 'install.log' contains:

 

#####################################################
# SucKIT version 1.3b by Unseen <unseen@broken.org> #
#####################################################

RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]
#####################################################
# SucKIT version 1.3b by Unseen <unseen@broken.org> #
#####################################################

RK_Init: idt=0xffc17800, FUCK: Can't find sys_call_table[]

[...] Muliple Times [...]

 

which looks like something in the installation of the rootkit didn't succeed 100% ("Can't find sys_call_table[]"), and it has been attempted several times.
Good. This is why we are still able to see some tracks, most likely.
TO a certain degree we are lucky in this analisys thanks to this factor.
It might have given an error because another LKM was already installed, such as Sebek. But I have NOT verified if this was the case.
If the hacker had been successful in installing a complete kernel rootkit, it could have hidden himself better...
And this is why some of the processes show up EVEN with the trojaned 'ps' executable... the kernel has not been patched well so that the processes are not 'hidden' properly (or: not as good as in the hacker's intentions, most likely).

But anyway something HAS indeed been done.
Someone has been here. That is for sure.

Shall we take a look at a couple of those strange ports?

 

telnet localhost 2003

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
SSH-1.5-By-ICE_4_All ( Hackers Not Allowed! )
Protocol mismatch.

 

This one has a modified SSHD (modified at least in the banner 'Hackers Not Allowed!' - we'll see later about it in greater details as from Answer N.5).

 

telnet localhost 3128

And this one port that looks like Squid (software I would not expect on an exposed, internet facing server, or at least not available from the outside...)... this is because squid is not mentioned in the processes... but 3128 is its well-known port.
..and in fact it is something else:

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
SSH-1.5-1.2.32
Protocol mismatch.

 

Here is another SSHD ! Unbelievable !
Again, we'll see in Answer N.5.

 

telnet localhost 65336

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
:Welcome!psyBNC@lam3rz.de NOTICE * :psyBNC2.3.1

 

...yeah...right....
after having "hidden" himself so well, the hacker is using a BOUNCER. We'll see better about this service and the connection it has in Answer N. 4.

 

OK, I want to see it clearer.
Luckily we have this FANTASTIC resource that is: the md5 hashes of the files on the system...which is always handy to have in such cases, and which is best prectice to generate just after a system's installation, to keep a reference of his "clean" state.

/mnt/cdrom/statbins/linux2.2_x86/md5sum -c /mnt/floppy/host79-2003-08-06.md5 > /mnt/floppy/md5compare.out

and on another machine:

less md5compare.out |grep FAILED > md5compareFAIL.out
less md5compareFAIL.out

so we get a list of only the modifed files:

/var/lib/slocate/slocate.db: FAILED
/var/lib/random-seed: FAILED
/var/lib/logrotate.status: FAILED
/var/log/messages: FAILED
/var/log/lastlog: FAILED open or read
/var/log/secure: FAILED
/var/log/maillog: FAILED
/var/log/wtmp: FAILED
/var/log/sa/sa14: FAILED open or read
/var/log/sa/sa15: FAILED open or read
/var/log/sa/sar14: FAILED open or read
/var/log/sa/sa16: FAILED open or read
/var/log/sa/sar15: FAILED open or read
/var/log/sa/sa06: FAILED open or read
/var/log/samba/log.smbd: FAILED open or read
/var/log/samba/smbd.log: FAILED open or read
/var/log/samba/log.nmbd: FAILED open or read
/var/log/samba/localhost.log: FAILED open or read
/var/log/xferlog: FAILED open or read
/var/log/httpd/error_log: FAILED open or read
/var/log/httpd/ssl_engine_log: FAILED open or read
/var/log/httpd/access_log: FAILED open or read
/var/log/httpd/ssl_request_log: FAILED open or read
/var/log/httpd/access_log.1: FAILED open or read
/var/log/httpd/error_log.1: FAILED open or read
/var/log/dmesg: FAILED open or read
/var/log/cron: FAILED
/var/log/boot.log: FAILED
/var/log/rpmpkgs: FAILED open or read
/var/cache/man/whatis: FAILED
/var/cache/samba/smbd.pid: FAILED
/var/cache/samba/connections.tdb: FAILED
/var/cache/samba/nmbd.pid: FAILED
/var/run/utmp: FAILED
/var/run/runlevel.dir: FAILED
/var/run/syslogd.pid: FAILED
/var/run/klogd.pid: FAILED
/var/run/apmd.pid: FAILED
/var/run/sshd.pid: FAILED
/var/run/sendmail.pid: FAILED
/var/run/gpm.pid: FAILED
/var/run/crond.pid: FAILED
/var/run/ftp.rips-all: FAILED open or read
/var/spool/anacron/cron.daily: FAILED
/var/spool/anacron/cron.weekly: FAILED
/tmp/root.md5: FAILED open or read
/etc/rc.d/init.d/functions: FAILED
/etc/rc.d/rc.sysinit: FAILED
/etc/mail/statistics: FAILED
/etc/aliases.db: FAILED
/etc/adjtime: FAILED
/etc/samba/secrets.tdb: FAILED
/etc/httpd/conf/httpd.conf: FAILED
/usr/bin/top: FAILED
/bin/netstat: FAILED
/bin/ls: FAILED
/bin/ps: FAILED <-- my "feeling" was right about this one, let's say "instinctively" :)
/sbin/ifconfig: FAILED

 

So we see that at least those last five (5) executables (top, netstat, ls, ps, ifconfig) have ben replaced, and many other files fail the check (they might have ben delected, I thought immediately, especially the log files). We'll see more on the replaced executables in Answer N.8.

We in fact find out that most log files have been wiped (the files in /var/log).

 

/mnt/cdrom/statbins/linux2.2_x86/ls -la /var/log >/mnt/floppy/lsx.out

total 40
drwxr-xr-x 2 root root 4096 Aug 10 15:30 .
drwxr-xr-x 17 root root 4096 Jul 14 13:54 ..
-rw------- 1 root root 676 Aug 10 15:54 boot.log
-rw------- 1 root root 3665 Aug 10 20:30 cron
-rw------- 1 root root 16694 Aug 10 20:30 maillog
lrwxrwxrwx 1 root root 9 Aug 10 15:30 messages -> /dev/null
-rw------- 1 root root 179 Aug 10 18:58 secure
-rw------- 1 root root 0 Aug 10 13:33 spooler
-rw-r--r-- 1 root root 0 Aug 10 13:33 wtmp

 

most files are simply wiped, and the mail syslog file, 'messages' is redirected to /dev/null, so that syslog finds the link and it actually works without complaining, but in practice nothing gets logged anymore. The file wtmp contains (usually) the data about the last shell accesses/logins that get showed in human-readable format by the 'last' command. This has also been emptied.

Let's see then what we can still find on the blocks of the disk.
Let's search for deleted inodes with the ils command:

/mnt/cdrom/statbins/linux2.2_x86/ils /dev/sda1 >/mnt/floppy/inodes.out

class|host|device|start_time
ils|sbm79.dtc.apu.edu|/dev/sda1|1060572624
st_ino|st_alloc|st_uid|st_gid|st_mtime|st_atime|st_ctime|st_dtime|st_mode|st_nlink|st_size|st_block0|st_block1
3187|a|48|0|1060464891|1060547558|1060555930|45307|100600|0|0|0|0
3555|f|0|0|1060572000|1060572000|1060572000|1060572000|100600|0|0|0|0
3559|f|0|0|1060572000|1060572000|1060572000|1060572000|100600|0|0|0|0
3579|f|0|0|1060572000|1060572000|1060572000|1060572000|100600|0|0|0|0
3580|f|0|0|1060558200|1060558200|1060558200|1060558200|100600|0|0|0|0
3581|f|0|0|1060558200|1060558200|1060558200|1060558200|100600|0|0|0|0
3582|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3583|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3584|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3585|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3586|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3587|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3588|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3589|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3590|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3591|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3592|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3593|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3594|f|48|48|1060550034|1060549983|1060550034|1060550034|100755|0|0|0|0
3595|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3596|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3597|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3598|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3599|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3600|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3601|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3602|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3603|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3604|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3605|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3606|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3607|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3608|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3609|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3610|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3611|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3612|f|48|48|1060550034|1060549983|1060550034|1060550034|100755|0|0|0|0
3613|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
3614|f|48|48|1060550034|1060549983|1060550034|1060550034|100644|0|0|0|0
35835|f|0|0|1060572573|1060572573|1060572573|1060572573|100000|0|0|0|0
45307|a|0|0|1060556678|1060547637|1060556678|46912|100600|0|6468|112745|112750
45308|a|48|0|1060513321|1060513321|1060513321|46916|100600|0|0|0|0
45309|a|48|0|1060513321|1060547558|1060513321|45308|100600|0|0|0|0
46607|a|0|0|1060547636|1060547636|1060547636|46924|40700|0|4096|113609|0
46733|a|0|0|1060547636|1060547636|1060547636|45309|40755|0|4096|113929|0
46912|a|48|48|1060550034|1060550034|1060550034|46607|40755|0|4096|112749|0
46914|a|0|0|1060555929|1060195432|1060555929|46935|100644|0|22795530|114381|16010
46916|a|0|0|1058243135|1060195432|1058243135|46934|100644|0|0|0|0
46920|a|0|0|1060513321|1060513321|1060513321|46733|100644|0|0|0|0
46924|a|0|0|1060465239|1060195432|1060465239|46920|100644|0|207|114393|0
46934|a|0|0|1060546997|1060513321|1060546997|46914|100644|0|253|114422|0
46935|a|0|0|1060555929|1060513321|1060555929|0|100644|0|23335716|114400|114401
76979|f|0|0|1060547636|1060547636|1060547636|1060547636|40755|0|0|0|0
77078|f|0|0|1060513324|1060195432|1060513324|1060513324|100644|0|0|0|0
77079|f|0|0|1060513324|1060195432|1060513324|1060513324|100644|0|0|0|0
77080|f|0|0|1060513324|1060195432|1060513324|1060513324|100644|0|0|0|0
77081|f|0|0|1060513325|1060195432|1060513325|1060513325|100644|0|0|0|0
77645|f|0|0|1060547636|1060195801|1060547636|1060547636|100644|0|0|0|0
77646|f|0|0|1060547636|1060513324|1060547636|1060547636|100644|0|0|0|0
77647|f|0|0|1060547636|1060547400|1060547636|1060547636|100644|0|0|0|0
77648|f|0|0|1060547636|1060513324|1060547636|1060547636|100644|0|0|0|0
104395|f|0|0|1060572000|1060572000|1060572000|1060572000|100400|0|0|0|0
104397|f|0|0|1060555800|1060555800|1060555800|1060555800|100400|0|0|0|0

 

There is a bunch of inodes that hopefully will lead us to recover some previously deleted data.

Patiently, I started using 'debugfs'.
defugfs can show (among other things) which inodes points to which blocks, and gives the possibility to DUMP those blocks to a new file, effectively reconstructing the deleted data.
There was an issue with older version (and filesystem ext2) where only the first 12 blocks of an inode were actually dumped.
But I tried to be "positive-thinking" (maybe crazy, but I have been lucky) and remembered that with the new ext3 filesystem this limit is gone.

For extra information you're advised to read the "Linux Ext2fs Undeletion mini-HOWTO" at http://www.tldp.org/HOWTO/Ext2fs-Undeletion.html

 

On anothere Operating System the recovery would have ben much more painful, and require the use of tools such as The Coroner's Tolkit (TCT) or others.
Linux is getting very easy and friendly in these things, instead.

 

debugfs
open /dev/sda1

Once in debugfs interactive mode, and opened the right disk, we use - patiently - the 'stats' command followed by each of the available inodes (found by ils), and we see which one of them have blocks associated.

Example:

and we try to dump those to new files in an attempt to recover them.
This can take a long time and patience but it is worth since we recovered also some big files and various logs - we'll see in a minute.

I dumped to a floppy al the small files first, and I left the two big ones at the end, since they wouldn't have fit on a floppy. See also Answer N.2 to understand this issue better.

 

So we have now found (to summarize the most important points):

a. the /lib/.x directory
b. a bunch of recovered files
c. we know which files have ben changed, removed (then we can further investigated on the replaced / modifed, and match the deleted with what just recovered.

 

To get some extra insight, we run (from the CD with the trusted binaries) the famous chkrootkit (see complete output):

Checking `ifconfig'... INFECTED
Checking `ls'... INFECTED
Checking `ps'... INFECTED
Checking `top'... INFECTED
Searching for suspicious files and dirs, it may take a while... /usr/lib/perl5/5.6.0/i386-linux/.packlist /lib/.x /lib/.x/.boot /lib/.x
Searching for anomalies in shell history files... Warning: `//root/.bash_history' is linked to another file
Checking `bindshell'... INFECTED (PORTS: 3049)
Checking `lkm'... You have 4 process hidden for ps command
Warning: Possible LKM Trojan installed
eth0 is PROMISC

 

Now hat I knew WHERE to look for and I had the data to analyse, let's get a last touch to facilitate the reconstruction of the sequence of events.

Since ls -la shows the date in the format 'Month dd hh:mm', etc... We can grep for files based on the timestamp. I know that most likly many people will feel sick because there is definitely a bunch of more elegant ways to do this. But I don't know the other ways, and this is a 'quick and dirty' solution. Sorry :)

so for example /mnt/cdrom/statbins/linux2.2_x86/ls -laR / |grep 'Aug 10 12' >/mnt/floppy/exportfile.txt is showing the files modified on August 10th, at any 12:xx time (between 12:00 and 12:59).

I just searched most of the day with this method. Files in /lib/.x have a time around 15:53. So the break in must have happened before.
this information will be used, together with the undeleted log files to understand the times and methods of the break in. More precise analisys is in Answer N. 9. Till now is only data-collection.

Files modified at 11:xx
Files modified at 12:xx
Files modified at 13:xx
Files modified at 14:xx
Files modified at 15:xx
Files modified at 16:xx
Files modified at 17:xx
Files modified at 18:xx
Files modified at 19:xx
Files modified at 20:xx

 

 

   To answer N.2 --> Explain the impact that your actions had on the running system.  Home