We briefly state our findings here, with a detailed analysis of how we arrived at these findings following.

The binary is a remotely controllable program which can be used to execute arbitrary commands on the compromised system, and provides capabilities for several types of denial of service (DoS) attacks. The binary uses a rarely-used IP protocol number (Network Voice Protocol, or NVP) for control packets, which thus slip by many firewalls and IDS systems which are oriented toward TCP, UDP, and ICMP attacks. The binary can generate several DoS attacks including DNS reply floods, DNS server floods, SYN floods, and IP fragment attacks.

Next, we present a high level timeline that includes the sequence in which we discovered what the binary did. The tools and methods used are described, and examples are given where appropriate. The purpose of this document is to show how things are discovered, not to give in-depth documentation of all the discovered functionality. For a thorough documentation of the workings of the binary, see the technical advisory.

Timeline of the events

May 10 [ 9:30-11:30 am]

May 10 [ 2:30-4:30 pm]

May 12 [ 10:30-12:00 am]

May 13 [ 9:30 am - 1:30 pm ]

May 13 [ 2:30-4:30 pm]

May 14 [ 2:00-5:00 pm]

May 15 [ 9:30 am - 2:30 pm ]

May 16 [ 5:00-6:00 pm and 10:00-11:30 pm]

May 17 [9:45 to 11:45 am]

At this point we have completely determined all the functionality of the unknown binary, and have a control program that can exercise all functionality of the binary. Focus now turned to writing up the results as required for the reverse engineering challenge.

Tools used

Methods summary

We had great success with a general technique of experimental execution interleaved with examining the disassembled code. We think this went much faster than would have been possible just looking at assembly code. Furthermore, experimenting alone cannot discover all functionality in an unknown binary, since you cannot be sure that your experiments have exercised all possible flows in the code. By setting parameters to known values (for example, integers starting at 1) and seeing what happens, it is possible to make almost certain guesses about the purpose of the parameters (for example, seeing a SYN flood to address shows that the address comes from the first 4 parameters). Then with this knowledge, the functions in the disassembled code can be searched for uses of the parameters to see if the usage is consistent with our guess. A similar pattern of experimenting and examining code can be seen with the use of strace and using the output to zero in on particular function calls (particular in the beginning).

On the experimental side, the tools we found most valuable were strace and a sniffer (we usually used ethereal, although snort was used in some cases). When examining code, nothing we tried worked as well as simply having the code in an editor (emacs) and using the built-in searching and keyboard macro capabilities to label things as they were discovered. We had high hopes initially for IDA Pro, but the free version we used was more frustrating that it was worth. Perhaps the full commercial version would be easier to use.

Finally, while we discussed attaching to the running binary using gdb so that we could actively trace through the code, we never felt the need to do this. Perhaps in a more complex binary this would be something that would have a higher payoff.