Scan of the Month 33: Anti Reverse Engineering Uncovered

By Nicolas Brulez - 0x90(at)Rstack(dot)org



Rather than doing another complete analysis of the binary, i will rather present the techniques i have used in the challenge, and how i have implemented them. The Scan of the Month 33 was released by the Honeynet Project in November 2004. I invite everyone to read the excellent submissions we received this month once they have read my paper. I am presenting the binary from the protection author point of view, while they presented it from the analyst point of view. You will learn the methods and techniques used to Protect / Unprotect a binary with this month's challenge. A lot of weaknesses were left on purpose in this binary and they will be presented here.

Contents

  • The Challenge
  • Identify and explain any techniques in the binary that protect it from being analyzed or reverse engineered
  • Something uncommon has been used to protect the code from beeing reverse engineered, can you identificate what it is and how it works?
  • Provide a mean to "quickly" analyse this uncommon feature
  • Which tools are the most suited for analysing such binaries, and why?
  • Identify the purpose (fictitious or not) of the binary
  • What is the binary waiting from the user? Please detail how you found it
  • Bonus Question - What techniques or methods can you think of that would make the binary harder to reverse engineer?
  • Conclusion
  • Acknowledgement
  • About the Author



  • The Challenge:
    This month's challenge is to analyze an unknown binary, in an effort to reinforce the value of reverse engineering, and improve
    (by learning from the security community) the methods, tools and procedures used to do it. This challenge is similar to SotM 32.
    However, this binary has mechanisms implemented to make the binary much harder to analyze, to protect against reverse engineering. 
    Skill Level: Advanced/Expert
    
    All we are going to tell you about the binary is that it was 'found' on a WinXP system and has now be sent to you for analysis. 
    You will have to analyse it in-depth and get as much information as possible about its inner working, and what is the goal of the binary.
    The main goal of this challenge is to teach people how to analyse heavily armored binaries. Such techniques could be used in the future, 
    and its time to get used to them.

    1 - Identify and explain any techniques in the binary that protect it from being analyzed or reverse engineered

    Many techniques have been used in order to slow down analysis and break reverse engineers tools:

    The final protection of the binary is a complete Virtual Machine i wrote for the challenge. I have designed a Virtual CPU that will interpret my own Assembly language. The Virtual Machine is quite simple to understand and isn't very complex.

    Virtual Machines seem to be a new trend in protection systems, so i thought it could be a good thing to write one for such a challenge. The instruction encoding is very trivial, and could have been a lot harder to understand. The first Version i had in mind was a lot more complex. I wanted not only to have a pseudo language, but also to program the instructions handlers emulating real x86 instructions. Each handler would be a few hundred instructions long and a lot harder to analyse.

    A small program has been written with this Virtual Machine Assembly language, and it was used to authenticate the user running the binary.

    Read next part for further informations

    2 - Something uncommon has been used to protect the code from beeing reverse engineered, can you identificate what it is and how it works?

    Even though, a few protection systems are using some kind of Virtual Machines, those aren't very common. Especially in Malwares and other exploits.

    Virtual CPU description and Inner working:

    Registers:

    REGISTERS STRUC
    	R0_		dd	?  ; 000
    	R1_		dd	?  ; 001
    	R2_		dd	?  ; 002
    	COUNTER_	dd	?  ; 003
    	EIP_ 		dd	?  ; 004 -> reserved
    	STATE_		dd 	?  ; 005
    REGISTERS ENDS
    

    This is the original structure from my code source. Every registers is a DWORD. Some registers weren't used because they are reserved for futur version of the Virtual Machine. One can read "EIP_". I planned to add another information per instruction, but i didn't do it, because i didn't want it to be too complex. I will add the ability to change the Instruction pointer for any instruction. The result will be a completely mad code flow. The instruction order in the file will have nothing to do with the real execution flow.

    The STATE Register is some kind of mini Eflags. This register changes depending of other instructions.

    The COUNTER Register is used for loop instructions. Similar to the ECX register when we use the LOOP instruction.

    regs	REGISTERS	<>
    
    R0	equ	000b
    R1	equ	001b
    R2	equ	010b
    COUNTER equ 	011b
    EIP 	equ 	100b
    STATE	equ	101b
    

    Here are a few other definitions used in my program.I started to represent the registers in binary because i wanted to do complex opcode decoding. I will do that for another version ;)

    Registers Initialisation:

    		mov		dword ptr [regs.R0_],"livE" 	; Registers are initialized with a Slayer Song Title.
    		..	
    		mov		dword ptr [regs.R1_],"saH " 	; Evil Has No Boundaries!
    		..		
    		mov		dword ptr [regs.R2_]," oN "
    		..		
    		mov		dword ptr [regs.COUNTER_],"nuoB"
    		..		
    		mov		dword ptr [regs.EIP_],"irad"
    		..		
    		mov		dword ptr [regs.STATE_],"! se"
    

    At the start of the VM, i first begin to initialise my own registers with the song's title of a thrash metal band. This title was selected because i planned to do real evil things with the Virtual Machine. It isn't as hard as the initial version, but still evil enough to keep that funny string ;-).

    There is about 34 Instructions in the Virtual Machine. (I count instructions having different utilisation as unique)

    I will present a few instruction handlers to explain the inner working of the Virtual Machine, but not every instruction will be presented here.

    Pcode Fectcher:

    The first thing the Virtual Machine does after the Register Init is to Fetch the Pcode entry point and jmp to the first Pcode handler.

    		movzx	eax, byte ptr [esi]			; ESI is Pcode Entry Point. This code gets the first instruction Prefix.
    				
    		mov 	edi, dword ptr [eax*4+poffset]		; It uses it with the offset table to find the Pcode family it has to execute.
    		
    		movzx	eax, byte ptr [esi+1] 		    	; get second byte, use it as an index into last table.
    								; The VM now knows what instruction it has to emulate and goes to it.
    		
    		JMPNEXT						; Emulate a jmp	dword ptr [eax*4+edi] with Exception Handling and Context Manipulation.
    								; Jmp to the next Pcode instruction handler
    

    Examples of Instructions implemented inside the Virtual Machine:

    Before i start with those examples, i would like to say that a few instructions present in the Virtual Machine weren't used and were left as decoy.Three of them are using Self modifying code. People are reporting that they don't work, but they should. The off by one difference is because the opcode is beeing called from other instruction handlers. Two instructions are modifying one instruction on the fly as they need to execute a particular piece of code. They then restore the instruction state. I am too lazy to check whether those instructions are really bugged or if they didn't use the good parameters. One of the _unused_ instruction HAS a bug, and i am glad some people noticed it. The instruction isn't used therefore, it is just a decoy instruction. The instruction is supposed to be a Virtual BSWAP, but it doesn't save the result of the swaping. Another unused instruction is the INT 3. This instruction allows one to put breakpoint in his Pcode program and trace with his debugger from that instruction. I left this instruction in the final Virtual Machine and im glad some people found it and abused it!.

    STOPVM

    The first instruction i will present here is a very simple one. It tells to the Virtual Machine to stops and the program will get back to normal x86 assembly program.

    @STOPVM:
    		pop  	dword ptr fs:[0]			; Im using SEH to jmp from handlers to handlers in the VM.
    		add  	esp,4					; Therefore i need to remove the handler installed before i do anything.
    		popad						; i restore the registers..
    		..
    		push	dword ptr [Pret]			; Put the Return Address (to get out of the VM) on the stack.
    		..		
    		xor	[esp],'HAX0	'			; Decrypt it with a funny string: HAXO(R)
    		..		
    		ret 						; Get out of the VM.
    

    This instruction is not using any bytecode fetcher because it doesn't need to jmp to another handler. I will now present a real instruction. A Virtual PUSH:

    LOAD

    @Load:
    		pop  dword ptr fs:[0]					
    		add  esp,4
    		popad
    
    ; Same as every handler, remove SEH and restore registers.
    
    		mov	eax,dword ptr [esi+2]				; Get into EAX the first operand of the instruction.
    		
    		xor	eax,37195411h					; Decrypt it.
    		
    		push 	eax						; Push it onto the stack.
    		
    		mov 	eax,0FFFFFF3Fh					; EAX = FFFFFF3Fh
    		
    		not	eax						; EAX = not(EAX) = C0h
    		
    		shr	eax,5						; EAX = EAX shr 5 = 6 : This is the instruction length
    		
    		lea	esi, [esi+eax]					; ESI = Instruction Pointer. Deplace the Instruction Pointer 6 bytes further.
    		
    		movzx	eax, byte ptr [esi]				; ESI now points to the new instruction to be executed.
    		
    		mov 	edi, dword ptr [eax*4+poffset]  		; It uses it with the offset table to find the Pcode family it has to execute.
    		
    		movzx	eax, byte ptr [esi+1] 		    		; get second byte, use it as an index into last table.
    									; The VM now knows what instruction it has to emulate and goes to it.
    		
    		JMPNEXT							; Emulate a jmp	dword ptr [eax*4+edi] with Exception Handling and Context
    									; Manipulation.
    									; Jmp to the next Pcode instruction handler
    

    As you can see from this little handler, the instruction is 6 bytes long. It takes only one parameter and it is placed 2 bytes after the start of the instruction. (ESI+2). The parameter is encrypted with 37195411h. The decrypted parameter is pushed on the stack and then the Virtual Machine calls the next instruction.

    From this, we can say that this instruction is a push. since push is already a x86 instruction, i named my virtual push : LOAD.

    One can use it like this: "LOAD number"

    VMXOR

     
    @VMXORDISPATCHER:
    		pop  	dword ptr fs:[0]
    		add  	esp,4
    		popad
    
    ; Same as every handler, remove SEH and restore registers.
    
    
    		movzx	eax, byte ptr [esi+2] 			; Get the Index Register to acces the Virtual CPU registers.
    		
    		mov	eax, dword ptr [regs+eax*4] 		; edi = Register value to know wich register is going to be concerned (RO, R1 , R2)
    								; EAX = value used by the XOR.
    	
    		movzx	ecx, byte ptr [esi+3]			; ECX = type of XOR. Byte ptr ? Word Ptr ? or Dword Ptr..
    		
    		jmp 	dword ptr [xortable+ecx*4]		; Jmp to the good handler accordingly.
    		
    
    @VMXORBPTR:  
    		
    		movzx	ecx, byte ptr [esi+4] 			; Get Index Register for the destination.
    		
    		mov	ecx, dword ptr [regs+ecx*4] 		; edi = Register value to know wich register is going to be used (RO, R1 , R2)
    		
    		xor	byte ptr [ecx],al			; XOR BYTE PTR
    	
    		add	esi,5					; Instruction Length is 5
    
    		movzx	eax, byte ptr [esi]			; ESI now points to the new instruction to be executed.
    		
    		mov 	edi, dword ptr [eax*4+poffset]  	; It uses it with the offset table to find the Pcode family it has to execute.
    		
    		movzx	eax, byte ptr [esi+1] 		    	; get second byte, use it as an index into last table.
    								; The VM now knows what instruction it has to emulate and goes to it.
    		
    		JMPNEXT						; Emulate a jmp	dword ptr [eax*4+edi] with Exception Handling and Context Manipulation.
    								; Jmp to the next Pcode instruction handler
    
    
    @VMXORWPTR:
    		
    		movzx	ecx, byte ptr [esi+4] 			; Get Index Register for the destination
    		
    		mov	ecx, dword ptr [regs+ecx*4] 		; edi = Register value to know wich register is going to be used (RO, R1 , R2)
    		
    		xor	word ptr [ecx],ax			; XOR WORD PTR
    		
    		add	esi,5					; Instruction Length is 5
    		
    		movzx	eax, byte ptr [esi]			; ESI now points to the new instruction to be executed.
    		
    		mov 	edi, dword ptr [eax*4+poffset]  	; It uses it with the offset table to find the Pcode family it has to execute.
    		
    		movzx	eax, byte ptr [esi+1] 		    	; get second byte, use it as an index into last table.
    								; The VM now knows what instruction it has to emulate and goes to it.
    		
    		JMPNEXT						; Emulate a jmp	dword ptr [eax*4+edi] with Exception Handling and Context Manipulation.
    								; Jmp to the next Pcode instruction handler
    
    @VMXORDPTR:
    
    		movzx	ecx, byte ptr [esi+4] 			; Get Index Register for the destination
    		
    		mov	ecx, dword ptr [regs+ecx*4] 		; edi = Register value to know wich register is going to be used (RO, R1 , R2)
    				
    		xor	dword ptr [ecx],eax			; XOR DWORD PTR
    		
    		add	esi,5					; Instruction Length is 5
    		
    		movzx	eax, byte ptr [esi]			; ESI now points to the new instruction to be executed.
    		
    		mov 	edi, dword ptr [eax*4+poffset]  	; It uses it with the offset table to find the Pcode family it has to execute.
    		
    		movzx	eax, byte ptr [esi+1] 		    	; get second byte, use it as an index into last table.
    								; The VM now knows what instruction it has to emulate and goes to it.
    		
    		JMPNEXT						; Emulate a jmp	dword ptr [eax*4+edi] with Exception Handling and Context Manipulation.
    								; Jmp to the next Pcode instruction handler		
    

    From this piece of code we can learn many things. The XOR instructions are coded with 5 Bytes. It has two parameters.One register has the value used to do the XOR and One Register has a pointer to the location to be xored.It also have a byte saying whether it is a XOR BYTE PTR, a XOR WORD PTR or a XOR DWORD PTR.

    This instruction handler is therefore handling Virtual XOR instruction.


    How were the virtual instruction used to create a program ?

    I will now show how i did to create virtual instruction, because the x86 assembler doesn't know them and will never compile a LOAD or a VMXOR.To do so, i used a very simple way: MACRO. For each instruction, a corresponding macro has been created, and is used to encode the instruction for me. This way i can write a program with my Assembly mnemonics without caring of the opcodes representation.I will now show the 3 Macros used for the examples Virtual Instruction descrived above

    STOPVM macro
    	db 	02,00
    endm

    This is the macro for the STOPVM instruction. Usage: STOPVM

    Load macro x
    	db 	00,00
    	dd	x xor 37195411h
    endm

    This is the macro for the LOAD instruction. Usage: LOAD x

    VMXOR macro  reg0,kind,reg1  
    	db 01,03,reg0,kind,reg1
    endm

    This is the macro for the VMXOR instruction. Usage: VMXOR Rx xPTR Rx



    P-code Program used in this challenge:

    The P-code is my own assembly language, thus IDA doesn't know anything about it. Here is how it looks under a disassembler:

    Ok, it doesn't look so good. So now, here is the complete program (copy pasted from my source) i have written with my OWN assembly language. Cool isn't it ? :-)

    [email protected]:
    
    Pcode1:
    
    	MOVE pcrypt COUNTER
    	LOADPTR startpcodecrypted
    	RestoreREG R0
    	MOVE 'S' R2
    
    decryptpcode:
    
    	VMXOR R2 BPTR R0
    	INCR R0
    	DECR COUNTER
    	BNZ decryptpcode
    	
    startpcodecrypted:
    	
    	MOVE 05CC80E31h R1
    	APICALL GetCommandLineA
    	SCANB " " 255h
    	BZ youwishdude
    	DLOAD R0 R2 
    
    	ADDREG R2 01D9BDC45h
    	ADDREG R1 74519745h		
    	SUBREG R2 0AD45DFE2h
    	ADDREG R1 0DEADBEEFh	
    	ADDREG R2 "hell"
    	SUBREG R1 17854165h	
    	SUBREG R2 "Awai"
    	ADDREG R1 "show"
    	ADDREG R2 "its "
    	SUBREG R1 " no "	
    	ADDREG R2 "driv"
    	ADDREG R1 "merc"	
    	SUBREG R2 "nuts"
    	SUBREG R1 "y!!!"	
    	SUBREG R2 "eh?!"
    	ANDREG R2 0DFFFFFFFh
    	
    	LOADREG R2
    	LOADREG R1
    	CMPQ firstcheckdone
    	CLEAR COUNTER
    	BZ youwishdude
    	
    firstcheckdone:
    
    	INCR R0
    	ADDREG R0 2
    	INCR R0   			
    	WLOAD R0 R1 			
    	
    	LOADREG R1
    	
    	RESTOREREG R2     	
    	LOADREG R0        	
    	LOADPTR tricky-98547h 	
    	RESTOREREG R0         	
    	ADDREG R0 98548h      	
    	DECR R0               	
    	
    	VMXOR R2 WPTR R0
    	
    	RESTOREREG R0  
    
    	BLOAD R0 R2
    	ADDREG R0 2
    	
    	BLOAD R0 R1
    	RADD R2 R1 
    	VMCALL sub_check_routine
    		
    tricky:
    	
    	ENCRYPTEDCLEAR COUNTER  		; This one get patched at run time!
    
    cracked:	
    	LOADREG COUNTER
    	LOADPTR congrats
    	APICALL printf	
    	CleanStack 8 
    	BR outout
    	
    youwishdude:
    
    	LOAD 0
    	LOADPTR notgood
    	APICALL printf	
    	CleanStack 8
    outout:	
    	STOPVM
    
    
    sub_check_routine:
    
    	MOVE 'L' R1
    	INCR R1 
    	INCR R1	
    	ADDREG R1 5 
    	DECR R1 
    	SUBREG R1 4 
    	SUBREG R2 'Z'
    	CMPREG R1 R2
    	BNZ youwishdude
    
    	DECR R0         
    	BLOAD R0 R2     
    	ADDREG R0 2     
    	BLOAD R0 R1     
    	RADD R2 R1      
    	INCR R2         
    	SUBREG R2 4Eh
    	
    	LOADPTR retdecrypt-0DEADh 	; push ptr to patch
    	RESTOREREG R0         		
    	ADDREG R0 0DEACh      		
    	INCR R0               		
    	VMXOR R2 BPTR R0	
    
    	MOVE msgcrypt COUNTER
    	LOADPTR goodboy
    	RestoreREG R0
    	INCR R2
    
    decryptmsg:
    
    	VMXOR R2 BPTR R0
    	INCR R0
    	DECR COUNTER
    	BNZ decryptmsg
    
    retdecrypt:	
    
    	VMRETCRYPTED
    
    
    
    [email protected]:
    
    notgood		db	"Please Authenticate!",10,13,0
    
    
    goodboy:
    congrats	db	"Welcome...",10,13
    		db  	"Exploit for it doesn't matter 1.x Courtesy of Nicolas Brulez",0 
    goodboyend:
    

    This little routine is the password protection used in the binary. For more informations about it, i will let you read the submissions. As you can see, the password protection is VERY SHORT. I could have written a very complex algo with hundreds of lines to make it harder to analyse. Also this code is clear of junk. I could also have placed P-code Junk instructions inside the program. The password check was very simple and sadly, some people concentrated on the password rather than the Virtual Machine. Next time i will make it a lot more complex so people has no choice but to analyse the Virtual Machine and Instruction set.

    You can compare the original P-code program here with the one inside the submissions to realize that they have done a very good job.

    A Few notes regarding the Password Protection

    The password is checked using a very simple algorithm, but it is also used to decrypt yet another part of the pcode program. There is a little weakness allowing one to find the correct value without any brute forcing or analysis of the Opcodes:

    Here is the encrypted String :

    		0x14, 0x26, 0x2F, 0x20, 0x2C, 0x2E, 0x26, 0x6D,
    		0x6D, 0x6D, 0x49, 0x4E, 0x06, 0x3B, 0x33, 0x2F,
    		0x2C, 0x2A, 0x37, 0x63, 0x25, 0x2C, 0x31, 0x63,
    		0x2A, 0x37, 0x63, 0x27, 0x2C, 0x26, 0x30, 0x2D,
    		0x64, 0x37, 0x63, 0x2E, 0x22, 0x37, 0x37, 0x26,
    		0x31, 0x63, 0x72, 0x6D, 0x3B, 0x63, 0x00, 0x2C,
    		0x36, 0x31, 0x37, 0x26, 0x30, 0x3A, 0x63, 0x2C,
    		0x25, 0x63, 0x0D, 0x2A, 0x20, 0x2C, 0x2F, 0x22,
    		0x30, 0x63, 0x01, 0x31, 0x36, 0x2F, 0x26, 0x39,
    		0x43
    

    Everyone knows that a C String ends with a null byte. Therefore, the value used to encrypt this string is 0x43. The key is the last byte of the encrypted string. X xor 0 = X. :-)

    The other possible ways to find the good value was to look at the code structure.. We were doing a Call routine, therefore we must have an instruction to do a RET. This instruction is the Virtual RET implemented in the Virtual Machine. From this, we just had to find the opcode of this instruction to compute the key.

    3 - Provide a mean to "quickly" analyse this uncommon feature.

    With this question, i was expecting a disassembler for my Virtual Machine. A few people sent me fully working disassemblers, so i didn't write yet another one. I invite you once again to have a look at their submissions.One of the author emailed me after the deadline with a working IDA processor module with source code included. This Processor module wasn't used in my judgement because it was sent after the deadline, but it is well worth studying it. It will be uploaded on the honeynet web site shortly after the publication of the Results.

    4 - Which tools are the most suited for analysing such binaries, and why?

    In my opinion the best tools to analyse such binaries are Interactive Disassemblers or CPU emulators. The disassembler can be used to analyse the code statically, to remove the obfuscations, to decrypt binaries etc. If it offers possibility to write processor module, you can even write a disassembler for the Virtual Machine and thus, do a full static analysis of the whole thing. A CPU emulator can be used to quickly decrypt the code , layers etc. If it can be scripted not to show the obfuscations you have a perfect weapon. I don't like Debuggers because they aren't reliable. I could have easily written a driver to hook debug interupts to decrypt the binary for instance. Debuggers would have been useless and would have rebooted the computer if used.

    5 - Identify the purpose (fictitious or not) of the binary.

    This binary is waiting for an user to authenticate with a password that is passed to the application through the command line. Once the user has been identified, the binary will print a little message. It looks like a fake exploit. In the real world, it could have been a real exploit protected from prying eyes.

    6 - What is the binary waiting from the user? Please detail how you found it.

    The binary is waiting for a password through the command line. The password is used to access the real program. To find this password, you have to Reverse Engineer the binary. Decrypt every layers to access the Virtual Machine. This Virtual Machine has a virtual program used to check the password entered. One has to Reverse Engineer the Virtual Machine (or trace it blindly) in order to understand its instruction set. Then it is just a very simple algo using a few easy operations to reverse. I invite you to read submissions for details about the algo itself.

    Bonus Question - What techniques or methods can you think of that would make the binary harder to reverse engineer?

    This binary has a lot of security flaws that were left on purpose and it has a lot of things needing improvements.

    Conclusion

    Anti Reverse Engineering Techniques can be used to really slow down the analysis of a binary. Malwares could be using such techniques in a near futur and it is time to get used to it. Even though most of the malwares are programmed by clueless idiots without any programming skill, there is a minority able to write complex code. In the futur we could find exploit binaries on compromised systems that would be protected against Reverse Engineering to hide the vulnerability exploited. Spywares could also use such techniques to hide their activity. This binary had a lot of vulnerabilities, yet it was really challenging , even with a trivial password protection algorithm. The protection has been written within a week (a few hours per day), so with a little more effort, it can be a LOT harder.Finally, I would like to point out that Reverse Engineering isn't a pirate technique and that it is used by the Security Community on a daily basis. Some people in France doesn't seem to agree though..

    Acknowledgement

    I would like to thank the following people:

    About the Author

    Nicolas Brulez

    Chief of Security for Digital River woking on the SoftwarePassport/Armadillo protection system, Nicolas specializes in anti-reverse engineering techniques to defend against software attacks. He has been active in researching viral threats and sharing that research with various anti-virus companies. He regularly writes for the French security magazine MISC and has authored a number of papers on reverse engineering. He currently teaches assembly programming and reverse engineering in French engineering schools.

    The author has more than 7 years of Reverse Engineering Experience on Windows Operating Systems and is currently doing Research on Pocket PC devices. He plans to write a Protection system for those devices.


    Nicolas Brulez - 0x90(at)rstack(dot)org
    Chief of Security - The Armadillo Software Protection System
    http://www.siliconrealms.com/armadillo.htm