Shellzer : a tool for the dynamic analysis of malicious shellcode

This thesis describes shellzer, a tool for analyzing shellcode extracted from web-based malware and malicious PDF documents. Shellzer takes in input the shellcode that has to be analyzed, and the additional resources that are essential in order to carry out a correct analysis. As an output, the tool returns a report that contains detailed informations about the actions performed by the shellcode, including the Windows APIs that have been called (with their arguments and return values), the URLs associated with the resources downloaded from the web, and the additional malware that has been retrieved, decrypted and finally executed by the shellcode. The functioning of the tool is based on the PyDbg library, that allows to dynamically instrument a generic Windows executable. Specifically, the shellcode's execution has been instrumented at single-instruction level. The analyzer has been designed in a way to automatically handle all the evasion techniques that are commonly used by shellcodes. Furthermore, we emulate that the shellcode is executed in a specific execution context, and we simulate that all the external resources required by shellcode are available, even if, actually, this is not the case: otherwise, some shellcode end their execution beforehand, while others simply crash. This is achieved by dynamically modifying the arguments and the return value of some specific APIs, when they are called. This technique has also been used in order to avoid that the shellcode could compromise the analyzer during its execution. Moreover, for performance reasons, a loop detector algorithm has been implemented, in a way that we avoid to instrument at single-instruction level the execution of a piece of code that has already been analyzed. This tool has been tested with over 24000 shellcode that have been previously extracted by Wepawet, and only in the 2% of the cases it has not be possible to analyze them due to a shellzer's limit. At the end of the thesis, we also provide a detailed overview about the goals of the shellcode we have analyzed.