How to do malware reverse engineering

Share this…

Malicious Software can be Virus, Worm, Trojan Horse, Rootkit, Bot, DoS Tool,Exploit kit, Spyware. The objective of malware analysis is to gain an understanding of how specific pieces of malware work. There are important questions that must be answered. Like, how did this machine become infected and what exactly does this malware do? In this article we will try to understand with the help of information security solutions expert Bill Smith, the basics of malware analysis and how you can start doing malware analysis.

Who Analyzes Malware?

There are different kinds of people and organization that do malware analysis. All of them fall under these categories:

  • CSIRTs
  • Security product developers
  • Security service providers
  • Anti-malware researchers
  • Software developers
  • Law enforcement

Why there is a need to Analyze Malware?

Following are the reasons behind malware analysis.

  • To have an incident response procedure.
  • For doing product development and product improvement like antivirus.
  • For creating signatures for protection against malware.
  • To create countermeasure solutions.
  • To do analysis and resolution of vulnerability.
  • To track and catch the criminals who create malware.

Malware Analysis Methods

As per information security training experts, to do analysis of malware you have to follow these steps:

1. Environment Setup

Set up a controlled machine, which is not connected to your network, also you should be able to restore the machine anytime.

2. Malware collection

For environment setup you need to download the malware file first, and then you need to change its extension. As per suggestions of ethical hacking training experts, after changing the file you can copy the file in write protected disc as this can help you isolate malware in some cases.

3. Surface analysis

Retrieve surface information from targets without execution. Motive of surface analysis is to get

  • Hash value
  • File type
  • Strings
  • Anti-virus software results

4. Runtime analysis

In this step you can execute malware and monitor its behavior. You can use various automated or manual analysis methods. You can use monitoring tools on sandbox system for analysis. The whole environment can be dedicated or isolated Native OS or Virtual System explains Mike Stevens, specialist of information security training course.


5. Static Analysis

In static analysis you read the code in binary file and understand its functionality. You will need OS knowledge, assembly basics, efficient reading techniques and anti-analysis techniques. If the binary code is packed you will have to unpack it. Also to understand the binary, you will have to decompile o disassemble/debug the binary.

Following tools can be used for static analysis:


IDA Interactive DisAssemble: Disassembles more than 50 architectures



Hex-rays Decompiler: x86/ARM binary to C source code.

VB Decompiler: Visual Basic binary to Visual Basic source code

.NET Reflector: .NET binary to .NET source code




OllyDbg: World famous x86 debugger

Immunity Debugger: Python familiar x86 debugger



To understand the code you can start with windows API from MSDN Library and understand what the API does. You can also check for arguments and brunch on condition. While you use a Disassembler, you can read, rename and comment instructions to understand the code. You can learn more about Disassembler in ethical hacking training course.

6. Encoding (Obfuscation) in Malware

Sometimes the programmer will encode the code to make it difficult for you to do analysis.

Strings are encoded like File name, Registry entry name, Server address stored in the binary and also HTTP data packets can be encoded using various methods. Some of the encoding methods are

  • xor (exclusive or)
  • ror/rol (rotate right/left)
  • base64
  • RC4
  • AES




Also all the malware used these days use C&C server (command and control server) to get the commands and send back the results and data. Hackers can create C&C servers using hacked servers, websites or email accounts. Twitter & Facebook accounts can also be used as C&C server so that hackers can be traced back. How to create C&C server using social media accounts is a part of information and data security training of iicybersecurity.

7. Anti Runtime Malware Analysis

Some types of malware are clever enough to detect analysis activity thus have a logic to avoid analysis by malware analysts explains information security solutions expert Bill Smith. Some of the techniques used to detect malware analysis are:

Debugger: To check for debuggers the malware checks for Breakpoints, Exception handling.

Virtual Machine: To check for virtual machine the malware checks for Interface, CPU behavior, Support tools (like Virtual box).

Analysis tools: To check for malware analysis tool like IDAPro the malware checks for Window name, Module name.




The malware sometimes also check for Computer name, Disk size, Cursor position to avoid malware analysis and after detecting that the malware analysis is being done the malware does something else or does nothing. We will cover more in-depth details over malware analysis in next article with the help of Mike Stevens, ethical hacking training professor.