Reverse engineering is the process of analysing and understanding a system, a process etc. It is not solely restricted to software: you can reverse engineering watches, televisions, cars, bikes, camera - basically any system you can think of.
In cybersecurity, we usually deal with reverse engineering software: analysing executables using several tools and techniques to understand what it does, how it performs those actions and act on it. The primary application area is in malware analysis where analysts take apart a malware(virus, worm, trojans, bots etc) to understand how it works and develop defences against them. Software reverse engineering is also used in other areas such as finding vulnerabilities, performance and binary analysis and more. It is also an independent field of research on it’s own. There are several job opportunities for skilled reverse engineers in the field since it is not a very popular skill and there is a shortage of skilled reverse engineers in the industry.
(Off late, reverse engineering hardware has also gained a lot of attention but it requires significantly advanced knowledge and several specialized tools. Hence we will not focus on hardware reversing here.)
How to reverse engineer software?¶
There is no standard methodology or technique to reverse engineering software. Every executable you analyse will likely require a different approach depending on the executable itself and what your objective for the reverse engineering process is. However, we will some basic techniques and approaches that will be useful and required in most scenarios. Additionally, these are a necessary foundation for developing further skills in reverse engineering.
Reverse engineering analysis approaches¶
Broadly speaking, there are two approaches to analysing and reverse engineering executables:
Static analysis: In this approach, we do not execute the executable. Instead, we use specialized tools and prior knowledge to infer several useful information about the executable.
Dynamic analysis: In this approach, we execute the executable and observe its behaviour during execution, using a different set of specialized tools. By observing its behaviour, we can infer useful information about the executable.
In most cases, you will need a combination of static and dynamic analysis to gain a better and complete understanding of the executable. Both analyses approach reveal usually distinct pieces of information and sometimes, one might be faster than the other. By combining the two, the reverse engineering process can become much faster. Don’t worry: we will learn all the tools needed and some examples to demonstrate(and also test your skills in) these tools.
General things to remember before starting¶
Reverse engineering is a difficult and time consuming task. It requires paying great attention to detail and connecting various concepts together to make inferences about program behaviour. Very often, it’s also not necessary to reverse engineering the entire executable: depending on the objective for reverse engineering, it might enough to focus on a specific part of the executable. You will find that you will get frustrated several times through the process: that is very normal! Take breaks and work through it at your own pace. Do not rush through to find the answer; focus on learning concepts and approaches well. We have chosen very simple examples to illustrate the concepts clearly but without your efforts and focus, it will be hard to learn. This is not to discourage you, it is something you have to remember when reversing. Reverse engineering is an extremely fun activity because it’s very engaging and frustrating at the same time. After you arrive at the solution and satisfy the objective, there is a strong sense of joy and accomplishment but that requires in many cases persisting continuously at the task. So it is a difficult task but can be extremely rewarding(financially and personally).
With that said, let’s get started!