Static/Signature Detection

YARA is one of the tools used for static detection. Foe example if a shellcode contains a byte sequence that begins with FC 48 83 E4 F0 E8 C0 00 00 00 41 51 41 50 52 51 then this can be used to detect that the payload is Msfvenom’s x64 exec payload. The same can be used with strings inside the binary.

So, it is advisable to avoid hardcoded values and instead dynamically retrieve or calculate the values.

Hashing Detection

Saving the hash (MD5 or SHA256) of the malware and comparing with the security solution’s database.

Heuristic Detection

• Static Heuristic Detection - Involves decompiling the suspicious program and comparing code snippets to known malware that are already known and are in the heuristic database. If a particular percentage is matched, the program is flagged.
• Dynamic Heuristic Detection - Suspicious behavior is analyzed by running it inside a sandbox environment.
  • Connecting to the internet to fetch shellcode.
  • Allocating RWX memory.
  • Writing shellcode to memory and executing it.
    • The sequence is considered as malicious behavior.

Behavior-based Detection

Eg. If the malware performs process injection into notepad.exe and connects to the internet, this will likely cause the process to be terminated immediately due to the high likelihood that this is a malicious activity.

To avoid behavior based detection make the malware process behave as benign as possible, like avoiding spawn a cmd.exe child process.

API Hooking

This is a combination of real-time and behavior based detection.

EDRs hook to the commonly abused APIs and then analyze the parameters of these APIs in real time. As the parameters passed will be de-obfuscated or decrypted.

How to bypass?

• DLL Unhooking
• Direct Syscalls

These topics will be covered in further modules.

IAT Checking