Skip to content
← Ethical Hacking · advanced · 14 min · 19 / 31

Malware Analysis

Static and dynamic analysis, sandbox execution, YARA rules, deobfuscation, and reverse engineering malicious code.

malware analysisstatic analysisdynamic analysisYARAsandboxreverse engineeringdeobfuscationGhidra

Real-World Analogy

A bomb disposal technician doesn’t defuse a device by intuition — they study the mechanism first: what type of trigger, what circuits, what order of operations. Malware analysis is the same: understand what it does before you touch it.

Lab Setup (Safety First)

# NEVER analyze malware on your main machine
# Use an isolated VM with:
# - No host-only adapters that malware could pivot to host
# - Snapshot before analysis (restore after)
# - Flare-VM (Windows) or REMnux (Linux) — pre-configured analysis distros

# Install Flare-VM (Windows analysis)
# https://github.com/mandiant/flare-vm

# Install REMnux (Linux analysis)
# https://remnux.org/

# Network isolation options:
# - NAT but with FakeNet-NG (intercepts network calls, doesn't let them out)
# - Host-only adapter (can reach host but not internet)
# - No network (for most static analysis)

Static Analysis — No Execution

Examine the file without running it.

File Identification

# Identify the file type (don't trust the extension)
file malware.exe
file malware.pdf   # might actually be an EXE

# Check hashes — search VirusTotal
md5sum malware.exe
sha256sum malware.exe
# Search: virustotal.com (68 AV engines check it)

# Quick automated check
curl -s --request POST \
  --url 'https://www.virustotal.com/vtapi/v2/file/scan' \
  --form apikey='YOUR_KEY' \
  --form file=@malware.exe | jq '.permalink'

Strings Extraction

# Basic strings
strings malware.exe
strings -n 8 malware.exe   # minimum 8 chars (reduce noise)
strings -e l malware.exe   # Unicode strings (Windows malware often uses Unicode)

# Look for:
# - URLs and IPs (C2 servers)
# - Registry keys (persistence)
# - File paths (what does it create/modify)
# - Function names (API calls reveal behavior)
# - Error messages (reveal programming language/compiler)
# - Encoded strings (base64, hex — sign of obfuscation)

strings malware.exe | grep -iE "(http|https|ftp)://"
strings malware.exe | grep -iE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}"
strings malware.exe | grep -i "HKEY_\|SOFTWARE\\\|Run"

PE Header Analysis (Windows)

# PE files = Windows executables
# PEiD, Detect-It-Easy — identify packer/compiler
die malware.exe  # Detect-It-Easy CLI
# Output: "PE64, Microsoft Visual C++ 2019"
# Or: "Packed: UPX 3.96" → needs unpacking

# PEview — examine PE structure
peview malware.exe

# Python: pefile
pip install pefile
python3 << EOF
import pefile
pe = pefile.PE('malware.exe')

# Compilation timestamp
print("Compile time:", pe.FILE_HEADER.TimeDateStamp)

# Imports — what Windows APIs does it call?
for entry in pe.DIRECTORY_ENTRY_IMPORT:
    print(f"\nDLL: {entry.dll.decode()}")
    for imp in entry.imports:
        if imp.name:
            print(f"  {imp.name.decode()}")
EOF

Suspicious Windows API imports:

CreateRemoteThread   → process injection
VirtualAllocEx       → allocating memory in other processes
WriteProcessMemory   → writing to other processes
SetWindowsHookEx     → keylogging
CreateService        → persistence as service
RegSetValueEx        → registry persistence
WinExec / ShellExecute → executing commands
InternetOpenUrl      → network communication
CryptEncrypt         → encrypting data (ransomware?)
FindFirstFile/FindNextFile → file enumeration (ransomware?)

Unpacking

Packed malware compresses/encrypts the real code. Dynamic analysis often forces unpack.

# UPX (most common)
upx -d malware.exe   # decompress
upx -d -o unpacked.exe malware.exe

# Generic unpacking:
# 1. Run in debugger
# 2. Wait for OEP (original entry point) — where real code starts
# 3. Dump memory at that point
# Tools: x64dbg + ScyllaHide plugin (PE Rebuilder)

Dynamic Analysis — Controlled Execution

Run the sample and watch what it does.

Process Monitor (Procmon)

Procmon — records all file, registry, and network activity

Filters to set:
Process Name is malware.exe

Watch for:
- CreateFile → what files does it create/read?
- WriteFile → what does it write?
- RegSetValue → what registry keys does it set?
- Process Create → does it spawn child processes?

Network Traffic Analysis

# Wireshark — capture while malware runs
wireshark &
# Start capture, run malware (in VM!), stop capture

# FakeNet-NG — intercepts network calls, simulates internet services
# Prevents real C2 connection, captures what the malware tries to do
fakenet --config-dir configs/

# Inetsim — simulate DNS, HTTP, FTP, SMTP servers
inetsim
# All DNS queries return FakeNet IP, HTTP connections are served generic pages
# Malware makes real network calls → captured by Inetsim

Sandbox Analysis (Automated)

# Any.run — interactive sandbox (free tier)
# https://app.any.run — upload file, see live behavior

# Cuckoo Sandbox — self-hosted
pip install cuckoo
cuckoo init
cuckoo -d   # start in debug mode
curl -F "file=@malware.exe" http://localhost:8090/tasks/create/file

# VirusTotal sandbox
# Upload → Behavior tab → see API calls, network, files

# Joe Sandbox
# Hybrid Analysis (https://hybrid-analysis.com) — free tier

Disassembly and Decompilation

# Ghidra — NSA's free decompiler (best free option)
# https://ghidra-sre.org

# Open project → Import file → Analyze → CodeBrowser
# Decompiler view: C-like pseudocode from assembly

# Radare2 — open source, CLI-based
r2 malware.exe
# Commands:
# aaa         → analyze all
# afl         → list functions
# pdf @main   → print disassembly of main
# VV          → visual graph mode
# ?           → help

# x64dbg — Windows debugger for dynamic analysis
# Open malware.exe → set breakpoints → step through execution
# F7: step into   F8: step over   F9: run   F2: breakpoint

# IDA Pro (commercial, industry standard)
# Community version available for non-commercial use

Common Obfuscation Techniques

# XOR encoding — simple but common
key = 0x41
encoded = bytes([b ^ key for b in encoded_shellcode])

# Base64 PowerShell (common in phishing)
# Decode:
import base64
cmd = "cG93ZXJzaGVsbCAtZW5jb2RlZA=="
print(base64.b64decode(cmd).decode('utf-16'))

# String reversal
reversed_url = "moc.live/nettekcer.www//:sptth"[::-1]

# ROT13
import codecs
codecs.decode("GERIRY", "rot13")  # TREVER

# CyberChef handles all of these automatically

YARA Rules — Malware Detection

YARA is a pattern-matching tool for classifying malware families.

# Write a YARA rule
rule Ransomware_WannaCry {
    meta:
        author = "Analyst"
        description = "WannaCry Ransomware"
        hash = "db349b97c37d22f5ea1d1841e3c89eb4"

    strings:
        $s1 = "tasksche.exe" ascii
        $s2 = "WanaCrypt0r" ascii wide
        $s3 = "WANNACRY" ascii nocase
        $pdb = "C:\\projects\\wannacry" ascii
        
        $ransom_note = {
            57 61 6E 6E 61 43 72 79 70 74  // WannaCrypt in hex
        }

    condition:
        uint16(0) == 0x5A4D          // MZ header (PE file)
        and filesize < 10MB
        and (2 of ($s*) or $ransom_note or $pdb)
}
# Scan with YARA
yara rule.yar malware.exe
yara rule.yar /suspicious/directory/ -r   # recursive

# YARA rule sources:
# - Signature-Base by Florian Roth (github.com/Neo23x0/signature-base)
# - Awesome YARA rules collection
# - VirusTotal Intelligence (paid)

# Compile rules for performance
yarac rules.yar rules.yrc
yara rules.yrc malware.exe

Behavioral Analysis Techniques

# Regshot — take registry snapshot before and after malware runs
# Diff shows exactly what registry keys were added/modified/deleted

# Process Hacker — real-time process, memory, and network monitoring
# More powerful than Task Manager for malware analysis

# API Monitor — hook API calls, see exact parameters
# Reveals: file paths, registry keys, URLs, crypto operations

# Detect command and control (C2) communication patterns
# - Periodic beaconing (every N seconds) → likely C2 heartbeat
# - DNS queries for random-looking domains → DGA (Domain Generation Algorithm)
# - Large outbound data → exfiltration
# - HTTPS to single IP (not a CDN) → encrypted C2

# DGA detection — compute Alexa-based score
python3 dga_detector.py suspicious_domain.com

Real Project: Analyze WannaCry

# Get the WannaCry sample (from theZoo malware repository - controlled environment only)
# sha256: 24d004a104d4d54034dbcffc2a4b19a11f39008a575aa614ea04703480b1022c

# In isolated VM:
# 1. Strings analysis
strings wannacry.exe | grep -iE "(http|https|\.onion|bitcoin)"

# 2. PE imports — look for crypto APIs
python3 -c "
import pefile
pe = pefile.PE('wannacry.exe')
for entry in pe.DIRECTORY_ENTRY_IMPORT:
    print(entry.dll.decode())
    for imp in entry.imports:
        if imp.name and b'Crypt' in imp.name:
            print(' ', imp.name.decode())
"

# 3. Open in Ghidra — find the encryption routine
# Search for CryptEncrypt, CryptImportKey, CryptGenRandom

# 4. Find the killswitch domain:
strings wannacry.exe | grep -i "www\."
# Found: www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com

# 5. Run in Cuckoo sandbox — see complete behavior report