Skip to content
λmaldev wiki/
pagesYARA Memory Scanning
T1622WindowsLinuxmacOSYARAPythonVolatility

YARA Memory Scanning

updated 2026-08-0410 min readthehackersbrain
Authorized use only. This material is published for detection engineering, malware analysis and authorized red-team engagements. Running these techniques against systems you do not own or have written permission to test is illegal.

Overview

YARA was designed for file scanning, but its most valuable application in malware detection is against live process memory. Running a rule set across every mapped region of every process finds:

  • Decrypted shellcode that was never written to disk
  • PE headers in private memory (hollowed or reflectively-loaded images)
  • Beacon configuration blobs embedded in allocated buffers
  • Direct-syscall stubs in non-image executable memory

Memory scanning is noisy on production systems without careful tuning. The pages below walk through writing effective rules, limiting scan scope to high-signal regions, and deploying them in both offline forensics (Volatility) and continuous on-host contexts.

Note.

Scanning every page of every process is expensive. A practical on-host scanner focuses on private (MEM_PRIVATE), executable (PAGE_EXECUTE_*) regions. That filter reduces the scan surface by 90% or more on a typical Windows host while covering all the high-value injection targets.

Anatomy of a useful memory rule

A memory rule differs from a file rule in two ways: it cannot rely on PE structure (there may be no headers), and it must tolerate variable base addresses and ASLR. Good memory rules use byte strings, not PE section references:

rules/process_hollow.yarYARA
rule ProcessHollow_MZ_In_Private_Memory
{
  meta:
      description = "MZ header mapped as MEM_PRIVATE — not backed by a file on disk"
      author      = "maldev-wiki"
      mitre       = "T1055.012"

  strings:
      $mz     = { 4D 5A }   // MZ magic
      $pe_sig = { 50 45 00 00 } // PE\0\0 from NT headers offset

  condition:
      // MZ at offset 0 of the region, PE sig within the first 512 bytes
      $mz at 0 and $pe_sig in (0..512)
}
rules/direct_syscall.yarYARA
rule DirectSyscall_Stub
{
  meta:
      description = "mov r10,rcx / mov eax,imm / syscall in non-image memory"
      author      = "maldev-wiki"
      mitre       = "T1106"

  strings:
      // 49 89 CA   = mov r10, rcx
      // B8 ?? ?? ?? ?? = mov eax, <SSN>
      // 0F 05      = syscall
      $stub = { 49 89 CA B8 ?? ?? ?? ?? 0F 05 }

  condition:
      $stub
}
rules/cobalt_beacon.yarYARA
rule CobaltStrike_Beacon_Config
{
  meta:
      description = "XOR-0x69 encoded Cobalt Strike beacon configuration block"
      author      = "maldev-wiki"
      reference   = "SentinelOne beacon config parser"
      mitre       = "T1071.001"

  strings:
      // Config block starts with a magic that XORs to 0x0001 after key
      // This is the decoded magic after XOR with 0x69
      $magic = { 00 01 }
      // Typical config field type markers after decoding
      $field_marker = { 00 01 00 01 }

  condition:
      for any i in (1..#magic) : (
          uint16(@magic[i]) == 0x0100
      )
}

Scanning live memory in Python

scan.pyPython
import ctypes, ctypes.wintypes as wt, yara, sys

# Compile all rules from a directory
rules = yara.compile(filepaths={
  f: f for f in __import__('glob').glob('rules/*.yar')
})

PROCESS_VM_READ  = 0x0010
PROCESS_QUERY_INFORMATION = 0x0400
MEM_COMMIT       = 0x1000
PAGE_EXECUTE_READ      = 0x20
PAGE_EXECUTE_READWRITE = 0x40
PAGE_EXECUTE_WRITECOPY = 0x80

class MEMORY_BASIC_INFORMATION(ctypes.Structure):
  _fields_ = [
      ("BaseAddress",       wt.LPVOID),
      ("AllocationBase",    wt.LPVOID),
      ("AllocationProtect", wt.DWORD),
      ("RegionSize",        ctypes.c_size_t),
      ("State",             wt.DWORD),
      ("Protect",           wt.DWORD),
      ("Type",              wt.DWORD),
  ]

def scan_pid(pid):
  k32  = ctypes.windll.kernel32
  proc = k32.OpenProcess(PROCESS_VM_READ | PROCESS_QUERY_INFORMATION,
                         False, pid)
  if not proc:
      return

  addr  = 0
  mbi   = MEMORY_BASIC_INFORMATION()
  sz    = ctypes.sizeof(mbi)
  EXEC  = {PAGE_EXECUTE_READ, PAGE_EXECUTE_READWRITE, PAGE_EXECUTE_WRITECOPY}

  while k32.VirtualQueryEx(proc, addr, ctypes.byref(mbi), sz):
      # only scan committed, private, executable pages
      if (mbi.State == MEM_COMMIT and
              mbi.Protect in EXEC and
              mbi.Type == 0x20000):  # MEM_PRIVATE
          buf = (ctypes.c_char * mbi.RegionSize)()
          read = wt.SIZE_T(0)
          k32.ReadProcessMemory(proc, addr, buf, mbi.RegionSize,
                                ctypes.byref(read))
          if read.value:
              matches = rules.match(data=bytes(buf))
              for m in matches:
                  print(f"[!] PID {pid} @ {addr:#x} — {m.rule}")
      addr += mbi.RegionSize

  k32.CloseHandle(proc)

if __name__ == "__main__":
  scan_pid(int(sys.argv[1]))

Offline scanning with Volatility 3

volatility.shshell
# Scan all process memory in a captured image
$ vol -f memory.raw windows.vadyarascan     --yara-rules rules/process_hollow.yar     --pid 0    # 0 = all processes

# Target a specific process by name
$ vol -f memory.raw windows.vadyarascan     --yara-rules rules/direct_syscall.yar     --pid $(vol -f memory.raw windows.pslist | grep svchost | awk '{print $2}')

# Search for strings across all memory (quick triage)
$ vol -f memory.raw windows.vadyarascan     --yara-string "/C2_HOSTNAME/"    # grep-style for IOC hunting

Rule quality checklist

Criterion Guidance
Specificity Rule should not match anything in a clean Windows install
Resilience Avoid offsets that change with ASLR; use relative byte sequences
Performance Limit wide patterns (?? wildcards); anchor with short fixed strings
Scope Memory rules: add pe.is_pe only if the target is a full PE; omit for shellcode
Version coverage Test against multiple Windows versions and build dates

False-positive rate on production is the hardest thing to get right. An on-host scanner that generates one alert per host per day will be ignored within a week. Tune against a known-clean golden image before deploying to production, and suppress on the %SystemRoot% path if scanning on-disk.

Detection

YARA
Rules that match on structural byte patterns in private RX memory regions, not just file scans.
VOLATILITY
yarascan plugin applies YARA rules across all process memory maps, including hidden regions.
EDR
Modern EDR engines run YARA-equivalent signature matching on memory pages at allocation and protection-change events.

Variants & related

T1055.012Process HollowingMZ header in private memory is a canonical YARA target.→T1027.009Encrypted Shellcode LoaderScanning after decryption catches the plaintext shellcode before it executes.→T1106Direct SyscallsThe mov r10,rcx / syscall stub is a compact, reliable YARA signature.→
Was this page useful?edit this page ↗