Technical Analysis of DLL Injection and API Hooking on Windows
A comprehensive examination of process injection via kernel Asynchronous Procedure Calls (APCs) and function interception utilizing Microsoft Detours.
1. Why This Matters
Modern security solutions, including Data Loss Prevention (DLP) systems, anti-cheat engines, and sophisticated debugging frameworks, utilize two primary methodologies: DLL injection and API hooking. DLL injection involves executing custom code within the address space of a remote process, while API hooking facilitates the interception of calls to system libraries. Mastery of these techniques is fundamental for engineers developing Windows drivers, security software, or low-level system utilities.
This guide analyzes both techniques through first principles, substantiated by practical kernel and user-mode C implementation. It details the architecture of kernel-mode DLL injection into arbitrary user-mode processes via APCs and demonstrates the use of Microsoft Detours to intercept outbound HTTP traffic across diverse TLS stacks prior to encryption.
2. The Windows Process Memory Model
In the Windows memory model, each process operates within an isolated virtual address space. By design, Process A cannot access the memory of Process B without utilizing specific operating system mechanisms. Consequently, modifying or observing the behavior of a target process necessitates injecting code into its address space—a procedure defined as DLL injection.
Figure 1 — Each process has an isolated virtual address space. The kernel is shared. A DLL injected into a process (HookDll.dll ✓) gains full access to that process's memory.
System libraries such as ntdll.dll and kernel32.dll are mapped into every user-mode process at a uniform virtual address, determined during the boot sequence via Address Space Layout Randomization (ASLR). To optimize physical memory usage, the Windows Memory Manager employs a prototype Page Table Entry (PTE) mechanism. This allows multiple processes to share the same physical memory pages for these libraries, effectively reducing the system's overall RAM footprint.
Uniformity in the address space is a functional requirement of the physical page sharing architecture. If system DLLs were loaded at unique virtual addresses for each process, relocation fixups would alter the byte values within the code pages, triggering Copy-on-Write (CoW) operations. This would necessitate private copies for every process, nullifying memory sharing benefits. To prevent this, Windows establishes a constant system-wide base address for core libraries at boot time.
For developers implementing injection, this implies that the absolute virtual address of LoadLibraryW is consistent across all user-mode processes within a single boot session. However, because this address is re-randomized upon every reboot, injection logic must resolve the address dynamically at runtime to ensure stability and compatibility.
3. DLL Injection Techniques
3.1 The Common Approaches
Several established techniques facilitate DLL injection, each presenting distinct advantages regarding privilege requirements, disk residency, and visibility to kernel-level telemetry. Understanding these common methods provides the necessary context to appreciate why the kernel APC methodology, detailed in Section 3.2, represents a significant architectural shift in injection strategy.
3.1.1 CreateRemoteThread + LoadLibraryW
This traditional technique utilizes standard Win32 API functions without requiring a kernel driver. The implementation typically follows a structured five-step sequence:
OpenProcess with PROCESS_VM_WRITE | PROCESS_VM_OPERATION | PROCESS_CREATE_THREAD — obtain a cross-process handle.
VirtualAllocEx — allocate a small RW page inside the target's address space.
WriteProcessMemory — write the DLL path string (wide, null-terminated) into that page.
CreateRemoteThread with start address = LoadLibraryW VA, argument = DLL path page — create a thread in the target that calls LoadLibraryW.
WaitForSingleObject, VirtualFreeEx, CloseHandle — wait for the load to complete, then clean up.
HANDLE hProc = OpenProcess(
PROCESS_VM_WRITE | PROCESS_VM_OPERATION | PROCESS_CREATE_THREAD,
FALSE, targetPid);
/* Allocate space for the DLL path inside the target */
SIZE_T pathBytes = (wcslen(dllPath) + 1) * sizeof(wchar_t);
LPVOID pPath = VirtualAllocEx(hProc, NULL, pathBytes,
MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
WriteProcessMemory(hProc, pPath, dllPath, pathBytes, NULL);
/* Create a thread whose entry point IS LoadLibraryW.
LoadLibraryW(pPath) runs inside the target process. */
HANDLE hThread = CreateRemoteThread(
hProc, NULL, 0,
(LPTHREAD_START_ROUTINE)LoadLibraryW, /* same VA in all processes */
pPath, /* lpLibFileName argument */
0, NULL);
WaitForSingleObject(hThread, INFINITE);
VirtualFreeEx(hProc, pPath, 0, MEM_RELEASE);
CloseHandle(hThread);
CloseHandle(hProc);
Detection surface
ObRegisterCallbacks — EDR kernel callbacks intercept the OpenProcess call and strip PROCESS_CREATE_THREAD from the access mask before returning the handle. The call succeeds but the injector cannot create a thread in the target.
ETW Threat Intelligence (ETW-Ti) fires EtwTiLogCreateThreadRemote on NtCreateRemoteThread — unconditionally, at kernel level, regardless of whether the injector bypassed user-mode hooks via direct syscall.
The newly created thread has StartAddress = LoadLibraryW (inside kernel32.dll). A thread whose entry point is a system DLL export rather than an application function is a high-confidence indicator of compromise.
PsSetCreateThreadNotifyRoutine callbacks registered by EDR drivers fire for the new thread; cross-process thread creation (creator PID ≠ target PID) is flagged immediately.
3.1.2 SetWindowsHookEx
Windows hooks intercept messages before they are delivered to the target window procedure. When a hook is installed system-wide (dwThreadId = 0), the OS must load the hook DLL into every process that processes window messages, so the hook procedure can execute in the thread's own context. The OS performs this load automatically — the injector creates no visible remote thread.
/* Injector — installs a WH_GETMESSAGE hook across all GUI threads */
HMODULE hDll = LoadLibraryW(L"hook_payload.dll");
HOOKPROC proc = (HOOKPROC)GetProcAddress(hDll, "GetMsgProc");
/* dwThreadId = 0 → system-wide scope.
The OS calls LoadLibraryW("hook_payload.dll") inside every process
the next time that process calls GetMessage or PeekMessage. */
HHOOK hHook = SetWindowsHookExW(WH_GETMESSAGE, proc, hDll, 0);
/* hook_payload.dll must export:
LRESULT CALLBACK GetMsgProc(int nCode, WPARAM wParam, LPARAM lParam)
It can do arbitrary work in DllMain DLL_PROCESS_ATTACH. */
The payload DLL must export a valid hook procedure and exist as a resolvable file on disk. A limitation of this method is its reliance on message loops; consequently, it only affects processes that process Win32 window messages. It cannot target console applications, system services, or other non-GUI processes. Additionally, system-wide hooks introduce significant overhead and are highly conspicuous in production environments.
Detection surface
SetWindowsHookExW is a user32.dll export; a userspace EDR hook on it catches all callers that do not invoke the underlying NtUserSetWindowsHookEx syscall directly.
System-wide hooks (threadId = 0) are flagged as high-risk by Windows Defender and most EDR products; WH_GETMESSAGE and WH_CBT are the most commonly abused types.
The payload DLL must be on disk with a valid absolute path resolvable by the target — fileless injection is not possible with this method.
3.1.3 Thread Hijacking (SuspendThread + SetThreadContext)
Rather than spawning a new thread, this method co-opts an existing thread within the target process. The injector suspends the thread, captures its register state, and redirects the Instruction Pointer (RIP) to LoadLibraryW. After the DLL is loaded, a restoration stub re-establishes the original thread context. This approach is stealthier as it does not add an entry to the process's thread list.
OpenProcess + OpenThread(THREAD_SUSPEND_RESUME | THREAD_GET_CONTEXT | THREAD_SET_CONTEXT).
SuspendThread — pause execution of the chosen thread.
GetThreadContext(CONTEXT_FULL) — capture the full register file.
VirtualAllocEx + WriteProcessMemory — write the DLL path and a restore stub into the target.
Patch the context: RCX = DLL path VA (argument 1), RIP = LoadLibraryW VA, RSP aligned to 16 bytes minus 8 (to satisfy the x64 ABI's post-call alignment requirement).
SetThreadContext + ResumeThread — apply the modified context and resume.
CONTEXT ctx = {};
ctx.ContextFlags = CONTEXT_FULL;
SuspendThread(hThread);
GetThreadContext(hThread, &ctx);
/* Write DLL path + a tiny stub that restores ctx and jumps back */
LPVOID pPath = WriteToTarget(hProc, dllPath, pathBytes);
LPVOID pRestore = WriteRestoreStub(hProc, &ctx); /* target-side code */
/* Redirect: RCX = lpLibFileName, RIP = LoadLibraryW */
ctx.Rcx = (ULONG64)pPath;
ctx.Rip = (ULONG64)pfnLoadLibraryW;
/* Push a fake return address pointing to our restore stub */
ctx.Rsp -= sizeof(ULONG64);
WriteProcessMemory(hProc, (LPVOID)ctx.Rsp, &pRestore, sizeof(ULONG64), NULL);
SetThreadContext(hThread, &ctx);
ResumeThread(hThread);
/* Thread executes LoadLibraryW(pPath), then returns to pRestore, which
re-applies the original context and resumes normal execution. */
Thread hijacking is technically demanding and potentially unstable. If the target thread is holding a critical section or performing a timed wait, redirection may lead to deadlocks or application crashes. Professional implementations mitigate these risks by specifically targeting threads in an alertable wait state, ensuring they are idle and not currently holding synchronization locks.
Detection surface
OpenThread with THREAD_SET_CONTEXT is monitored by ObRegisterCallbacks; EDR can strip the access right to prevent SetThreadContext from succeeding.
ETW-Ti EtwTiLogSetContextThread fires on NtSetContextThread when the caller and the target thread belong to different processes — unconditionally visible at kernel level.
During the brief hijack window, the thread's RIP points outside any mapped image region (it points to the DLL path page). Periodic memory scanners that sample thread contexts can catch this transient state.
3.1.4 AppInit_DLLs and the Shim Engine
The AppInit_DLLs registry key is a legacy persistence mechanism. During the initialization of user32.dll, the system reads this key and loads the specified DLLs into the process. This ensures that any application linking to user32.dll—which encompasses the majority of Windows software—automatically loads the payload without further intervention.
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows]
"AppInit_DLLs"="C:\\payload\\inject.dll"
"LoadAppInit_DLLs"=dword:00000001
"RequireSignedAppInit_DLLs"=dword:00000000
; RequireSignedAppInit_DLLs = 1 is enforced (unkillable) when
; UEFI Secure Boot is enabled — unsigned payloads will NOT load.
On Windows 8 and subsequent versions, UEFI Secure Boot enforces the Requirement of signed AppInit_DLLs. Unsigned payloads will fail to load, effectively neutralizing this technique on modern, hardened endpoints. This method remains viable only on legacy systems or environments where Secure Boot is disabled.
The Application Compatibility Shim Engine (shimeng.dll) offers an alternative vector. Shims are loaded by the Windows loader prior to the application's entry point. While abusing this infrastructure requires administrative privileges to install custom databases, it provides a persistent injection mechanism that survives system reboots without requiring binary modification.
Detection surface
AppInit_DLLs is one of the most-monitored registry keys in the Windows security ecosystem; CmRegisterCallbackEx callbacks in EDR drivers catch writes to it in real time.
Shim installs via sdbinst.exe generate Application event log entries (Event ID 1033) and create artifacts under HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\InstalledSDB.
Both methods require the payload to exist on disk at the registered path; fileless variants are not possible.
3.1.5 Reflective DLL Injection
Techniques relying on LoadLibraryW generate several forensic signals, including entries in the Process Environment Block (PEB) and ETW events. Reflective DLL injection bypasses these detections by implementing a custom loader within the DLL itself. This allows the DLL to map itself into memory from a buffer, avoiding standard OS loading procedures and the associated telemetry.
The reflective loader performs the essential tasks typically handled by the Windows loader: it maps PE sections, applies base relocations, and resolves imports. By managing these operations internally, the injected module remains hidden from most standard process enumeration tools.
Locate the DLL's own in-memory image — scan backward from the return address to find the MZ/PE signature.
Allocate a region in the current process using NtAllocateVirtualMemory.
Copy each PE section to its target RVA within the allocation.
Apply base relocations — patch every absolute address for the actual load delta.
Resolve imports — walk the import directory, find each dependency in PEB.Ldr, and resolve function addresses without calling GetProcAddress (to avoid user-mode hooks).
Call the DLL's own DllMain(DLL_PROCESS_ATTACH) directly.
/* Injector: ships raw DLL bytes to the target, not a path string */
LPVOID pRemote = VirtualAllocEx(
hProc, NULL, dllFileSize,
MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);
WriteProcessMemory(hProc, pRemote, rawDllBytes, dllFileSize, NULL);
/* Compute the RVA of ReflectiveLoader within the on-disk image,
add it to the remote base to get the in-target VA */
DWORD rlRva = FindReflectiveLoaderRva(rawDllBytes);
LPVOID pLoader = (LPVOID)((ULONG_PTR)pRemote + rlRva);
/* One CreateRemoteThread call — ReflectiveLoader maps the rest itself.
The DLL never touches LoadLibraryW; it does not appear in
EnumProcessModules or CreateToolhelp32Snapshot. */
CreateRemoteThread(hProc, NULL, 0,
(LPTHREAD_START_ROUTINE)pLoader, NULL, 0, NULL);
Detection surface
The injected DLL does NOT appear in PEB.Ldr. EnumProcessModules, CreateToolhelp32Snapshot, and any scanner that walks the loader list will not see it. Detection requires a VAD walk: a MEM_PRIVATE + executable region that contains a valid PE header is anomalous — legitimately loaded DLLs are always MEM_IMAGE (mapped from a section object backed by a file on disk).
VadFlags.PrivateMemory = 1 combined with executable protection is the canonical VAD-level indicator of reflective or shellcode injection. The Memory Manager sets PrivateMemory = 1 for allocations made with NtAllocateVirtualMemory; it is never set for file-backed section mappings.
ETW-Ti EtwTiLogReadWriteVm fires on the WriteProcessMemory call that delivers the DLL bytes — unconditionally visible at kernel level.
The LdrLoadDll ETW event is absent. A PrivateMemory executable region without a corresponding LdrLoadDll event is a high-confidence correlation for reflective loading.
Note: Kernel APC injection (the focus of section 3.2) is a delivery mechanism, not a loading mechanism. It can deliver any payload — a plain LoadLibraryW call, a reflective loader stub, or arbitrary shellcode — by queuing the execution as a user-mode APC on a target thread. The APC mechanism bypasses all user-mode syscall monitoring but does not by itself eliminate the LoadLibraryW or LdrLoadDll signals unless combined with a reflective loader.
3.2 Kernel APC Injection — Deep Dive
An APC is a kernel-mode mechanism for executing code in the context of a specific thread. When a user-mode APC is queued, it is delivered as the thread transitions from kernel mode back to user mode. This provides a powerful injection vector, as the APC executes with the full privileges and memory access of the target thread.
The following strategy outlines the implementation of this technique:
Write the DLL file path into the target process's user-mode address space.
Queue a user-mode APC on a target thread, setting the NormalRoutine to LoadLibraryW and NormalContext to the address of the DLL path string.
When the thread next transitions to user mode, the kernel delivers the APC, calling LoadLibraryW(dllPath) in the target process — loading the DLL.
Figure 2 — The six-phase kernel APC injection sequence, from process detection through DLL load.
3.3 The Injection Code — Step by Step
Phase 1: Capture LoadLibraryW's Address
Due to the consistent nature of system DLL mapping in the Windows memory model, a kernel driver only needs to resolve the address of LoadLibraryW once per boot session. This captured address remains valid for all subsequent injection operations until the next system restart.
The driver utilizes PsSetLoadImageNotifyRoutine to register a callback that monitors image mapping. By inspecting the file paths of loaded modules, the driver can identify kernel32.dll and resolve the export address of LoadLibraryW through a manual walk of the PE export table, storing the result for future use.
The mechanism is PsSetLoadImageNotifyRoutine. This kernel API registers a callback that fires every time any PE image (executable or DLL) is mapped into any process's address space, or into the kernel itself. The callback fires inside the thread that triggered the load — for user-mode DLLs, that is a thread inside the loading process. The driver inspects the image file name on every callback, looking for a path that ends in \kernel32.dll. When found, it walks the PE export table directly in memory to resolve LoadLibraryW's virtual address, then stores it atomically.
Callback registration (DriverEntry)
/* Register once in DriverEntry */
NTSTATUS status = PsSetLoadImageNotifyRoutine(VgFltLoadImageNotify);
if (!NT_SUCCESS(status)) {
/* Non-fatal: injection will not work but file monitoring still works */
DbgPrint("[driver] PsSetLoadImageNotifyRoutine failed: %08X\n", status);
}
/* Deregister in DriverUnload — mandatory to prevent dangling pointer */
PsRemoveLoadImageNotifyRoutine(VgFltLoadImageNotify);
Callback signature
/* Prototype (ntddk.h):
typedef VOID (NTAPI *PLOAD_IMAGE_NOTIFY_ROUTINE)(
_In_opt_ PUNICODE_STRING FullImageName, // NT device path, e.g.
// \Device\HarddiskVolume3\Windows\System32\kernel32.dll
_In_ HANDLE ProcessId, // PID that loaded the image; 0 for kernel
_In_ PIMAGE_INFO ImageInfo // load address, size, flags
);
*/
VOID NTAPI VgFltLoadImageNotify(
PUNICODE_STRING FullImageName,
HANDLE ProcessId,
PIMAGE_INFO ImageInfo)
{
/* FullImageName can be NULL if the kernel cannot resolve the path */
if (!FullImageName || !ImageInfo) return;
/* Skip kernel image loads (ProcessId == 0) */
if (ProcessId == 0) return;
/* Skip if already resolved — InterlockedCompareExchange prevents
the second-thread overwrite race */
if (InterlockedCompareExchangePointer(&g_pfnLoadLibraryW, NULL, NULL))
return;
/* Match the image name suffix — FullImageName is a full NT device path;
kernel32.dll always ends with \kernel32.dll (case-insensitive) */
UNICODE_STRING suffix;
RtlInitUnicodeString(&suffix, L"\\kernel32.dll");
if (!RtlSuffixUnicodeString(&suffix, FullImageName, TRUE)) return;
/* Walk the PE export table at ImageInfo->ImageBase to find LoadLibraryW */
PVOID resolved = NULL;
__try {
resolved = ResolveExport(ImageInfo->ImageBase, "LoadLibraryW");
} __except (EXCEPTION_EXECUTE_HANDLER) {
DbgPrint("[driver] ResolveExport exception: %08X\n", GetExceptionCode());
return;
}
if (resolved) {
/* Atomic write — all subsequent readers see either NULL or the full VA */
InterlockedExchangePointer(&g_pfnLoadLibraryW, resolved);
DbgPrint("[driver] LoadLibraryW resolved: %p\n", resolved);
}
}
Kernel Data Structures — Phase 1
Kernel APIs — Phase 1
PE Export Table Walk — Annotated
Using the ImageBase and export directory information, the resolution logic performs a standard PE lookup. This operation is performed within a structured exception handler to ensure that malformed or paged-out memory does not compromise kernel stability:
static PVOID ResolveExport(PVOID ImageBase, PCSTR FuncName)
{
/* Step 1: DOS header — e_lfanew is the offset to the NT headers */
PIMAGE_DOS_HEADER dos = (PIMAGE_DOS_HEADER)ImageBase;
if (dos->e_magic != IMAGE_DOS_SIGNATURE) return NULL; /* 'MZ' */
/* Step 2: NT headers — Signature + FileHeader + OptionalHeader */
PIMAGE_NT_HEADERS64 nt = (PIMAGE_NT_HEADERS64)(
(PUCHAR)ImageBase + dos->e_lfanew);
if (nt->Signature != IMAGE_NT_SIGNATURE) return NULL; /* 'PE\0\0' */
/* Step 3: Export directory — DataDirectory[0] */
ULONG expRva = nt->OptionalHeader
.DataDirectory[IMAGE_DIRECTORY_ENTRY_EXPORT]
.VirtualAddress;
if (!expRva) return NULL; /* no exports */
PIMAGE_EXPORT_DIRECTORY exp =
(PIMAGE_EXPORT_DIRECTORY)((PUCHAR)ImageBase + expRva);
/* Step 4: The three export arrays — all contain RVAs, not VAs */
PULONG nameRvas = (PULONG) ((PUCHAR)ImageBase + exp->AddressOfNames);
PUSHORT ordinals = (PUSHORT)((PUCHAR)ImageBase + exp->AddressOfNameOrdinals);
PULONG funcRvas = (PULONG) ((PUCHAR)ImageBase + exp->AddressOfFunctions);
/* Step 5: Linear scan by name (binary search is possible if names are
sorted, which the PE spec requires — but strcmp is fast enough here) */
for (ULONG i = 0; i < exp->NumberOfNames; i++) {
PCSTR exportName = (PCSTR)((PUCHAR)ImageBase + nameRvas[i]);
if (strcmp(exportName, FuncName) == 0) {
/* ordinals[i] is the index into funcRvas[] for this name */
ULONG funcRva = funcRvas[ordinals[i]];
return (PVOID)((PUCHAR)ImageBase + funcRva);
}
}
return NULL; /* function not found in export table */
}
Note: The callback fires with the target process as the current process. ImageInfo->ImageBase is the VA of the DLL inside that process's address space — which, for kernel32.dll, equals the VA in every other user-mode process (same boot-time ASLR base). Reading PE headers from ImageBase is safe because the section is mapped read-only before the callback fires; the __try/__except guard covers the pathological case of a damaged or still-loading image.
Phase 2: Allocate a user-mode buffer in the target
To facilitate data transfer, the driver attaches to the target process's virtual address space. Once attached, it allocates memory and copies the necessary DLL path string. This temporary attachment ensures the data is correctly placed within the target's memory before the APC is queued.
KeStackAttachProcess(IoThreadToProcess(Thread), &apcState);
__try {
st = ZwAllocateVirtualMemory(NtCurrentProcess(), &userBuf, 0, &bufBytes,
MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
if (NT_SUCCESS(st))
RtlCopyMemory(userBuf, DllPath, (PathLen + 1) * sizeof(WCHAR));
} __except (EXCEPTION_EXECUTE_HANDLER) { st = GetExceptionCode(); }
KeUnstackDetachProcess(&apcState);
Phase 3: Initialize and queue the APC
The APC is initialized by specifying the target thread and the resolved address of LoadLibraryW as the entry point. On x64 architectures, the address of the DLL path is passed in the RCX register, satisfying the calling convention for LoadLibraryW without requiring an intermediate wrapper function.
/* Allocate the KAPC structure from non-paged pool */
PKAPC apc = ExAllocatePool2(POOL_FLAG_NON_PAGED, sizeof(KAPC), 'injT');
KeInitializeApc(
apc,
Thread, /* target thread */
OriginalApcEnvironment,
ApcKernelRoutine, /* frees the KAPC pool alloc */
NULL, /* no rundown routine */
(PKNORMAL_ROUTINE)(ULONG_PTR)g_pfnLoadLibraryW, /* user: LoadLibraryW */
UserMode,
userBuf); /* NormalContext → RCX */
/* Queue — returns FALSE if the thread is terminating (benign race) */
KeInsertQueueApc(apc, NULL, NULL, IO_NO_INCREMENT);
Phase 4: APC fires — the kernel routine and user-mode delivery
Upon delivery, a kernel-mode routine first cleans up the APC resources. Subsequently, the thread executes the user-mode payload, effectively loading the DLL into the target process. This seamless transition ensures the injection is completed with minimal impact on the thread's original execution flow.
static VOID NTAPI ApcKernelRoutine(
_In_ PKAPC Apc,
_Inout_ PKNORMAL_ROUTINE *NormalRoutine, /* unchanged — LoadLibraryW */
_Inout_ PVOID *NormalContext, /* unchanged — DLL path VA */
_Inout_ PVOID *SystemArgument1,
_Inout_ PVOID *SystemArgument2)
{
ExFreePoolWithTag(Apc, 'injT');
/* execution continues to LoadLibraryW(userBuf) in user mode */
}
4. API Hooking with Microsoft Detours
4.1 What Is Detours?
Microsoft Detours is a robust library for performing inline function patching. It redirects execution from a target function to a custom hook by overwriting the function prologue with a jump instruction. To maintain original functionality, Detours creates a "trampoline" that preserves the overwritten bytes, allowing the hook to call the original implementation.
The library is available as an open-source project and is typically linked statically into the hook DLL to facilitate low-level interception.
4.2 Inline Hooks vs. IAT Patching
There are two primary methods for API redirection; Detours utilizes the more comprehensive inline patching approach:
IAT (Import Address Table) patching
IAT patching involves modifying the import table of a PE binary. While relatively simple to implement, its scope is limited to calls made through the import table. It fails to intercept calls where function addresses are resolved dynamically via GetProcAddress.
Inline (preamble) patching — what Detours does
In contrast, inline patching overwrites the function's machine code directly. This ensures that all calls to the target function are intercepted, regardless of the resolution method. The use of a trampoline allows for transparent forwarding to the original code when required.
Figure 3 — Left: the original WinHttpSendRequest preamble. Right: after DetourAttach, the first 5 bytes are replaced with a JMP to the hook. A trampoline holds the saved bytes and jumps back into the rest of the original function.
4.3 The Detours Transaction API
Detours manages hook installations through atomic transactions. This approach prevents race conditions and ensures that multiple related hooks are applied simultaneously. A standard implementation pattern involves beginning a transaction, updating the target threads, and committing the changes.
/* Step 1: declare Real_* pointers initialized to the real function addresses */
static PFN_WinHttpSendRequest Real_WinHttpSendRequest =
(PFN_WinHttpSendRequest)WinHttpSendRequest;
static PFN_HttpSendRequestW Real_HttpSendRequestW =
(PFN_HttpSendRequestW)HttpSendRequestW;
/* ... one per hooked function ... */
/* Step 2: install all hooks atomically */
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread()); /* suspend-proof the current thread */
DetourAttach(&(PVOID &)Real_WinHttpSendRequest, Hook_WinHttpSendRequest);
DetourAttach(&(PVOID &)Real_HttpSendRequestW, Hook_HttpSendRequestW);
DetourAttach(&(PVOID &)Real_InternetWriteFile, Hook_InternetWriteFile);
/* ... */
LONG err = DetourTransactionCommit();
/* err == NO_ERROR → all hooks active; Real_* now point to trampolines */
Note: DetourUpdateThread must be called for every thread that might be executing inside a function being patched at the moment of commit. In DLL_PROCESS_ATTACH only the current thread exists in the DLL's context, so one call suffices. For later dynamic re-hooking you would enumerate all threads.
4.4 Writing a Hook Function
A hook function must strictly adhere to the signature of the target API. It acts as an intermediary, inspecting or altering arguments before optionally passing them to the original function via the trampoline. This allows for fine-grained control over system behavior without altering the calling application's logic.
static BOOL WINAPI Hook_WinHttpSendRequest(
HINTERNET hRequest,
LPCWSTR lpszHeaders,
DWORD dwHeadersLength,
LPVOID lpOptional, /* inline POST body */
DWORD dwOptionalLength,
DWORD dwTotalLength,
DWORD_PTR dwContext)
{
/* look up the URL stored when WinHttpOpenRequest was called */
wchar_t url[1024] = {};
ReqGetUrl(hRequest, url, ARRAYSIZE(url));
/* log the body preview to DebugView */
LogBody(url, lpOptional, dwOptionalLength);
/* query the verdict pipe — ALLOW or BLOCK? */
if (!QueryVerdict(url, lpOptional, dwOptionalLength)) {
/* BLOCK: caller sees a send failure */
SetLastError(ERROR_ACCESS_DENIED);
return FALSE;
}
/* ALLOW: call the real function through the Detours trampoline */
return Real_WinHttpSendRequest(
hRequest, lpszHeaders, dwHeadersLength,
lpOptional, dwOptionalLength, dwTotalLength, dwContext);
}
4.5 Removing Hooks on DLL Unload
Proper cleanup is essential during DLL detachment. All hooks must be explicitly removed to prevent the application from attempting to execute jumps to memory that is no longer mapped, which would result in immediate process termination.
if (reason == DLL_PROCESS_DETACH) {
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourDetach(&(PVOID &)Real_WinHttpSendRequest, Hook_WinHttpSendRequest);
DetourDetach(&(PVOID &)Real_HttpSendRequestW, Hook_HttpSendRequestW);
/* ... mirror of every DetourAttach call ... */
DetourTransactionCommit();
}
5. Advanced Hooking — Covering Multiple TLS Stacks
While WinHttp and WinInet are common, modern applications utilize a variety of TLS stacks. A robust security solution must provide comprehensive coverage across different libraries to ensure all outbound traffic is monitored.
Figure 4 — The hook DLL intercepts all outbound traffic by hooking each TLS stack separately. All paths converge on the same verdict pipe before reaching the network.
5.1 Winsock — Raw TCP Plaintext
For raw TCP traffic, hooks are applied to ws2_32.dll exports. These hooks identify and filter out TLS-encrypted records to avoid redundant processing, focusing instead on plaintext payloads that are forwarded to the centralized policy engine.
static BOOL IsTlsRecord(const void *buf, int len) {
if (!buf || len < 3) return FALSE;
const BYTE *b = (const BYTE *)buf;
return (b[0] >= 0x14 && b[0] <= 0x18 && b[1] == 0x03);
}
static int WINAPI Hook_send(SOCKET s, const char *buf, int len, int flags)
{
if (buf && len > 0 && !IsTlsRecord(buf, len)) {
wchar_t url[256]; BuildTcpUrl(s, url, ARRAYSIZE(url));
if (!QueryVerdict(url, buf, (DWORD)len)) {
WSASetLastError(WSAEACCES);
return SOCKET_ERROR; /* BLOCK */
}
}
return Real_send(s, buf, len, flags); /* ALLOW */
}
5.2 SChannel / SSPI — Windows Native TLS
Applications utilizing the native Windows SChannel provider are monitored by hooking EncryptMessage. This facilitates interception of data in its plaintext state immediately prior to encryption. Additionally, the Server Name Indication (SNI) is captured to provide host-level context for the connection.
/* InitializeSecurityContextW — extract the TLS SNI hostname */
static SECURITY_STATUS SEC_ENTRY Hook_InitSecCtxW(
PCredHandle phCred, PCtxtHandle phCtx, SEC_WCHAR *pszTargetName, ...)
{
SECURITY_STATUS st = Real_InitSecCtxW(phCred, phCtx, pszTargetName, ...);
if ((st == SEC_E_OK || st == SEC_I_CONTINUE_NEEDED) && pszTargetName)
CtxStore(phNewCtx->dwLower, phNewCtx->dwUpper, pszTargetName);
return st;
}
/* EncryptMessage — intercept plaintext BEFORE encryption */
static SECURITY_STATUS SEC_ENTRY Hook_EncryptMessage(
PCtxtHandle phCtx, ULONG fQOP, PSecBufferDesc pMsg, ULONG SeqNo)
{
for (ULONG i = 0; i < pMsg->cBuffers; ++i) {
if ((pMsg->pBuffers[i].BufferType & ~SECBUFFER_ATTRMASK) == SECBUFFER_DATA) {
wchar_t url[512];
BuildHttpsUrl(phCtx, url, ARRAYSIZE(url));
if (!QueryVerdict(url, pMsg->pBuffers[i].pvBuffer,
pMsg->pBuffers[i].cbBuffer))
return SEC_E_INTERNAL_ERROR; /* BLOCK — aborts the TLS record */
}
}
return Real_EncryptMessage(phCtx, fQOP, pMsg, SeqNo);
}
5.3 Dynamic OpenSSL / BoringSSL
Runtimes such as Python and certain Electron frameworks often utilize OpenSSL. The hook DLL scans the process for these specific modules and dynamically attaches to the SSL_write and SSL_read functions to intercept encrypted traffic.
static const wchar_t *candidates[] = {
L"libssl-3-x64.dll", L"libssl-1_1-x64.dll",
L"libssl.dll", L"boringssl.dll",
};
HMODULE hSsl = NULL;
for (int i = 0; i < ARRAYSIZE(candidates) && !hSsl; ++i)
hSsl = GetModuleHandleW(candidates[i]);
if (hSsl) {
Real_SSL_write = (PFN_SSL_write)(PVOID)GetProcAddress(hSsl, "SSL_write");
DetourTransactionBegin();
DetourAttach(&(PVOID &)Real_SSL_write, Hook_SSL_write);
DetourTransactionCommit();
}
5.4 Static BoringSSL — Byte-Pattern Scanning
When libraries are statically linked, as is common with Node.js bundles, standard module enumeration is insufficient. In these cases, byte-pattern scanning is employed to locate the internal TLS functions within the main executable's code section based on their unique binary signatures.
Walk the PE section table to find the .text section (CNT_CODE + MEM_EXECUTE).
Scan the .text bytes for a unique inner-function byte sequence (~19 bytes) that is unlikely to appear elsewhere.
Walk back a fixed offset from the match to the function prologue and verify a second signature.
If both match, call DetourAttach on the raw address. If either fails, log and continue — no crash.
static const BYTE inner_sig[] = {
0x48, 0x31, 0xE0, /* xor rax, rsp */
0x48, 0x89, 0x44, 0x24, 0x50, /* mov [rsp+50h], rax */
0x48, 0x8B, 0x41, 0x30, /* mov rax, [rcx+30h] */
0x48, 0x8B, 0x88, 0xA0, 0x00, 0x00, 0x00, /* mov rcx, [rax+0A0h] */
};
static const BYTE prolog_sig[] = {
0x41, 0x57, 0x41, 0x56, 0x41, 0x55, 0x41, 0x54, /* push r15–r12 */
0x56, 0x57, 0x55, 0x53, /* push rsi/rdi/rbp/rbx */
0x48, 0x83, 0xEC, 0x58, /* sub rsp, 58h */
0x48, 0x89, 0xCE, /* mov rsi, rcx */
};
static const int kPrologToInner = 26;
PBYTE inner = ScanForPattern(textBase, textSize, inner_sig, sizeof(inner_sig));
PBYTE hit = inner - kPrologToInner;
if (memcmp(hit, prolog_sig, sizeof(prolog_sig)) == 0) {
Real_do_ssl3_write = (PFN_do_ssl3_write)(PVOID)hit;
DetourTransactionBegin();
DetourAttach(&(PVOID &)Real_do_ssl3_write, Hook_do_ssl3_write);
DetourTransactionCommit(); /* hook installed at raw RVA */
}
Note: Byte signatures are version-specific. When the target binary is updated, the scan silently finds no match — the hook is not installed, but the application continues to run normally (fail-open). Production tools automate signature extraction from new builds via a reverse engineering pipeline.
6. The Verdict Pipe — Decoupling Policy from the Hook DLL
To decouple logic from the injection mechanism, the hook DLL communicates with a separate policy agent via a named pipe. This architectural choice allows the policy to be updated dynamically without re-injecting the DLL and ensures that complex decision-making logic is isolated from the target process.
Figure 5 — The verdict pipe wire protocol. The hook DLL is the client; the agent is the server. Fail-open: if the pipe is unavailable the hook allows the request.
/* QueryVerdict — called from every hook before allowing or blocking */
static BOOL QueryVerdict(const wchar_t *url, LPCVOID body, DWORD bodyLen)
{
/* Wait up to 200 ms for the pipe server to be ready */
WaitNamedPipeW(L"\\.\pipe\HookVerdict", 200);
HANDLE h = CreateFileW(L"\\.\pipe\HookVerdict",
GENERIC_READ | GENERIC_WRITE, 0, NULL,
OPEN_EXISTING, 0, NULL);
if (h == INVALID_HANDLE_VALUE) return TRUE; /* fail-open */
/* Send: URL (UTF-8, null-terminated) | DWORD bodyLen | body bytes */
char urlUtf8[2048];
WideCharToMultiByte(CP_UTF8, 0, url, -1, urlUtf8, sizeof(urlUtf8)-1, 0, 0);
DWORD written;
WriteFile(h, urlUtf8, (DWORD)strlen(urlUtf8) + 1, &written, NULL);
WriteFile(h, &bodyLen, sizeof(DWORD), &written, NULL);
if (bodyLen) WriteFile(h, body, bodyLen, &written, NULL);
/* Receive: 1 byte — 0x00 ALLOW, 0x01 BLOCK */
BYTE verdict = 0;
DWORD read;
ReadFile(h, &verdict, 1, &read, NULL);
CloseHandle(h);
return (verdict == 0x00);
}
7. The Complete System Picture
The integrated system utilizes a kernel minifilter driver for process monitoring and injection, a hook DLL for traffic interception, and a user-mode agent for policy enforcement. This multi-layered approach provides a comprehensive monitoring framework for Windows endpoints.
Figure 6 — End-to-end architecture. The kernel driver injects the hook DLL via APC and receives file-access events through the filter port. The hook DLL queries the agent's verdict pipe for each outbound request. The agent also sends policy to the driver.
The operational lifecycle of the system is as follows:
Driver loads → registers ProcessNotify, ThreadNotify, LoadImageNotify callbacks.
Agent connects to the filter communication port, sends policy (monitored extensions, hook DLL path).
First kernel32.dll load → LoadImageNotify captures LoadLibraryW's address.
User launches a monitored application → ProcessNotify fires → PID queued for injection.
Windows creates the first thread in that process → ThreadNotify fires → VgDoInject allocates user buffer, writes DLL path, queues APC.
Thread transitions to user mode → APC fires → LoadLibraryW(dllPath) → DLL loads.
DLL_PROCESS_ATTACH → DetourTransactionCommit installs all hooks.
Application calls WinHttpSendRequest → JMP to hook → QueryVerdict → ALLOW/BLOCK.
8. Common Pitfalls
Incompatibility with Arbitrary Code Guard (ACG)
ACG is a security feature that prohibits the modification of executable memory. Consequently, Detours cannot apply patches to processes where ACG is active, such as Chrome renderer processes. Developers must account for these protections when designing injection targets.
The injected user-mode buffer leaks
Memory allocated for the DLL path in the target process must be properly managed. In production environments, this memory should be released following a successful load notification to prevent memory leaks within the target application.
KeInsertQueueApc returns FALSE — the APC is silently dropped
If a target thread is terminating during the injection attempt, the APC may be dropped. While often a minor issue for long-running applications, robust systems should implement retry logic to ensure the hook is successfully delivered to subsequent threads.
FltGetFileNameInformation must not be called on paging I/O
Minifilter callbacks must avoid complex operations during paging I/O to prevent deadlocks. Drivers should always verify the I/O type and skip processing on paging paths to ensure system stability.
Extension matching: no leading dot
The Windows Filter Manager provides file extensions without the leading dot. Storing and comparing extensions with the dot included will result in failed matches. Precise string handling is critical for the effectiveness of file monitoring filters.
Chunked upload bodies cannot be blocked at the WriteFile call
For protocols using chunked transfers, blocking decisions must occur at the initial request stage. Attempting to block during mid-stream writes can lead to protocol corruption. For audit purposes, these calls should be logged, but enforcement should remain at the session initiation point.
9. Building and Testing the Hook DLL
9.1 Build Requirements
Enterprise WDK (EWDK) ISO — a self-contained cl.exe, MSBuild, and Windows SDK that does not require a Visual Studio installation.
Microsoft Detours — clone from GitHub, build with nmake to produce detours.lib and the detours.h header.
A VM (Hyper-V or VMware) with kernel debugging enabled (KDNET or serial) for driver testing.
Sysinternals DebugView running inside the VM to capture OutputDebugString output from the injected DLL.
9.2 Building the Hook DLL
The hook DLL should be compiled with professional-grade tools like the Enterprise WDK. It is essential to configure correct include and library paths to ensure all user-mode dependencies and the Detours library are correctly linked.
cl.exe /LD /O2 /MD /W3 ^
hook_dll.cpp ^
/I %MSVC%\include ^
/I %KIT%\Include\10.0.26100.0\um ^
/I %KIT%\Include\10.0.26100.0\ucrt ^
/I %KIT%\Include\10.0.26100.0\shared ^
/I vendor\detours\include ^
/Fe_out\hook_dll.dll ^
/link ^
/LIBPATH:%MSVC%\lib\x64 ^
/LIBPATH:%KIT%\Lib\10.0.26100.0\um\x64 ^
/LIBPATH:%KIT%\Lib\10.0.26100.0\ucrt\x64 ^
vendor\detours\lib.X64\detours.lib ^
winhttp.lib wininet.lib ws2_32.lib secur32.lib
9.3 Verifying Injection and Hooks
Verification is performed by monitoring the output of the injected DLL. Tools like DebugView allow for real-time observation of injection status and hook activity, confirming that the interception layer is functioning as intended across all targeted stacks.
[hook-dll] DllMain: DLL_PROCESS_ATTACH
[hook-dll] loaded PID=1234 WinHttp+WinInet+Winsock+SChannel Detours=0 (OK)
[hook-dll] connect s=128 dst=13.107.4.50:443
[hook-dll] SChannel TLS target=api.openai.com ctx=abc123:456def
[hook-dll] SChannel ALLOW encrypt -> api.openai.com (312 bytes)
[hook-dll] WinHttp BLOCK https://api.openai.com/v1/upload
9.4 Kernel Debugger Breakpoints
Using a kernel debugger, developers can set specific breakpoints to trace the lifecycle of the injection and hooking process. This facilitates the identification of failures in APC queuing, DLL loading, or function patching.
bp MyDriver!DllInjectViaApc ; fires when APC is about to be queued
bp MyDriver!ApcKernelRoutine ; fires just before LoadLibraryW executes
bp MyDriver!QueueInjection ; fires when a monitored process is detected
bp MyDriver!LoadImageNotify ; fires on every DLL load (watch for kernel32)
10. Recommended Reading
Windows Internals, 7th Edition — Russinovich, Solomon, Ionescu. The canonical reference for process creation, APC internals, the loader (LDR), PE format, and kernel memory management. Read Part 1 chapters on processes and the loader before working with injection.
Microsoft Detours (github.com/microsoft/Detours) — The README covers the full API (DetourAttach, DetourTransactionBegin/Commit, DetourDetach), trampoline layout, known limitations (ACG, x86 vs. x64 jump encoding differences), and build instructions.
WDK: PsSetCreateProcessNotifyRoutineEx — MSDN documentation for each kernel callback used in injection (PsSetCreateProcessNotifyRoutineEx, PsSetCreateThreadNotifyRoutine, PsSetLoadImageNotifyRoutine). Each page documents firing context, allowed operations, and IRQL constraints.
WDK: KeInitializeApc / KeInsertQueueApc — These APIs are not documented in the public WDK headers but are stable across Windows 10/11 x64. The declarations used here match the binary ABI confirmed by kernel reverse engineering.
WDK: Filter Manager Minifilter Design Guide — Covers FLT_REGISTRATION, pre/post callback return values, FLT_FILE_NAME_INFORMATION, context management, and the fail-open pattern. Essential background for the file-monitoring side of a DLP driver.
Sysinternals: DebugView, Process Explorer — DebugView captures DbgPrint output from kernel drivers and OutputDebugString from user-mode DLLs. Process Explorer's DLL view is the fastest way to confirm a DLL was successfully injected.
Conclusion
DLL injection and API hooking remain core pillars of Windows security architecture. By utilizing kernel APCs for delivery and Microsoft Detours for interception, developers can implement high-performance monitoring solutions that operate beneath traditional security boundaries. These methodologies provide the necessary visibility for data protection and system analysis.
In an increasingly complex software ecosystem, supporting multiple TLS stacks and static runtimes is mandatory for complete coverage. A successful implementation requires a sophisticated blend of kernel-mode coordination, binary pattern recognition, and robust user-mode policy management to ensure comprehensive endpoint security.
No comments:
Post a Comment