Byte Sized Brainstorm: Practicing Memory Leak Analysis

Memory Leak Analysis

Following are the tools and the sample application that will be used for the following simple memory leak analysis.

Application:-

http://windbg.info/apps/46-crashme.html

CrashMe is a simple application that implements several common debug situations and scenarios. We can download the source code, build and run it. We can initiate the debugging scenario by clicking on the appropriate button.

Tools:-

1) gflags & umdh.

The user-mode dump heap (UMDH) utility works with the operating system to analyze Windows heap allocations for a specific process. UMDH locates which routine in a specific process is leaking memory.

umdh and gflags are the tools that are part of Windows debugger package. Windows debugger is part of Windows SDK package.

The most important data in the UMDH logs are the stack traces of the heap allocations.

Before using umdh , we must configure system properties to collect stack traces, enable page heap.

Following are the steps to analyze memory leaks using umdh tool.

Step 1:- Set the Symbol path for the executable

Copy the symbol file for the executable :- CrashMe , to a folder C:\Symbols and set the environment variable _NT_SYMBOL_PATH. Set both the windows server path and the folder path where the executables symbols are copied.

_NT_SYMBOL_PATH :-

symsrv*symsrv.dll*C:\localsymbols*http://msdl.microsoft.com/download/symbols;C:\Symbols;

Step 2:- Enable stack trace database and page heaps

Run gflags to set the global flags and to set flags for the application to enable page heaps and stack trace.

We also can set the flags for the application at command line as follows.

              gflags /i CrashMe.exe +ust

              gflags /i CrashMe.exe +uht

Step 3:- Start the application

Step 4:- Collect the initial umdh log

The process name argument to umdh is the PID as seen in the process explorer.

Step 5:- Perform the scenario (Suspected scenario that is leaking memory).

For this application , CrashMe.exe the scenario is obvious. Memory leaks when the Memory leak button is clicked.

But in realistic scenarios, leaks might be in multiple threads , in different sub modules(dlls) and due to different memory allocators. We might need to collect multiple umdh logs at different instances.

But our context is very simple here, collect the logs again after performing the scenario(clicking the button).

Step 7:- Collect the umdh log after the scenario.

Step 6:- Compare UMDH logs ("Before log" with "After Log").

UMDH can compare two different log files and display the change in their respective allocation sizes. You can use the greater-than symbol (>) to redirect the results into a third text file.

Step 7:- Analyze the umdh diff log file.

From the log file we could figure out the following

The increase in bytes between before and after scenario is :- 269484032

Increase in allocations between before and after scenario is :- 257

And the stack trace which resulted in this memory allocation.

Step 8:- Code Review with the help of stack text obtained in previous step.

From the above logs, its clear that the function to be reviewed is :- CCrashMeDlg::OnBnClicked_MemoryLeak . And the line number is 455.

CrashMe!CCrashMeDlg::OnBnClicked_MemoryLeak+3E (c:\windbg\crashme\crashme\crashmedlg.cpp, 455)

2) DebugDiag

All the analysis tools have broadly two steps 1) Collect the information/memory dumps 2) Analyze the Dumps.
DebugDiag has two tools 1) DebugDiag Collection tool 2) DebugDiag Analysis tool.

Since the current scenario is simple, we need not require Debug Diag to collect the memory dumps, we also can use sysinternal tool like procdump (provided it collects all the counters required for our analysis scenario).

Debug Diagnostic Tool allows us to specify the rules to collect the dumps. Or it also allows the user to select the process and collect dumps whenever required.

Debug Diagnostic Tool allows to specify the triggers for crash dump generation. We can specify the memory thresholds or processing thresholds at which it should collect the dumps.

If we are working with a service that never terminates or applications that run for long periods, we can configure the rules based on the type of analysis we wanted to perform.

For memory leak analysis, this tool will inject a DLL into the specified process and monitor memory allocations over time. A dump is then generated, and the dump is analyzed to determine what allocations are not being freed and most likely causing the memory leak. Allocations generally fall into 3 groups: caching, short term allocations that will be freed later, and memory leaks. All three allocation methods have very distinct allocation patterns when measured over time. The leak tracking feature calculates a leak probability using a formula that is based on these allocation patterns as measured over a specific time period.

Note:- We need to disable the pageheap flag that was enabled in previous scenario for DebugDiag to successfully collect information.

Difference between Debug Diagnostic Tool and UMDH tool

UMDH tool only focuses on Windows heap. But Debug Diag can also hook into other allocators like COM allocators and Virtual Memory allocators.

Step 1:- Set the Symbol path environment vairable _NT_SYMBOL_PATH

Step 2:- Launch the application.

Step 3:- Launch DebugDiag Collection tool and configure the rule to collect dumps

Selecting the rule type allows the Debug Diag to collect required counters for the type of analysis we wanted to do.

Since our scenario is simple, we know when the leak happens and can explicitly collect dumps after the scenario is performed. But for realistic scenarios we need to specify the configuration :- how and when the tool should collect the memory dumps. A single dump wouldn't help us analyze the memory leaks. Multiple dumps needs to be collected at regular intervals or when a particular threshold is reached or any other triggering condition specified in the rule, so that they can be compared and most probable unclaimed memory and its allocator can be identified.

After configuring the collection rule, need to select the application for which we are collecting the memory dumps.

Then configure the location where the dumps needs to be stored.

Save the rule to collect the dumps.

For our scenario, we need not configure completely the dump collection rule(leave the defaults) since we wanted to collect the dumps explicitly before the scenario and after the scenario.

Now the tool starts monitoring the process.

Step 4:- Take the dump before the scenario is executed

Click on the processes view, select the process and take the full dump . This would be the information before the suspected scenario is executed.

Step 5:- Execute the scenario

Step 6:- Take the memory dump after the scenario is executed.

Step 7:- Launch the DebugDiag Analysis tool

Using the DebugDiag Analysis tool, we can choose the analyzer , set the symbol paths , select the dumps that were collected previously and trigger analysis.

Currently we use the tool to analyze the memory dumps for possible memory leaks. After selecting the analyzer, click on the settings icon and set the symbol path for the application binaries and windows binaries.

Step 7:- Start the Analysis

Click on Add Data Files, select the dumps to be analyzed and select the button Start Analysis

Once the analysis is done, it generates a detailed report related to memory allocations.

Step 8:-Analyze the report

Step 9:-Review the pointed code and fix the leak

3) !heap extension in WinDbg

If tools doesn't give much information about the leak, then using the WinDbg debugger is the last resort. We should manually walk through the heap, figure out the stack traces that allocated the suspected heap. Then review and correct the function pointed out in the stack text.

Debugger can be used for both 1) Live Debugging 2) Post Mortem debugging.

1) Live Debugging

    Run the application with debugger.
   Attaching a process with debugger can be done in two ways
        a) Invasive attach
               Break-in thread is created. There can be only one invasive debugger attached to a process at any time.
       b) Noninvasive attach
               We don't attach to the process as a debugger. We can examine memory, but cannot step through the application. We can attach several noninvasive debuggers to a process.

2) Post mortem debugging.
    Take the dump of the application. Open the dump in the debugger(WinDbg) , load the symbols and analyse the dump independent to the application.

In this example will be doing the post mortem debugging.

Step 1 :- Set the global flags to enable pageheap and stack trace database

Step 2 :- Launch the application

Step 2 :- Execute the scenario

Step 3 :-Take a full dump of the process using Process Explorer

Step 4:- Launch WinDbg and set the symbols location for application and windows binaries

Step 5:- Open the memory dump in WinDbg

Step 6:- Ensure symbols are loaded for the module

Step 4:- List down handle specific allocation statistics for every Alloc size.

Command :- !heap -stat -h 0
-stat :- Displays usage statistics for the specified heap.
-h <handle> :- Displays usage statistics for only the heap at Handle. If Handle is 0 or omitted, then usage statistics for all heaps are displayed.

Step 5:- Choose the AllocSize with maximum TotalMem and display summary information for that heap.

Step 6:- Get the call-stack with source information for the first user address

Since page heap is enabled -p option displays various forms of page heap information.

Byte Sized Brainstorm

Practicing Memory Leak Analysis