WinDbg is a beast, this guide will barely touch the surface but hopefully it will be enough to get you started.
- Write a memory dump
- Download and install WinDbg
- Analysing the dump on another machine
- Open a memory dump
- Recommended extensions
- Commands
- Troubleshooting
The first step is to write a memory dump. I recommend using ProcDump from Sysinternals. You can set up rules to write a dump when certain conditions are met (i.e., exception thrown, CPU usage, memory usage...). The common use case is to write a dump using a PID (process Id):
procdump64.exe -r -a -ma <process-id>-ma: write a 'Full' dump file-r: dump using a clone-a: avoid outage
🚨 ProcDump comes as two flavours: x86 and x64. You'll need to use the same bitness than the process you're capturing. If you use the incorrect bitness, the dump will be mostly unusable.
You can use a process name instead of a PID:
procdump64.exe -ma <process-name>🚨 when running ProcDump for the first time you'll need to accept the Sysinternals license agreement:
procdump64.exe -accepteula -ma <process-id>🚨 if you decide to use something else don't forget you need a full dump (instead of a mini dump).
You can get WinDbg by getting the WinDbg Preview from the store (Windows 10 Anniversary Update and higher only) or installing the Windows SDK.
If you're running Windows 10 Anniversary Update and higher you can install WinDbg Preview from the store. I wrote a guide about WinDbg Preview.
Download the Windows SDK.
In the installation wizard, select Debugging Tools for Windows and clear all the other components.
WinDbg comes in two flavours: x86 and x64. Both will be installed, when loading a dump make sure you select the one matching the bitness of your process.
You'll need to get the following DLLs from the machine where the dump was written:
mscordacwks.dllSOS.dll
They're located in the matching version of the .NET framework: C:\Windows\Microsoft.NET\Framework for x86 and C:\Windows\Microsoft.NET\Framework64 for x64. The versions listed there are the CLR versions, you can determine the CLR version based on the .NET Framework version.
Once you've downloaded the two DLLs you need to load them in WinDbg:
For SOS: .load C:\path-to-dll\SOS.dll
- No output is expected
For mscordacwks: .cordll -lp C:\{directory-in-which-mscordacwks-is-located}
- Do not include
mscordacwks.dllin the path (i.e. if the location isC:\dlls\mscordacwks.dllthe command should be.cordll -lp C:\dlls) - The output should be
CLR DLL status: Loaded DLL {full-path-name}
File -> Open Crash Dump...
When opening the dump WinDbg will display the following information:
Ensure you're working with a full dump and that sympath is as expected (see Configure the symbols).
sympath: the location(s) where WinDbg will look for symbols.
Having symbols will make the debugging experience much nicer. When building, either publish your PDBs to a symbols server or store them as artifacts. Then when needed copy them to C:\symbols\local (or whatever you set your sympath to).
You can view the value of sympath by issuing the following command:
.sympath
You can set sympath via a setting, during the debugging session or via an environment variable. If you want to know more, you can read Symbol path for Windows debuggers.
- Open the
Filemenu -> SelectSymbol File Path ... - Enter the below in the textbox, select
Reloadand clickOk
C:\symbols\local;srv*C:\symbols\microsoft*https://msdl.microsoft.com/download/symbols
File -> Save Workspace will persist your sympath!
You can load the symbols during the session:
.sympath C:\symbols\local;srv*C:\symbols\microsoft*https://msdl.microsoft.com/download/symbols
.reload
- Variable name:
_NT_SYMBOL_PATH - Variable value:
C:\symbols\local;srv*C:\symbols\microsoft*https://msdl.microsoft.com/download/symbols
- Open the
Filemenu -> SelectSource File Path ... - Enter the below in the textbox, select
Reloadand clickOk
C:\symbols\source
File -> Save Workspace will persist your srcpath!
WinDbg is a bit dry but luckily extensions provide nifty commands.
Load the CLR debugging extensions. Informally known as Son of Strike.
.loadby sos clr
.loadby sos coreclr
My favourite and the masterpiece of Steve Johnson.
Download links:
Copy the x64 / x86 DLLs to their respective folders:
C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\winextC:\Program Files (x86)\Windows Kits\10\Debuggers\x86\winext
If you can't find this path, look for windbg.exe (where.exe /R C:\ windbg.exe).
.load sosex
Download MEX.
.load mex
Download NetExt.
.load netext
Download links:
Official site.
.load cmkd
Download links:
Official site.
.load tracer
- regular commands (e.g.:
k) debug processes - dot commands (e.g.:
.sympath) control the debugger - extension commands (e.g.:
!handle) come fromWinDbgextensions
🌟 Get latest exception:
!analyze -v
Display the Process Environment Block (PEB) - includes environment variables and command line:
!peb
Go (F5) - continue execution:
g
Objects per generation:
!FinalizeQueue
Convert hex to decimal
0:000> ?12c0
Evaluate expression: 4800 = 00000000`000012c0
List of drivers:
lm t.lon
- Exiting current operation:
Ctrl + Break key - Focus textbox:
ALT + 1
Will add hyperlinks you can click instead of having to type the commands yourself (might be on by default in latest versions):
.prefer_dml 1
🌟 List SOS commands
!sos.help
🌟 List SOSEX commands
!sosex.help
🐢 Build an index heap and make searching in the heap faster:
!sosex.bhi
List the content of the finalization queue:
!sosex.finq
Freachable queue:
!sosex.frq
Detect deadlocks:
!sosex.dlk
See what each managed thread is waiting for (if anything):
!sosex.mwaits
Displays the fields of an object or type, optionally recursively:
!sosex.mdt address
Sets a managed breakpoint
!sosex.mbp File.cs 13
List MEX commands
!mex.help
🌟 ASP.NET: short history of requests which have run and currently running requests
!mex.aspxpagesext
Get all the objects in the managed heaps. Bigger objects are at the bottom:
!dumpheap -stat
Once you identified a problematic type:
!dumpheap -type Full.Namespace.Type
What's keeping an object alive:
!sosex.mroot <address>
See managed heaps:
!eeheap -gc
Summary of memory by types:
!address -summary
Find all the exceptions on the heap:
!dumpheap -type Exception -stat
Find the addresses of this exception:
!dumpheap -type System.Threading.ThreadAbortException
Print exception using the address:
!pe 0x01c601ac
See all exceptions of dump:
!mex.dumpallexceptions
# Or if you're lazy:
!mex.dae
# Or if you want to see all types having "Exception" in their name:
!dumpheap -type Exception -stat
🌟 List managed threads with what they're currently doing
!mex.mthreads
Alternatively you can use !sos.threads
The first column is the thread ID which then can be used for all the commands below (i.e. in this case we're interested in thread 142).
See what a managed thread is doing:
~142e !sos.CLRStack
See what an unmanaged thread is doing:
~142k
Switch to thread:
~142s
🌟 🐢 Group threads by unique stacks
!mex.us
This command is great to identify all the threads doing the same thing (and potentially being blocked at the same point).
Disassemble:
!U \d <address>
Displays a disassembly around the current instruction with interleaved source, IL and ASM code:
!sosex.mu <address>
0:000> !syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
52 20ee3118 229 2 20fc6ba0 9628 42 0a13ee8c System.Object
# Abbreviated
The third column (MonitorHeld) indicates how many threads are trying to acquire the same lock. In this case it is (229 - 1) / 2 = 114. You can read more about it in this SO answer.
WinDbg errors can be a bit cryptic. In this section I say no more!
You might get an error when trying to load SOS:
0:000> .loadby sos clr
The call to LoadLibrary(D:\Windows\Microsoft.NET\Framework\v4.0.30319\sos) failed, Win32 error 0n126
"The specified module could not be found."
Please check your debugger configuration and/or network access.
This dump has been written on an Azure Web App and the path is pointing to a hard-drive somewhere in the cloud.
- Download the
SOS.dllfrom the folder included in the error (D:\Windows\Microsoft.NET\Framework\v4.0.30319\in this case) - Load the
SOS.dllusing this command:.load C:\path-to-dll\SOS.dll
You can see which mscordacwks.dll is loaded:
0:000> .cordll
CLR DLL status: Loaded DLL c:\symbols\mscordacwks_x86_x86_4.7.2563.00.dll\5A334E146eb000\mscordacwks_x86_x86_4.7.2563.00.dll
0:000> !runaway2
No export runaway2 found
You need to load the extension containing this command. A quick Google will identify the name of the extension.
This is what happens when you open a 32-bit memory dump with WinDbg (X64):
0:000> .load C:\dumps\SOS.dll
The call to LoadLibrary(C:\dumps\SOS.dll) failed, Win32 error 0n193
"%1 is not a valid Win32 application."
Please check your debugger configuration and/or network access.
Use WinDbg (X86) instead.
System.ExecutionEngineException, System.StackOverflowException and System.OutOfMemoryException are created as soon as the process starts. This means that you will always see them on the heap even if they haven't been thrown.
See Why do I see ExecutionEngineException, StackOverflowException and OutOfMemoryException on the heap?.
If some of your symbols are not loading as expected you can turn on debugging messages when WinDbg attempts to load symbols:
!sym noisy
Once you're done, don't forget to reset it to its initial state:
!sym quiet