Post-mortem debugging of .NET applications using WinDbg
Debugging is a skill you usually learn under pressure, when things are going awry with an application or service just gone live. It is never a pleasure to encounter such bugs because, although they happen quite frequently in your production environment, they are particularly hard to reproduce in your test environment.
For managed applications, you can learn a new skill that will save you some face, called post-mortem debugging. WinDbg is a splendid tool that is often used to debug running processes, but can also be used to analyze process crash dumps.
Dump process memory
User mode process memory dumps can be obtained through several means
Windows Error Reporting (WER)
Configure WER to dump process memory when a user mode application crashes.
Right click on a process and select Create dump file. 32-bit Task Manager at %WINDIR%\SysWOW64\Taskmgr.exe should be used to dump memory of a 32-bit process.
Attach to a process and dump memory from the File menu, or the command line
.dump /ma filename.dmp
You need to have enough disk space because
dmp files can be rather big.
Once you have the crash dump file, you can open it with WinDbg, and examine it using several useful commands
Shows detailed information about the current exception.
Shows thread times. This can be really useful to find badly behaved threads that are consuming a lot of CPU.
Shows a list of threads currently in execution.
Displays all stacks of all the threads of the current process. You can view the stack trace of a single thread using the
Sets the thread with ID
nas the current thread.
You can run
.sympath C:\SymbolCache to download and load symbols from a specified path. To load symbols from Microsoft and Nuget.org, run
.reload /f to force debugger to reload symbols for all modules from the specified path. Run
.symopt +0x40 followed by
.reload /f, or
.reload /i, if you want to load a symbol file even if it does not match the module. This can be useful if you want to use a symbol file built from the same source code.
!chksym Module to check symbol information for a particular module, and
!chksym Module Symbol to check if a module matches a symbol file. Use an underscore character for each space character in the module name, or symbol path.
!sym noisy if you want to see detailed symbol loading information when these commands are run.
WinDbg is most useful for debugging managed application using the following extensions
This extension is distributed along with the .NET framework can be loaded using
.loadby sos mscorwks, or
.loadby sos clrfor .NET 4.
This has several useful additions to the core SOS extension. It can be loaded using a command such as
Has several useful commands, especially commands for debugging ASP.NET applications. It can be loaded using a command such as
The SOS extension has several useful commands, particularly
Prints the stack trace of the current thread. This only works for managed threads. If you have loaded symbols for your assemblies, you can see very useful information in the stack trace.
Dumps the objects from the heap. You can dump objects of a specific type by using the
!dumpheap -type System.Threading.Thread. This can be really useful to list all objects and their addresses. By knowing the number of objects of a particular type in use, you can debug resource exhaustion problems.
Dumps information about the object at the address specified. Value of primitive fields of the object are displayed and other references can be further examined using this command.
Prints information about an array
The SOSEX extension has the following commands that are particularly useful
Prints all objects that reference the object specified.
Searches for possible deadlocks.
The Psscor2 extension has one particularly useful command, among several others, that can come in handy when troubleshooting network related issues
Prints the IP address of the specified IPAddress instance.
This short post is meant to whet your appetite for post-mortem debugging and to point you in the right direction.