Getting Started with kernel dumps

[sachin]

Hello, I am new to kernel dump analysis so pardon me if you find these questions trivial. I have been working on user dumps for some time and there are few common steps that i would take as soon as i get my hands on any user dump. These steps will give some visibility of what kind of problem is at hand.

1. First find out what kind of dump is this, crash dump / hang dump.
2. Find out how the dump was collected, adplus / cbd / debugdiag / Dr Watson.
3. Full dump / mini dump || managed / native process?
4. Here are few commands which will help me get more information from the dump -
= Load sos if managed dump.
Native dump analysis common commands
===================================
- ~*kb 100 [will show all the stack of all the threads in the dump]
- !runaway [will show the threads in order of CPU time consumed], which will later on help me take a detailed look at the thread which has consumed highest quantum.
- ~[thread]s - this will switch the context to any thread that i want to look in detail.
- !heap to find out which component is holding on to memory
- lm - to find out the loaded modules and see if there are any modules which are old enough to cause problem or see their versions to see if we have problematic versions.
- !locks to display all the locks.
- !vm to see memory usage

Managed Dump analysis common commands
=====================================
!dumpheap -stat - to check memory usage of all the managed objects.
~*e!clrstack - to see the managed call stack of all the threads
!syncblk - to find any synchronization waits.
!do to dump any managed object and then drill into.
!dae to display all excetions which have not yet been cleaned up
!dso to see all the objects on a managed thread
!GCroot <object addr> this will show rooted objects on multiple threads...

Now this is no where a complete list, but these are enought to give you a sense of what kind of problem you are dealing with. I am currently looking for similar information, break down approach to analyzing kernel dumps.

I want to be able to switch process context, but i most of the times get an error :
0: kd> .process 905249e8
Process 905249e8 has invalid page directories
Dont know what i can do with these.

I want to be able to find out all the threads in the dump... what is the command. ~*k does not work.

What is the command to find the crashing thread / component.

how do enable a FULL kernel dump, i see no option for a FULL dump in "Startup and recovery" >> settings

Basically looking for commands which will help me get started with the analysis. Thanks in advance guys,.

[VDO]

Code: Select all

0: kd> .process 905249e8
Process 905249e8 has invalid page directories

This error usually happens when you have kernel dump and not a complete (full physical memory) dump. This is because page directories for user space counterpart (which are itself pages) are not included in the kernel memory dump and therefore WinDbg thinks that they are invalid (not accessible)

Also if you want to interrogate specific process in complete memory dump the following command is recommended which also reloads symbols for user space:

Code: Select all

.process [b]/r /p[/b] 905249e8

If you wand to see all stack use after switching and reloading symbols with the command above use:

Code: Select all

!process 905249e8

It will give the list of threads as well:

Code: Select all

        THREAD [b]862131a8[/b]  Cid 2b94.3ee4  Teb: 7ffda000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable
            841b3ab8  NotificationEvent
            86213220  NotificationTimer
        Not impersonating
        DeviceMap                 d6cac140
        Owning Process            8596b658       Image:         application.exe
        Wait Start TickCount      30851509       Ticks: 6644 (0:00:01:43.812)
        Context Switch Count      126             
        UserTime                  00:00:00.109
        KernelTime                00:00:00.015
        Win32 Start Address mscorwks!Thread::intermediateThreadProc (0x79f959e8)
        Start Address KERNEL32!BaseThreadStartThunk (0x7c82b5f3)
        Stack Init 9cfee000 Current 9cfedc60 Base 9cfee000 Limit 9cfeb000 Call 0
        Priority 11 BasePriority 10 PriorityDecrement 0
        ChildEBP RetAddr 
        9cfedc78 80832f7a nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
        9cfedca4 8082925c nt!KiSwapThread+0x284 (FPO: [Non-Fpo])
        9cfedcec 80937e6a nt!KeWaitForSingleObject+0x346 (FPO: [Non-Fpo])
        9cfedd50 80888c7c nt!NtWaitForSingleObject+0x9a (FPO: [Non-Fpo])
        9cfedd50 7c97ed54 nt!KiFastCallEntry+0xfc (FPO: [0,0] TrapFrame @ 9cfedd64)
        00c2fc94 7c972124 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
        00c2fc98 7c82bad8 ntdll!NtWaitForSingleObject+0xc (FPO: [3,0,0])
        00c2fd08 79e789c6 KERNEL32!WaitForSingleObjectEx+0xac (FPO: [Non-Fpo])
        00c2fd4c 79e7898f mscorwks!PEImage::LoadImage+0x1af (FPO: [Non-Fpo])
        00c2fd9c 79e78944 mscorwks!CLREvent::WaitEx+0x117 (FPO: [Non-Fpo])
        00c2fdb0 79f014ef mscorwks!CLREvent::Wait+0x17 (FPO: [Non-Fpo])
        00c2fe30 79f02143 mscorwks!ThreadpoolMgr::SafeWait+0x73 (FPO: [Non-Fpo])
        00c2fe94 79f95a2e mscorwks!ThreadpoolMgr::WorkerThreadStart+0x11c (FPO: [Non-Fpo])
        00c2ffb8 7c82608b mscorwks!Thread::intermediateThreadProc+0x49 (FPO: [Non-Fpo])
        00c2ffec 00000000 KERNEL32!BaseThreadStart+0x34 (FPO: [Non-Fpo])

If you want to interrogate specific thread more use this

Code: Select all

0: kd> .thread [b]862131a8[/b]
Implicit thread is now 862131a8

Now kv will work on this thread:

Code: Select all

0: kd> kv
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child             
9cfedc78 80832f7a 86213220 862131a8 86213250 nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
9cfedca4 8082925c 00000000 9cfedd14 00000000 nt!KiSwapThread+0x284 (FPO: [Non-Fpo])
9cfedcec 80937e6a 841b3ab8 00000006 8596b601 nt!KeWaitForSingleObject+0x346 (FPO: [Non-Fpo])
9cfedd50 80888c7c 00000174 00000000 9cfedd14 nt!NtWaitForSingleObject+0x9a (FPO: [Non-Fpo])
9cfedd50 7c97ed54 00000174 00000000 9cfedd14 nt!KiFastCallEntry+0xfc (FPO: [0,0] TrapFrame @ 9cfedd64)
00c2fc94 7c972124 7c82bad8 00000174 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
00c2fc98 7c82bad8 00000174 00000000 00c2fcdc ntdll!NtWaitForSingleObject+0xc (FPO: [3,0,0])
00c2fd08 79e789c6 00000174 00027cb8 00000000 KERNEL32!WaitForSingleObjectEx+0xac (FPO: [Non-Fpo])
00c2fd4c 79e7898f 00000174 00027cb8 00000000 mscorwks!PEImage::LoadImage+0x1af (FPO: [Non-Fpo])
00c2fd9c 79e78944 00027cb8 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117 (FPO: [Non-Fpo])
00c2fdb0 79f014ef 00027cb8 00000000 00000000 mscorwks!CLREvent::Wait+0x17 (FPO: [Non-Fpo])
00c2fe30 79f02143 0017be38 00027cb8 00000000 mscorwks!ThreadpoolMgr::SafeWait+0x73 (FPO: [Non-Fpo])
00c2fe94 79f95a2e 00000000 00000000 83666ba0 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x11c (FPO: [Non-Fpo])
00c2ffb8 7c82608b 0017e278 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49 (FPO: [Non-Fpo])
00c2ffec 00000000 79f959e8 0017e278 00000000 KERNEL32!BaseThreadStart+0x34 (FPO: [Non-Fpo])

It is also possible to use SOS in complete memory dumps:

http://www.dumpanalysis.org/blog/index. ... ory-dumps/

For some commands use this checklist (System hang and BSOD sections):

http://www.dumpanalysis.org/blog/index. ... checklist/