1) Without symbols
Without symbols, it is impossible to debug a hang issue. The reason is that a hang is caused because of some synchronization problems.
To implement synchronization, devolopers use synchronization objects such as Mutex, Critsec, Event, ...via APIs. The following MSDN article covers synchronization.
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndllpro/html/msdn_locktest.asp
So, again without symbols, the debugger is unable to resolve function calls and hence you will not be able to identify synchronization APIs. On the other hand,
some built-in debug extensions that help identify synchronization problems will not run properly. As an example, the " !locks " debug extension that shows information
about all the critical sections initialized in the process will not display any information if you don't have at least public symbols for "ntdll.dll"
2) With symbols including public symbols of "badExt.dll"
First, we need to add the symbol path(company wide symbols) as well as public symbols of our dll a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pub a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pub
Note: For more information on setting the symbol path for windbg, please refer to the debugger documentation for the .sympath command. This will describe
the "downstream store" feature that lets you download symbols to a symbol server of your own.
Basically, the first debugger command to run when analyzing a hang is the " ~*kv " .This command will display the stack of each thread in the process.
The command output will give you an idea about the threads running in the process. If you are familiar with the win32 synchronization objects, you could start
by looking for APIs that handle these objects such as "RtlEnterCriticalSection"
In this scenario, the following threads show calls to "RtlEnterCriticalSection" and "RtlWaitForCriticalSection"
17 Id: ac4.9c Suspend: 1 Teb: 7ffaa000 Unfrozen
# ChildEBP RetAddr Args to Child
00 012dff24 77f85c55 0000037c 00000000 00000000 NTDLL!NtWaitForSingleObject+0xb
01 012dff98 77f85bd1 01293000 01291a5f 01293018 NTDLL!RtlpWaitForCriticalSection+0x9e
02 012dffa0 01291a5f 01293018 00000001 0088f3b8 NTDLL!RtlEnterCriticalSection+0x46
03 012dffec 00000000 01291a39 00000000 00000000 badEXT!LockTables+0x7
18 Id: ac4.8c4 Suspend: 1 Teb: 7ffa8000 Unfrozen
# ChildEBP RetAddr Args to Child
00 0131ff24 77f85c55 00000378 00000000 00000000 NTDLL!NtWaitForSingleObject+0xb
01 0131ff98 77f85bd1 01293000 01291a95 01293000 NTDLL!RtlpWaitForCriticalSection+0x9e
02 0131ffa0 01291a95 01293000 00000001 0088f3b8 NTDLL!RtlEnterCriticalSection+0x46
03 0131fff0 01291a6f 00000000 00000000 00000000 badEXT!LockAll+0x7
The first parameter passed to "RtlEnterCriticalSection" is a pointer to the critical section object that the thread wants to
get a hold of. The following debugger command displays the critical section object address as well as the value of some data members of the critical section object.
0:000> !critsec 01293000
CritSec badEXT!cs1+0 at 01293000
LockCount 1
RecursionCount 1
OwningThread 9c
EntryCount 1
ContentionCount 1
*** Locked
0:000> !critsec 01293018
CritSec badEXT!cs2+0 at 01293018
LockCount 1
RecursionCount 1
OwningThread 8c4
EntryCount 1
ContentionCount 1
*** Locked
What these data members mean?
LockCount: Number of threads waiting on the Critical Section to be released
RecursionCount: Number of times the owning thread called "EnterCriticalSection"
OwningThread: ID of the thread owning the Critical Section
EntryCount: Number of times threads other than the owning thread called "EnterCriticalSection"
ContentionCount: Usually it is the same as EntryCount, but it might be different (it has to do with Lock Convoy, more reading http://sts/faster)
Another command very useful in analyzing hangs is the " !locks " which scans the whole process looking for critical section objects.
0:000> !locks
CritSec badEXT!cs1+0 at 01293000
LockCount 1
RecursionCount 1
OwningThread 9c
EntryCount 1
ContentionCount 1
*** Locked
CritSec badEXT!cs2+0 at 01293018
LockCount 1
RecursionCount 1
OwningThread 8c4
EntryCount 1
ContentionCount 1
*** Locked
What this shows is that there are 2 Critical Sections in this process. The first one is at (01293000 )and the second one is at (01293018).
Notice that thread ID "9c" owns Critical Section (01293000 ) and thread ID "8c4" owns critical section (01293018).
Note that thread ID is different from thread instance. Thread ID is the ID that the system assigns to a thread when it is spawned, the thread will keep this ID
as long as it is alive, while thread instance is just added by the debugger for presentation purposes. To get the mapping (Instance --> ID), you could issue
the following command
0:000> ~
. 0 Id: ac4.66c Suspend: 1 Teb: 7ffde000 Unfrozen
1 Id: ac4.ac8 Suspend: 1 Teb: 7ffdd000 Unfrozen
2 Id: ac4.9ec Suspend: 1 Teb: 7ffdc000 Unfrozen
3 Id: ac4.a78 Suspend: 1 Teb: 7ffda000 Unfrozen
4 Id: ac4.6ec Suspend: 1 Teb: 7ffd9000 Unfrozen
5 Id: ac4.840 Suspend: 1 Teb: 7ffd8000 Unfrozen
6 Id: ac4.75c Suspend: 1 Teb: 7ffdb000 Unfrozen
7 Id: ac4.400 Suspend: 1 Teb: 7ffd7000 Unfrozen
8 Id: ac4.630 Suspend: 1 Teb: 7ffd6000 Unfrozen
9 Id: ac4.8b4 Suspend: 1 Teb: 7ffd5000 Unfrozen
10 Id: ac4.8f8 Suspend: 1 Teb: 7ffd4000 Unfrozen
11 Id: ac4.6d0 Suspend: 1 Teb: 7ffaf000 Unfrozen
12 Id: ac4.5e0 Suspend: 1 Teb: 7ffae000 Unfrozen
13 Id: ac4.89c Suspend: 1 Teb: 7ffad000 Unfrozen
14 Id: ac4.f0 Suspend: 1 Teb: 7ffac000 Unfrozen
15 Id: ac4.5cc Suspend: 1 Teb: 7ffab000 Unfrozen
16 Id: ac4.758 Suspend: 1 Teb: 7ffa9000 Unfrozen
17 Id: ac4.9c Suspend: 1 Teb: 7ffaa000 Unfrozen
18 Id: ac4.8c4 Suspend: 1 Teb: 7ffa8000 Unfrozen
So thread instance 17 owns critical section (01293000 ). This thread is trying to get a hold of critical section (01293018). (because there is a call to NTDLL!RtlEnterCriticalSection ())
On the other hand, thread 18 owns critical section (01293018) and wants to get a hold of critical section (01293000 ). So what happens here is that thread 17 will not release (01293000)
until he gets a hold of (01293018). The same thing with thread 18 which will not release (01293018) until it gets a hold of (01293000). This is basically a deadlock.
It appears clear from the critical section addresses (badEXT!cs1) and (badEXT!cs2) that this deadlock is actually in the ISAPI extension "badext.dll". This is enough to tell the
customer that the culprit is is "badext.dll".
To verify that the symbols are lined up properly for our extension, issue this command.
0:006> lmv mbadext
start end module name 06470000 06475000
badext (pdb symbols) C :\IISDebugLabs\symbols\Pub
Loaded symbol image file: C:\IISDebuglabs\www\App\badext.dll Image path: C:\IISDebuglabs\www\App\badext.dll
Timestamp: Wed Jan 29 18:07:14 2003 (3E385EA2)
Checksum: 0000EBCD
File version: 5.0.0.0
Product version: 5.0.0.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
CompanyName: Microsoft Corporation
ProductName: Internet Information Services
InternalName: badEXT.dll
OriginalFilename: badEXT.dll
ProductVersion: 5.00.0984
FileVersion: 5.00.0984
FileDescription: badEXT Test
LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999
3) with Private Symbols "badExt.dll"
First, we need to add the symbol path(company wide symbols) as well as private symbols of our dll a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pri a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pri
Thread instances 17 and 18 look like respectively:
0:000> ~17kvn50
# ChildEBP RetAddr Args to Child
00 012dff24 77f85c55 0000037c 00000000 00000000 NTDLL!ZwWaitForSingleObject+0xb (FPO: [3,0,0]) [i386\usrstubs.asm @ 2004]
01 012dff98 77f85bd1 01293000 01291a5f 01293018 NTDLL!RtlpWaitForCriticalSection+0x9e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ntos\dll\resource.c @ 1612]
02 012dffa0 01291a5f 01293018 00000001 0088f3b8 NTDLL!RtlEnterCriticalSection+0x46 (FPO: [1,0,0]) [..\i386\critsect.asm @ 157]
03 012dffec 00000000 01291a39 00000000 00000000 badEXT!LockTables+0x7 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\extDBG\badext.cpp @ 506]
0:000> ~18kvn50
# ChildEBP RetAddr Args to Child 00 0131ff24 77f85c55 00000378 00000000 00000000 NTDLL!ZwWaitForSingleObject+0xb (FPO: [3,0,0]) [i386\usrstubs.asm @ 2004]
01 0131ff98 77f85bd1 01293000 01291a95 01293000 NTDLL!RtlpWaitForCriticalSection+0x9e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ntos\dll\resource.c @ 1612]
02 0131ffa0 01291a95 01293000 00000001 0088f3b8 NTDLL!RtlEnterCriticalSection+0x46 (FPO: [1,0,0]) [..\i386\critsect.asm @ 157]
03 0131fff0 01291a6f 00000000 00000000 00000000 badEXT!LockAll+0x7 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\extDBG\badext.cpp @ 517]
With Private symbols lined up as well as the source code of "badext.dll", Notice the lines of code of "badext.cpp" executing in each thread.
Thread 17
Badext.cpp line 506 looks like:
void LockTables( )
{
EnterCriticalSection( &cs1 );
DWORD sl = SleepEx( 250, FALSE );
EnterCriticalSection( &cs2 ); //Line 506
LeaveCriticalSection( &cs2 );
LeaveCriticalSection( &cs1 );
}
Thread 18
Badext.dll line 517 looks like:
void LockAll( )
{
EnterCriticalSection( &cs2 );
DWORD sl = SleepEx( 150, FALSE );
EnterCriticalSection( &cs1 ); //Line 517
LeaveCriticalSection( &cs1 );
LeaveCriticalSection( &cs2 );
}
Thread 17 is running the LockTables function code and thread 18 is running the function LockAll code. Thread 17 owned cs1 and put itself into the
sleep state for 250 ms before asking for cs2. This time would assure that thread 18 will have enough time to entre cs2 and ask for cs1. Hence a deadlock.