Debug Labs


 Deadlock in Component - Critical Section

This lab will demonstrate a deadlock in the COM Component. In this scenarion, two threads will be deadlocked trying to gain access to Critical Sections

 

1- Lab Setup
	

Note: This lab assumes that you have already installed the necessary lab files to run it.

1- Verify the site is running in medium protection

2- Verify that the process "dllhost.exe" mapped to the out of process pooled application is running and note down its PID

To find the PID of the desired dllhost, do the follwing:

- Open Component Services

- Expand "Component Services" Node

- Expand "Computers" Node

- Expand "My Computer" Node

- Select "COM+ Applications"

- Click "Status View" icon and notice the PID column on the right hand pane

3- Attach windbg to the dllhost process with the PID found in step 2

4- Attach windbg to the dllhost process with the PID found in step 2

a) Using the GUI:

- Click "File"

- Click "Attach to Process"

- Select "dllhost.exe" with the PID found in step 2 and click "OK"

b) Using the command prompt:

- Switch to thr folder where the debugging tools for windows is installed

- Type in "windbg -p <PID>" and hit Enter

5- In windbg command line, type in "g" with no quotation marks and press Enter

 

 

2- Run Scenario

1- Click the "Invoke Component" button to load the COM Component in the dllhost process memory space. You should see the ISAPI extension "badExt.dll"among the list of modules loaded in dllhost.exe
2- Click the "DL Component" button to cause the process hosting the COM Component to deadlock.

3- In Windbg, press "Ctrl- Break".

                                                                           

 

1- Debug Annotation

 

1) Without symbols

 

Without symbols, it is very hard to debug a hang issue. The reason is that a hang is caused because of some synchronization problems.

To implement synchronization, devolopers use synchronization objects such as Mutex, Critsec, Event, ...via APIs. The following MSDN article covers synchronization.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndllpro/html/msdn_locktest.asp

 

So, again without symbols, the debugger is unable to resolve function calls and hence you will not be able to identify synchronization APIs. On the other hand,

some built-in debug extensions that help identify synchronization problems will not run properly. As an example, the " !locks " debug extension that shows information

about all the critical sections initialized in the process will not display any information if you don't have at least public symbols for "ntdll.dll"

 

 

2) With symbols including public symbols of "badCom.dll"

 

 

First, we need to add the private symbol path(company wide symbols) as well as public symbols of our dll

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pub and hit Enter

b) Using the ".sympath+" command:

.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pub

 

Basically, the first debugger command to run when analyzing a hang is the " ~*kv " .This command will display the stack of each thread in the process.

The command output will give you an idea about the threads running in the process. If you are familiar with the win32 synchronization objects, you could start

by looking for APIs that handle these objects such as "RtlEnterCriticalSection"

 

In this scenario, the following threads show calls to "RtlEnterCriticalSection" and "RtlWaitForCriticalSection"

 

27 Id: 974.a68 Suspend: 1 Teb: 7ffd8000 Unfrozen

# ChildEBP RetAddr Args to Child

00 0210ff24 77f85c55 000005b8 00000000 00000000 ntdll!ZwWaitForSingleObject+0xb

01 0210ff98 77f85bd1 02034100 020331a3 02034138 ntdll!RtlpWaitForCriticalSection+0x9e

02 0210ffa0 020331a3 02034138 77fcb52d 00000000 ntdll!RtlEnterCriticalSection+0x46

03 0210ffec 00000000 0203317d 00000000 00000000 badcom!LockTables+0x26

 

28 Id: 974.860 Suspend: 1 Teb: 7ffd7000 Unfrozen

# ChildEBP RetAddr Args to Child

00 0214ff24 77f85c55 000005b4 00000000 00000000 ntdll!ZwWaitForSingleObject+0xb

01 0214ff98 77f85bd1 02034100 020331d9 02034120 ntdll!RtlpWaitForCriticalSection+0x9e

02 0214ffa0 020331d9 02034120 77fcb52d 00000000 ntdll!RtlEnterCriticalSection+0x46

03 0214fff0 020331b3 00000000 00000000 00000000 badcom!LockAll+0x26

04 00000000 00000000 00000000 00000000 00000000 badcom!LockAll

 

 

The first parameter passed to "RtlEnterCriticalSection" is a pointer to the critical section object that the thread wants to

get a hold of. The following debugger command displays the critical section object address as well as the value of some data members of the critical section object.

 

 

0:029> !critsec 02034138

CritSec badcom!cs2+0 at 02034138

LockCount 1

RecursionCount 1

OwningThread 860

EntryCount 1

ContentionCount 1

*** Locked

 

0:029> !critsec 02034120

CritSec badcom!cs1+0 at 02034120

LockCount 1

RecursionCount 1

OwningThread a68

EntryCount 1

ContentionCount 1

*** Locked

 

 

What these data members mean?

LockCount: Number of threads waiting on the Critical Section to be released

RecursionCount: Number of times the owning thread called "EnterCriticalSection"

OwningThread: ID of the thread owning the Critical Section

EntryCount: Number of times threads other than the owning thread called "EnterCriticalSection"

ContentionCount: Usually it is the same as EntryCount, but it might be different (it has to do with Lock Convoy, more reading http://sts/faster)

 

 

Another command very useful in analyzing hangs is the " !locks " which scans the whole process looking for critical section objects.

 

0:029> !locks

CritSec badcom!cs1+0 at 02034120

LockCount 1

RecursionCount 1

OwningThread a68

EntryCount 1

ContentionCount 1

*** Locked

 

CritSec badcom!cs2+0 at 02034138

LockCount 1

RecursionCount 1

OwningThread 860

EntryCount 1

ContentionCount 1

*** Locked

 

 

What this shows is that there are 2 Critical Sections in this process. The first one is at ( 02034120 )and the second one is at ( 02034138).

Notice that thread ID "a68" owns Critical Section ( 02034120 ) and thread ID "860" owns critical section ( 02034138).

 

Note that thread ID is different from thread instance. Thread ID is the ID that the system assigns to a thread when it is spawned, the thread will keep this ID

as long as it is alive, while thread instance is just added by the debugger for presentation purposes. To get the mapping (Instance --> ID), you could issue

the following command

 

0:029> ~

0 Id: 974.a44 Suspend: 1 Teb: 7ffde000 Unfrozen

1 Id: 974.7b4 Suspend: 1 Teb: 7ffdd000 Unfrozen

2 Id: 974.98c Suspend: 1 Teb: 7ffdc000 Unfrozen

3 Id: 974.988 Suspend: 1 Teb: 7ffda000 Unfrozen

4 Id: 974.8c Suspend: 1 Teb: 7ffdb000 Unfrozen

5 Id: 974.468 Suspend: 1 Teb: 7ffd6000 Unfrozen

6 Id: 974.7e8 Suspend: 1 Teb: 7ffd5000 Unfrozen

7 Id: 974.960 Suspend: 1 Teb: 7ffd4000 Unfrozen

8 Id: 974.a30 Suspend: 1 Teb: 7ffaf000 Unfrozen

9 Id: 974.af0 Suspend: 1 Teb: 7ffae000 Unfrozen

10 Id: 974.8e0 Suspend: 1 Teb: 7ffad000 Unfrozen

11 Id: 974.a7c Suspend: 1 Teb: 7ffac000 Unfrozen

12 Id: 974.998 Suspend: 1 Teb: 7ffaa000 Unfrozen

13 Id: 974.8a4 Suspend: 1 Teb: 7ffa9000 Unfrozen

14 Id: 974.588 Suspend: 1 Teb: 7ffa8000 Unfrozen

15 Id: 974.6b4 Suspend: 1 Teb: 7ffa7000 Unfrozen

16 Id: 974.a54 Suspend: 1 Teb: 7ffa6000 Unfrozen

17 Id: 974.ad8 Suspend: 1 Teb: 7ffa5000 Unfrozen

18 Id: 974.180 Suspend: 1 Teb: 7ffa4000 Unfrozen

19 Id: 974.700 Suspend: 1 Teb: 7ffa3000 Unfrozen

20 Id: 974.578 Suspend: 1 Teb: 7ffa2000 Unfrozen

21 Id: 974.4f4 Suspend: 1 Teb: 7ffa1000 Unfrozen

22 Id: 974.5f0 Suspend: 1 Teb: 7ffa0000 Unfrozen

23 Id: 974.9bc Suspend: 1 Teb: 7ff9f000 Unfrozen

24 Id: 974.128 Suspend: 1 Teb: 7ff9e000 Unfrozen

25 Id: 974.9f0 Suspend: 1 Teb: 7ff9d000 Unfrozen

26 Id: 974.af8 Suspend: 1 Teb: 7ffd9000 Unfrozen

27 Id: 974.a68 Suspend: 1 Teb: 7ffd8000 Unfrozen

28 Id: 974.860 Suspend: 1 Teb: 7ffd7000 Unfrozen .

.29 Id: 974.95c Suspend: 1 Teb: 7ffab000 Unfrozen

 

 

So thread instance 27 owns critical section ( 02034120 ). This thread is trying to get a hold of critical section ( 02034138 ). (because there is a call to NTDLL!RtlEnterCriticalSection ())

On the other hand, thread 18 owns critical section ( 02034138 ) and wants to get a hold of critical section ( 02034120 ). So what happens here is that thread 27 will not release ( 02034120 )

until he gets a hold of ( 02034138). The same thing with thread 28 which will not release ( 02034138) until it gets a hold of ( 02034120 ). This is basically a deadlock.

 

It appears clear from the critical section addresses (badCom!cs1) and (badCom!cs2) that this deadlock is actually in the Com Component "badCom.dll". This is enough to tell the

customer that the culprit is is "badCom.dll".

 

To verify that the symbols are lined up properly for our extension, issue this command.

 

 

0:029> lmv mbadcom

start end module name

02030000 02038000 badcom (pdb symbols) c:\iisdebuglabs\symbols\pub\badcom.pdb

Loaded symbol image file: badcom.dll

Image path: c:\IISDEB~1\www\app\badcom.dll

Timestamp: Tue Jun 10 22:45:51 2003 (3EE697DF) Checksum: 0000F229

File version: 1.0.0.1

Product version: 1.0.0.1

File flags: 0 (Mask 3F)

File OS: 4 Unknown Win32

File type: 2.0 Dll

File date: 00000000.00000000

CompanyName:

ProductName: badCOM

Module InternalName: badCOM

OriginalFilename: badCOM.DLL

ProductVersion: 1, 0, 0, 1

FileVersion: 1, 0, 0, 1

FileDescription: badCOM Module

LegalCopyright: Copyright 2003

 

 

3) with Private Symbols "badCom.dll"

 

First, we need to add the private symbol path(company wide symbols) as well as private symbols of our dll

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pri and hit Enter

b) Using the ".sympath+" command:

.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pri

 

Thread instances 27 and 28 look like respectively:

 

0:029> ~27kvn

# ChildEBP RetAddr Args to Child

00 0210ff24 77f85c55 000005b8 00000000 00000000 ntdll!ZwWaitForSingleObject+0xb (FPO: [3,0,0]) [i386\usrstubs.asm @ 2004]

01 0210ff98 77f85bd1 02034100 020331a3 02034138 ntdll!RtlpWaitForCriticalSection+0x9e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ntos\dll\resource.c @ 1612]

02 0210ffa0 020331a3 02034138 77fcb52d 00000000 ntdll!RtlEnterCriticalSection+0x46 (FPO: [1,0,0]) [..\i386\critsect.asm @ 157]

03 0210ffec 00000000 0203317d 00000000 00000000 badcom!LockTables+0x26 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\comDBG\scenarios.cpp @ 616]

 

 

0:029> ~28kvn # ChildEBP RetAddr Args to Child

00 0214ff24 77f85c55 000005b4 00000000 00000000 ntdll!ZwWaitForSingleObject+0xb (FPO: [3,0,0]) [i386\usrstubs.asm @ 2004]

01 0214ff98 77f85bd1 02034100 020331d9 02034120 ntdll!RtlpWaitForCriticalSection+0x9e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ntos\dll\resource.c @ 1612]

02 0214ffa0 020331d9 02034120 77fcb52d 00000000 ntdll!RtlEnterCriticalSection+0x46 (FPO: [1,0,0]) [..\i386\critsect.asm @ 157]

03 0214fff0 020331b3 00000000 00000000 00000000 badcom!LockAll+0x26 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\comDBG\scenarios.cpp @ 626]

04 00000000 00000000 00000000 00000000 00000000 badcom!LockAll (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\comDBG\scenarios.cpp @ 622]

 

With Private symbols lined up as well as the source code of "badcom.dll", Notice the lines of code of "scenarios.cpp" executing in each thread.

 

Thread 27

Scenarios.cpp line 616 looks like:

 

void LockTables( )

{

EnterCriticalSection( &cs1 );

DWORD sl = SleepEx( 250, FALSE );

EnterCriticalSection( &cs2 ); //Line 616

LeaveCriticalSection( &cs2 );

LeaveCriticalSection( &cs1 );

}

 

 

 

Thread 28

Scenarios.cpp line 622 looks like:

 

void LockAll( )

{

EnterCriticalSection( &cs2 );

DWORD sl = SleepEx( 150, FALSE );

EnterCriticalSection( &cs1 ); //Line 622

LeaveCriticalSection( &cs1 );

LeaveCriticalSection( &cs2 );

}

 

Thread 27 is running the LockTables function code and thread 28 is running the function LockAll code. Thread 27 owned cs1 and put itself into the

sleep state for 250 ms before asking for cs2. This time would assure that thread 28 will have enough time to entre cs2 and ask for cs1. Hence a deadlock.

 

 

 


- Microsoft Confidential -