Debug Labs


 Deadlock in Component - Critical Section

This lab will demonstrate a deadlock in the ISAPI Filter. In this scenario, two threads will be deadlocked trying to gain access to a Critical Section

 

1- Lab Setup
	

Note: This lab assumes that you have already installed the necessary lab files to run it.

1- Install the ISAPI Filter

- Right Click the Server Name

- Click "Properties"

- In the "WWW Service" Master Properties, click "Edit"

- In the "ISAPI Filters" tab, click "Add"

- In the "Filter Name" type in "BadFilter"

- In the "Executable", browse to C:\IISDebugLabs\www\App\badfil.dll

- Click "Apply" and click "Ok" all teh way out

2- Restart the IIS Admin service

- Right Click the Server Name

- Click "Restart IIS" and Click "Ok"

3- Attach windbg to inetinfo.exe

a) Using the GUI

- Click "File"

- Click "Attach to Process"

- Select "inetinfo.exe" and click "OK"

b) Using the command prompt

-Type in "windng -pn inetinfo.exe" without quotation marks and hit "Enter"

 

4- In windbg command line, type in "g" with no quotation marks and press Enter

 

 

2- Run Scenario

1- Click the "DL Filter" button to cause a deadlock
2- Press "Ctrl-Break" in Winddbg to break into inetinfo. 

                

 

1- Debug Annotation

 

1) Without symbols

 

Without symbols, it is very hard to debug a hang issue. The reason is that a hang is caused because of some synchronization problems.

To implement synchronization, devolopers use synchronization objects such as Mutex, Critsec, Event, ...via APIs. The following MSDN article covers synchronization.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndllpro/html/msdn_locktest.asp

 

So, again without symbols, the debugger is unable to resolve function calls and hence you will not be able to identify synchronization APIs. On the other hand,

some built-in debug extensions that help identify synchronization problems will not run properly. As an example, the " !locks " debug extension that shows information

about all the critical sections initialized in the process will not display any information if you don't have at least public symbols for "ntdll.dll"

 

 

2) With symbols including public symbols of "badFil.dll"

 

First, we need to add the public symbol path(company wide symbols) as well as symbols of our dll

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols;C:\IISDebugLabs\symbols\Pub and hit Enter

b) Using the ".sympath+" command:

0:000> .sympath+ SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols

Symbol search path is: SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols

 

0:000> .sympath+ C :\IISDebugLabs\symbols\Pub (where public symbols of badFil.dll are located)

Symbol search path is:SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols; C :\IISDebugLabs\symbols\Pub

 

Note: For more information on setting the symbol path for windbg, please refer to the debugger documentation for the .sympath command. This will describe

the "downstream store" feature that lets you download symbols to a symbol server of your own.

 

Basically, the first debugger command to run when analyzing a hang is the " ~*kv " .This command will display the stack of each thread in the process.

The command output will give you an idea about the threads running in the process. If you are familiar with the win32 synchronization objects, you could start

by looking for APIs that handle these objects such as "RtlEnterCriticalSection"

 

In this scenario, the following threads show calls to "RtlEnterCriticalSection" and "RtlWaitForCriticalSection"

 

20 Id: 998.898 Suspend: 1 Teb: 7ffa6000 Unfrozen

# ChildEBP RetAddr Args to Child

00 0198ff24 77f85c55 000006d8 00000000 00000000 ntdll!ZwWaitForSingleObject+0xb

01 0198ff98 77f85bd1 01733000 01732086 01733020 ntdll!RtlpWaitForCriticalSection+0x9e

02 0198ffa0 01732086 01733020 0129f2e0 0129f088 ntdll!RtlEnterCriticalSection+0x46

03 0198ffec 00000000 01732060 00000000 00000000 badfil!LockTables+0x26

 

 

21 Id: 998.a30 Suspend: 1 Teb: 7ffa5000 Unfrozen

# ChildEBP RetAddr Args to Child

00 019cff24 77f85c55 000006d4 00000000 00000000 ntdll!ZwWaitForSingleObject+0xb

01 019cff98 77f85bd1 01733000 017320bc 01733008 ntdll!RtlpWaitForCriticalSection+0x9e

02 019cffa0 017320bc 01733008 0129f2e0 0129f088 ntdll!RtlEnterCriticalSection+0x46

03 019cfff0 01732096 00000000 00000000 00000000 badfil!LockAll+0x26

04 00000000 00000000 00000000 00000000 00000000 badfil!LockAll

 

 

The first parameter passed to "RtlEnterCriticalSection" is a pointer to the critical section object that the thread wants to

get a hold of. The following debugger command displays the critical section object address as well as the value of some data members of the critical section object.

 

 

0:019> !critsec 01733020

CritSec badfil!cs2+0 at 01733020

LockCount 1

RecursionCount 1

OwningThread a30

EntryCount 1

ContentionCount 1

*** Locked

 

0:019> !critsec 01733008

CritSec badfil!cs1+0 at 01733008

LockCount 1

RecursionCount 1

OwningThread 898

EntryCount 1

ContentionCount 1

*** Locked

 

 

What these data members mean?

LockCount: Number of threads waiting on the Critical Section to be released

RecursionCount: Number of times the owning thread called "EnterCriticalSection"

OwningThread: ID of the thread owning the Critical Section

EntryCount: Number of times threads other than the owning thread called "EnterCriticalSection"

ContentionCount: Usually it is the same as EntryCount, but it might be different (it has to do with Lock Convoy, more reading http://sts/faster)

 

Another command very useful in analyzing hangs is the " !locks " which scans the whole process looking for critical section objects.

 

0:000> !locks

 

0:019> !critsec 01733020

CritSec badfil!cs2+0 at 01733020

LockCount 1

RecursionCount 1

OwningThread a30

EntryCount 1

ContentionCount 1

*** Locked

CritSec badfil!cs1+0 at 01733008

LockCount 1

RecursionCount 1

OwningThread 898

EntryCount 1

ContentionCount 1

*** Locked

 

 

(Locks with 0 count are ignored in this example)

 

 

What this shows is that there are 2 Critical Sections in this process. The first one is at ( 01733020)and the second one is at (01733008).

Notice that thread ID "a30" owns Critical Section ( 01733020 ) and thread ID "898" owns critical section ( 01733008).

 

Note that thread ID is different from thread instance. Thread ID is the ID that the system assigns to a thread when it is spawned, the thread will keep this ID

as long as it is alive, while thread instance is just added by the debugger for presentation purposes. To get the mapping (Instance --> ID), you could issue

the following command

 

 

0:019> ~~[a30]

21 Id: 998.a30 Suspend: 1 Teb: 7ffa5000 Unfrozen

 

0:019> ~~[898]

20 Id: 998.898 Suspend: 1 Teb: 7ffa6000 Unfrozen

 

 

So thread instance 21 owns critical section (01733020). This thread is trying to get a hold of critical section (01733008). (because there is a call to NTDLL!RtlEnterCriticalSection ())

On the other hand, thread 20 owns critical section (01733008) and wants to get a hold of critical section (01733020 ). So what happens here is that thread 20 will not release (01733008)

until he gets a hold of (01733020). The same thing with thread 21 which will not release (01733020) until it gets a hold of (01733008). This is basically a deadlock.

 

It appears clear from the critical section addresses (badFil!cs1) and (badFil!cs2) that this deadlock is actually in the ISAPI filter "badfil.dll". This is enough to tell the

customer that the culprit is is "badext.dll".

 

To verify that the symbols are lined up properly for our extension, issue this command.

 

0:019> lmv mbadfil

start end module name

015a0000 015a6000 badfil (pdb symbols) C:\IISDebugLabs\Symbols\Pri

Loaded symbol image file: badfil.dll

Image path: C:\IISDebugLabs\www\App\badfil.dll

Timestamp: Wed May 28 01:11:01 2003 (3ED444E5)

Checksum: 0000D9A0

File version: 5.0.0.0

Product version: 5.0.0.0

File flags: 0 (Mask 3F)

File OS: 40004 NT Win32

File type: 2.0 Dll

File date: 00000000.00000000

CompanyName: Microsoft Corporation

ProductName: Internet Information Services

InternalName: badFIL.dll

OriginalFilename: badFIL.dll

ProductVersion: 5.00.0984

FileVersion: 5.00.0984

FileDescription: badFIL Filter DLL

LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999

 

3) with Private Symbols "badFil.dll"

 

 

Point the debugger to use the private symbols

For company wide internal symbols, use \\symbols\symbols

For the component private symbols, use C:\IISDebugLabs\Symbols\Pri

 

 

Thread instances 21 and 21 look like respectively:

 

 

20 Id: 998.898 Suspend: 1 Teb: 7ffa6000 Unfrozen

# ChildEBP RetAddr Args to Child

00 0198ff24 77f85c55 000006d8 00000000 00000000 ntdll!ZwWaitForSingleObject+0xb (FPO: [3,0,0]) [i386\usrstubs.asm @ 2004]

01 0198ff98 77f85bd1 01733000 01732086 01733020 ntdll!RtlpWaitForCriticalSection+0x9e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ntos\dll\resource.c @ 1612]

02 0198ffa0 01732086 01733020 0129f2e0 0129f088 ntdll!RtlEnterCriticalSection+0x46 (FPO: [1,0,0]) [..\i386\critsect.asm @ 157]

03 0198ffec 00000000 01732060 00000000 00000000 badfil!LockTables+0x26 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 993]

 

 

21 Id: 998.a30 Suspend: 1 Teb: 7ffa5000 Unfrozen

# ChildEBP RetAddr Args to Child

00 019cff24 77f85c55 000006d4 00000000 00000000 ntdll!ZwWaitForSingleObject+0xb (FPO: [3,0,0]) [i386\usrstubs.asm @ 2004]

01 019cff98 77f85bd1 01733000 017320bc 01733008 ntdll!RtlpWaitForCriticalSection+0x9e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ntos\dll\resource.c @ 1612]

02 019cffa0 017320bc 01733008 0129f2e0 0129f088 ntdll!RtlEnterCriticalSection+0x46 (FPO: [1,0,0]) [..\i386\critsect.asm @ 157]

03 019cfff0 01732096 00000000 00000000 00000000 badfil!LockAll+0x26 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 1004]

04 00000000 00000000 00000000 00000000 00000000 badfil!LockAll (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 1000]

 

 

With Private symbols lined up as well as the source code of "badext.dll", Notice the lines of code of "badext.cpp" executing in each thread.

 

Thread 20

BadFil.cpp line 993 looks like:

 

void LockTables( )

{

EnterCriticalSection( &cs1 );

DWORD sl = SleepEx( 250, FALSE );

EnterCriticalSection( &cs2 ); //Line 993

LeaveCriticalSection( &cs2 );

LeaveCriticalSection( &cs1 );

}

 

 

 

Thread 21

BadFil.dll line 1004 looks like:

 

void LockAll( )

{

EnterCriticalSection( &cs2 );

DWORD sl = SleepEx( 150, FALSE );

EnterCriticalSection( &cs1 ); //Line 1004

LeaveCriticalSection( &cs1 );

LeaveCriticalSection( &cs2 );

}

 

Thread 20 is running the LockTables function code and thread 21 is running the function LockAll code. Thread 20 owned cs1 and put itself into the

sleep state for 250 ms before asking for cs2. This time would assure that thread 21 will have enough time to entre cs2 and ask for cs1. Hence a deadlock.

 

 

 


- Microsoft Confidential -