Debug Labs


 Deadlock in Component - Critical Section

This lab will demonstrate a deadlock in the CGI Executable. In this scenarion, two threads will be deadlocked trying to gain access to a Critical Section

 

1- Lab Setup
	

Note: This lab assumes that you have already installed the necessary lab files to run it.

1- Set the IIS Admin Service to interact with the Desktop, and verify the service is running under the system account.

- In "Administrative Tools" click "Services"

- Double Click "IIS Admin Service"

- Click the "Log On" Tab

- Verify that the radio button "Local System Account" is selected

- Check "Allow Services to interact with Desktop" checkbox

2- Set the "Image File Execution Options" for the CGI Executable "badCgi.exe"

- Start Registry Editor (Regedt32.exe).

- Locate the following registry key: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\]

- Create a subkey with the name "badCgi.exe"

- To the "badCgi.exe" subkey, add the REG_SZ value named Debugger and the full path to the debugger with the "-g" switch: eg: c:\debugger\windbg -g

 

3- Set the "CreateProcessAsUser" metabase prooperty to "false"

- Open the command prompt (cmd)

- Switch to "Inepub\Adminscripts" folder

- Type in: CScript.exe adsutil.vbs set w3svc/CreateProcessAsUser "false" and hit Enter

 

2- Run Scenario

2- Click the "DL CGI" button to cause the process hosting the ISAPI Extension to deadlock.
3- In Windbg, press "Ctrl- Break".

                

 

1- Debug Annotation

 

1) Without symbols

 

Without symbols, it is very hard to debug a hang issue. The reason is that a hang is caused because of some synchronization problems.

To implement synchronization, devolopers use synchronization objects such as Mutex, Critsec, Event, ...via APIs. The following MSDN article covers synchronization.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndllpro/html/msdn_locktest.asp

 

So, again without symbols, the debugger is unable to resolve function calls and hence you will not be able to identify synchronization APIs. On the other hand,

some built-in debug extensions that help identify synchronization problems will not run properly. As an example, the " !locks " debug extension that shows information

about all the critical sections initialized in the process will not display any information if you don't have at least public symbols for "ntdll.dll"

 

 

2) With symbols including public symbols of "badCgi.exe"

 

First, we need to add the private symbol path(company wide symbols) as well as public symbols of our exe

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pub and hit Enter

b) Using the ".sympath+" command:

.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pub

 

Basically, the first debugger command to run when analyzing a hang is the " ~*kv " .This command will display the stack of each thread in the process.

The command output will give you an idea about the threads running in the process. If you are familiar with the win32 synchronization objects, you could start

by looking for APIs that handle these objects such as "RtlEnterCriticalSection"

 

In this scenario, the following threads show calls to "RtlEnterCriticalSection" and "RtlWaitForCriticalSection"

 

1 Id: 978.8e4 Suspend: 1 Teb: 7ffdd000 Unfrozen

# ChildEBP RetAddr Args to Child

00 0028ff24 77f85c55 00000050 00000000 00000000 ntdll!ZwWaitForSingleObject+0xb (FPO: [3,0,0]) [i386\usrstubs.asm @ 2004]

01 0028ff98 77f85bd1 01003000 01001c78 01003038 ntdll!RtlpWaitForCriticalSection+0x9e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ntos\dll\resource.c @ 1612]

02 0028ffa0 01001c78 01003038 00230000 00000001 ntdll!RtlEnterCriticalSection+0x46 (FPO: [1,0,0]) [..\i386\critsect.asm @ 157]

03 0028ffec 00000000 01001c52 00000000 00000000 badcgi!LockTables+0x26

 

 

2 Id: 978.588 Suspend: 1 Teb: 7ffdc000 Unfrozen

# ChildEBP RetAddr Args to Child

00 002cff24 77f85c55 0000004c 00000000 00000000 ntdll!ZwWaitForSingleObject+0xb (FPO: [3,0,0]) [i386\usrstubs.asm @ 2004]

01 002cff98 77f85bd1 01003000 01001cae 01003020 ntdll!RtlpWaitForCriticalSection+0x9e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ntos\dll\resource.c @ 1612]

02 002cffa0 01001cae 01003020 00230000 00000001 ntdll!RtlEnterCriticalSection+0x46 (FPO: [1,0,0]) [..\i386\critsect.asm @ 157]

03 002cfff0 01001c88 00000000 00000000 00000000 badcgi!LockAll+0x26

04 00000000 00000000 00000000 00000000 00000000 badcgi!LockAll

 

 

The first parameter passed to "RtlEnterCriticalSection" is a pointer to the critical section object that the thread wants to

get a hold of. The following debugger command displays the critical section object address as well as the value of some data members of the critical section object.

 

 

0:003> !critsec 01003038

CritSec badcgi!cs2+0 at 01003038

LockCount 1

RecursionCount 1

OwningThread 588

EntryCount 1

ContentionCount 1

*** Locked

 

0:003> !critsec 01003020

CritSec badcgi!cs1+0 at 01003020

LockCount 1

RecursionCount 1

OwningThread 8e4

EntryCount 1

ContentionCount 1

*** Locked

 

 

What these data members mean?

 

LockCount: Number of threads waiting on the Critical Section to be released

RecursionCount: Number of times the owning thread called "EnterCriticalSection"

OwningThread: ID of the thread owning the Critical Section

EntryCount: Number of times threads other than the owning thread called "EnterCriticalSection"

ContentionCount: Usually it is the same as EntryCount, but it might be different (it has to do with Lock Convoy, more reading http://sts/faster)

 

Another command very useful in analyzing hangs is the " !locks " which scans the whole process looking for critical section objects.

 

0:000> !locks

CritSec badcgi!cs1+0 at 01003020

LockCount 1

RecursionCount 1

OwningThread 8e4

EntryCount 1

ContentionCount 1

*** Locked

CritSec badcgi!cs2+0 at 01003038

LockCount 1

RecursionCount 1

OwningThread 588

EntryCount 1

ContentionCount 1

*** Locked

 

 

What this shows is that there are 2 Critical Sections in this process. The first one is at (01003020 )and the second one is at (01003038).

Notice that thread ID "8e4" owns Critical Section (01003020) and thread ID "588" owns critical section (01003038).

 

Note that thread ID is different from thread instance. Thread ID is the ID that the system assigns to a thread when it is spawned, the thread will keep this ID

as long as it is alive, while thread instance is just added by the debugger for presentation purposes. To get the mapping (Instance --> ID), you could issue

the following command

 

0:003> ~

0 Id: 978.964 Suspend: 1 Teb: 7ffde000 Unfrozen

1 Id: 978.8e4 Suspend: 1 Teb: 7ffdd000 Unfrozen

2 Id: 978.588 Suspend: 1 Teb: 7ffdc000 Unfrozen .

3 Id: 978.28c Suspend: 1 Teb: 7ffdb000 Unfrozen

 

So thread instance 1 owns critical section (01003020). This thread is trying to get a hold of critical section (01003038). (because there is a call to NTDLL!RtlEnterCriticalSection ())

On the other hand, thread 2 owns critical section (01003038 ) and wants to get a hold of critical section (01003020 ). So what happens here is that thread 1 will not release (01003038 )

until he gets a hold of (01003020). The same thing with thread 2 which will not release (01003020) until it gets a hold of (01003038 ). This is basically a deadlock.

 

It appears clear from the critical section addresses (badCgi!cs1) and (badCgi!cs2) that this deadlock is actually in the CGI executable "badCgi.dll". This is enough to tell the

customer that the culprit is is "badCgi.dll".

 

To verify that the symbols are lined up properly for our extension, issue this command.

 

0:000> lmv mbadcgi

start end module name

01000000 01004000 badcgi (pdb symbols) c:\iisdebuglabs\symbols\pub\badcgi.pdb

Loaded symbol image file: badcgi.exe

Image path: C:\IISDebugLabs\www\app\badcgi.exe

Timestamp: Tue May 13 17:49:56 2003 (3EC16884) Checksum: 0000691D

 

3) with Private Symbols "badCgi.dll"

 

 

First, we need to add the private symbol path(company wide symbols) as well as private symbols of our exe

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pri and hit Enter

b) Using the ".sympath+" command:

.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pri

Thread instances 1 and 2 look like respectively:

 

 

1 Id: 978.8e4 Suspend: 1 Teb: 7ffdd000 Unfrozen

# ChildEBP RetAddr Args to Child

00 0028ff24 77f85c55 00000050 00000000 00000000 ntdll!ZwWaitForSingleObject+0xb (FPO: [3,0,0]) [i386\usrstubs.asm @ 2004]

01 0028ff98 77f85bd1 01003000 01001c78 01003038 ntdll!RtlpWaitForCriticalSection+0x9e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ntos\dll\resource.c @ 1612]

02 0028ffa0 01001c78 01003038 00230000 00000001 ntdll!RtlEnterCriticalSection+0x46 (FPO: [1,0,0]) [..\i386\critsect.asm @ 157]

03 0028ffec 00000000 01001c52 00000000 00000000 badcgi!LockTables+0x26 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\cgiDBG\badcgi.cpp @ 687]

 

 

2 Id: 978.588 Suspend: 1 Teb: 7ffdc000 Unfrozen

# ChildEBP RetAddr Args to Child

00 002cff24 77f85c55 0000004c 00000000 00000000 ntdll!ZwWaitForSingleObject+0xb (FPO: [3,0,0]) [i386\usrstubs.asm @ 2004]

01 002cff98 77f85bd1 01003000 01001cae 01003020 ntdll!RtlpWaitForCriticalSection+0x9e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ntos\dll\resource.c @ 1612]

02 002cffa0 01001cae 01003020 00230000 00000001 ntdll!RtlEnterCriticalSection+0x46 (FPO: [1,0,0]) [..\i386\critsect.asm @ 157]

03 002cfff0 01001c88 00000000 00000000 00000000 badcgi!LockAll+0x26 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\cgiDBG\badcgi.cpp @ 698]

04 00000000 00000000 00000000 00000000 00000000 badcgi!LockAll (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\cgiDBG\badcgi.cpp @ 694]

 

 

With Private symbols lined up as well as the source code of "badCgi.exe", Notice the lines of code of "badCgi.cpp" executing in each thread.

 

Thread 1

BadCgi.cpp line 687 looks like:

 

void LockTables( )

{

EnterCriticalSection( &cs1 );

DWORD sl = SleepEx( 250, FALSE );

EnterCriticalSection( &cs2 ); //Line 687

LeaveCriticalSection( &cs2 );

LeaveCriticalSection( &cs1 );

}

 

 

 

Thread 2

BadCgi.dll line 698 looks like:

 

void LockAll( )

{

EnterCriticalSection( &cs2 );

DWORD sl = SleepEx( 150, FALSE );

EnterCriticalSection( &cs1 ); //Line 698

LeaveCriticalSection( &cs1 );

LeaveCriticalSection( &cs2 );

}

 

Thread 1 is running the LockTables function code and thread 2 is running the function LockAll code. Thread 1 owned cs1 and put itself into the

sleep state for 250 ms before asking for cs2. This time would assure that thread 2 will have enough time to entre cs2 and ask for cs1. Hence a deadlock.

 

 

 

 


- Microsoft Confidential -