Debug Labs


Slow Response in Component

This lab will demonstrate a slow response because of component code.

 

1- Lab Setup
	

Note: This lab assumes that you have already installed the necessary lab files to run it.

1- Verify the site is running in medium protection

2- Verify that the process "dllhost.exe" mapped to the out of process pooled application is running and note down its PID

To find the PID of the desired dllhost, do the follwing:

- Open Component Services

- Expand "Component Services" Node

- Expand "Computers" Node

- Expand "My Computer" Node

- Select "COM+ Applications"

- Click "Status View" icon and notice the PID column on the right hand pane

3- Attach windbg to the dllhost process with the PID found in step 2

a) Using the GUI:

- Click "File"

- Click "Attach to Process"

- Select "dllhost.exe" with the PID found in step 2 and click "OK"

b) Using the command prompt:

- Switch to thr folder where the debugging tools for windows is installed

- Type in "windbg -p <PID>" and hit Enter

4- In windbg command line, type in "g" with no quotation marks and press Enter

 

 

2- Run Scenario

1- Click the "Invoke Extension" button to load the ISAPI Extension in the dllhost process memory space. You should see the ISAPI extension "badExt.dll"among the list of modules loaded in dllhost.exe
2- Click the "Cause XXs Delay" button to cause a slow response
 During that period of time, Press "Ctrl-Break" in Winddbg. 

					

                                                                           

                                                                                         

                                                                                         

1- Debug Annotation

 

1) Without symbols

 

a) Looking for the thread running the request.

 

The thread condidate in a slow response would most likely contain a call to "ZwDelayExecution()" .

 

In our example, here is the thread making that call.

 

 

4 Id: b18.9b8 Suspend: 1 Teb: 7ffdb000 Unfrozen

# ChildEBP RetAddr Args to Child

WARNING: Stack unwind information not available. Following frames may be wrong.

00 00a1f6f4 012917ec 00007530 00000000 0040c6a8 ntdll!ZwDelayExecution+0xb

01 00a1f70c 012914cd 00e01e90 00000001 00e01f5c badext!HttpExtensionProc+0x4c9

02 00a1f728 65d82b06 00e01e90 00e01f5c 00e01e90 badext!HttpExtensionProc+0x1aa

03 00a1f770 65d823a2 0040a8c8 00e02005 00e01e90 wam!SE_TABLE::ReleaseExtension+0xbe8

04 00a1f7b4 77d339c0 00406c08 00000000 000000a6 wam!SE_TABLE::ReleaseExtension+0x484

05 00a1f7dc 77d93570 65d8223f 00a1f7f8 00000005 RPCRT4!NdrServerInitialize+0x288

06 00a1fa54 77d949ac 000a36f0 000a1604 000cb5d0 RPCRT4!NdrStubCall2+0x570

07 00a1fab8 77b29e7f 000a36f0 000cb5d0 000a1604 RPCRT4!CStdStubBuffer_Invoke+0x9e

08 00a1fafc 77b29d6e 000cb5d0 000c970c 000a2558 ole32!HDC_UserMarshal+0x78a

09 00a1fb44 77aa7ae7 000cb5d0 000c9620 000a36f0 ole32!HDC_UserMarshal+0x679

0a 00a1fba8 77aa7a33 000a1604 00000000 000a36f0 ole32!CoGetCallContext+0x401

0b 00a1fbc4 77b29c89 000cb5d0 00000001 000a36f0 ole32!CoGetCallContext+0x34d

0c 00a1fbf4 77b29a89 000cb588 000a1604 000a36f0 ole32!HDC_UserMarshal+0x594

0d 00a1fcb4 77b2a12c 000cd9b8 00000000 000c96f0 ole32!HDC_UserMarshal+0x394

0e 00a1fcf4 77d33721 000cb588 000c96f0 0007a410 ole32!HDC_UserMarshal+0xa37

0f 00a1fd2c 77d33667 77b29f7b 0007a410 00a1fe0c RPCRT4!I_RpcFree+0x1ff

10 00a1fd84 77d33579 00000000 00000000 00a1fe0c RPCRT4!I_RpcFree+0x145

11 00a1fda4 77d48e81 0007a410 00000000 00a1fe0c RPCRT4!I_RpcFree+0x57

12 00a1fdd4 77d34bc6 0007a410 0007a3d4 00000000 RPCRT4!RpcSsDestroyClientContext+0xa6

13 00a1fe10 77d346c5 000a5530 000826a0 80030001 RPCRT4!NdrCorrelationFree+0x1120

  

Basically, since only export symbols are used here, we cannot far in analyzing this thread and finding out why this thread is slow in processing the request.

   

2) With symbols including public symbols of "badExt.dll"

 

 

First, we need to add the symbol path(company wide symbols) as well as public symbols of our dll

Let try in this example to use company wide public symbols

 

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols ;C:\IISDebugLabs\symbols\Pub and hit Enter

b) Using the ".sympath+" command:

.sympath+ SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols ;C :\IISDebugLabs\symbols\Pub

 

Note: For more information on setting the symbol path for windbg, please refer to the debugger documentation for the .sympath command. This will describe

the "downstream store" feature that lets you download symbols to a symbol server of your own.  

Visit also http://www.microsoft.com/whdc/ddk/debugging/symbols.mspx

 

 

Here is how the above thread would look like with public symbols.

 

4 Id: b18.9b8 Suspend: 1 Teb: 7ffdb000 Unfrozen

# ChildEBP RetAddr Args to Child 00 00a1f6d4 77e87617 00000000 00a1f6ec 0040c6d8 ntdll!NtDelayExecution+0xb

01 00a1f6f4 012917ec 00007530 00000000 0040c6a8 KERNEL32!SleepEx+0x32

02 00a1f70c 012914cd 00e01e90 00000001 00e01f5c badext!doSlow1+0x40

03 00a1f728 65d82b06 00e01e90 00e01f5c 00e01e90 badext!HttpExtensionProc+0x1aa

04 00a1f73c 65d82a97 00e01e90 00e01f5c 00e01e90 wam!HSE_APPDLL::ExecuteRequest+0x93

05 00a1f770 65d823a2 0040a8c8 00e02005 00e01e90 wam!WAM::InvokeExtension+0x35

06 00a1f7b4 77d339c0 00406c08 00000000 000000a6 wam!WAM::ProcessRequest+0x240

07 00a1f7dc 77d93570 65d8223f 00a1f7f8 00000005 RPCRT4!Invoke+0x30

08 00a1fa54 77d949ac 000a36f0 000a1604 000cb5d0 RPCRT4!NdrStubCall2+0x63d

09 00a1fab8 77b29e7f 000a36f0 000cb5d0 000a1604 RPCRT4!CStdStubBuffer_Invoke+0xec

0a 00a1fafc 77b29d6e 000cb5d0 000c970c 000a2558 ole32!SyncStubInvoke+0x61

0b 00a1fb44 77aa7ae7 000cb5d0 000c9620 000a36f0 ole32!StubInvoke+0xa8

0c 00a1fba8 77aa7a33 000a1604 00000000 000a36f0 ole32!CCtxComChnl::ContextInvoke+0xbb

0d 00a1fbc4 77b29c89 000cb5d0 00000001 000a36f0 ole32!MTAInvoke+0x18

0e 00a1fbf4 77b29a89 000cb588 000a1604 000a36f0 ole32!AppInvoke+0xb5

0f 00a1fcb4 77b2a12c 000cd9b8 00000000 000c96f0 ole32!ComInvokeWithLockAndIPID+0x29e

10 00a1fcf4 77d33721 000cb588 000c96f0 0007a410 ole32!ThreadInvoke+0x1b7

11 00a1fd2c 77d33667 77b29f7b 0007a410 00a1fe0c RPCRT4!DispatchToStubInC+0x84

12 00a1fd84 77d33579 00000000 00000000 00a1fe0c RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x100

13 00a1fda4 77d48e81 0007a410 00000000 00a1fe0c RPCRT4!RPC_INTERFACE::DispatchToStub+0x5e

14 00a1fdd4 77d34bc6 0007a410 0007a3d4 00000000 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xa9

15 00a1fe10 77d346c5 000a5530 000826a0 80030001 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0x1c6

16 00a1fe28 77d422ff 0007a318 00a1fe50 000a5530 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0x10c

17 00a1ff74 77d420d9 77d42528 000826a0 00000000 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x1eb

18 00a1ff78 77d42528 000826a0 00000000 00000000 RPCRT4!RecvLotsaCallsWrapper+0x9

19 00a1ffa8 77d424da 00094ce0 00a1ffec 77e887dd RPCRT4!BaseCachedThreadRoutine+0x4f

1a 00a1ffb4 77e887dd 00094d40 00000000 00000000 RPCRT4!ThreadStartRoutine+0x18

1b 00a1ffec 00000000 77d424c2 00094d40 00000000 KERNEL32!BaseThreadStart+0x52

This thread shows that the main entry in our ISAPI extension (HttpExtensionProc" is calling the function "doSlow1()" which in turn is calling the win32 API "SleepEx".

The first parameter passed to "sleepEx" is the time interval in Milliseconds (Hex Format) for which thread execution is to be suspended.

To evaluate this parameter, do the following:

 

0:015> ? 00007530

Evaluate expression: 30000 = 00007530

 

This shows that the execution of this thread is suspended for 30,000 ms = 30 sec. This is why the response is slow.  

 

To verify that the symbols are lined up properly for our extension, issue this command.

 

 

0:006> lmv mbadext

start end module name 06470000 06475000

badext (pdb symbols) C :\IISDebugLabs\symbols\Pub

Loaded symbol image file: C:\IISDebuglabs\www\App\badext.dll Image path: C:\IISDebuglabs\www\App\badext.dll

Timestamp: Wed Jan 29 18:07:14 2003 (3E385EA2)

Checksum: 0000EBCD

File version: 5.0.0.0

Product version: 5.0.0.0

File flags: 0 (Mask 3F)

File OS: 40004 NT Win32

File type: 2.0 Dll

File date: 00000000.00000000

CompanyName: Microsoft Corporation

ProductName: Internet Information Services

InternalName: badEXT.dll

OriginalFilename: badEXT.dll

ProductVersion: 5.00.0984

FileVersion: 5.00.0984

FileDescription: badEXT Test

LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999

 

3) with Private Symbols "badExt.dll"

 

First, we need to add the symbol path(company wide symbols) as well as private symbols of our dll

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pri and hit Enter

b) Using the ".sympath+" command:

.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pri

The stack backtrace of the thread that should process the request looks like:

 

4 Id: b18.9b8 Suspend: 1 Teb: 7ffdb000 Unfrozen

# ChildEBP RetAddr Args to Child 00 00a1f6d4 77e87617 00000000 00a1f6ec 0040c6d8 ntdll!NtDelayExecution+0xb (FPO: [2,0,0]) [i386\usrstubs.asm @ 532]

01 00a1f6f4 012917ec 00007530 00000000 0040c6a8 KERNEL32!SleepEx+0x32 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\synch.c @ 1672]

02 00a1f70c 012914cd 00e01e90 00000001 00e01f5c badext!doSlow1+0x40 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\extDBG\badext.cpp @ 355]

03 00a1f728 65d82b06 00e01e90 00e01f5c 00e01e90 badext!HttpExtensionProc+0x1aa (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\extDBG\badext.cpp @ 128]

04 00a1f73c 65d82a97 00e01e90 00e01f5c 00e01e90 wam!HSE_APPDLL::ExecuteRequest+0x93 (FPO: [1,0,3]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\wam\object\isplocal.cxx @ 450]

05 00a1f770 65d823a2 0040a8c8 00e02005 00e01e90 wam!WAM::InvokeExtension+0x35 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\wam\object\wamobj.cxx @ 913]

06 00a1f7b4 77d339c0 00406c08 00000000 000000a6 wam!WAM::ProcessRequest+0x240 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\wam\object\wamobj.cxx @ 643]

07 00a1f7dc 77d93570 65d8223f 00a1f7f8 00000005 RPCRT4!Invoke+0x30 [..\i386\stubless.asm @ 88]

08 00a1fa54 77d949ac 000a36f0 000a1604 000cb5d0 RPCRT4!NdrStubCall2+0x63d (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\ndr20\srvcall.c @ 1724]

09 00a1fab8 77b29e7f 000a36f0 000cb5d0 000a1604 RPCRT4!CStdStubBuffer_Invoke+0xec (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\ndr20\stub.c @ 1571]

0a 00a1fafc 77b29d6e 000cb5d0 000c970c 000a2558 ole32!SyncStubInvoke+0x61 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1136]

0b 00a1fb44 77aa7ae7 000cb5d0 000c9620 000a36f0 ole32!StubInvoke+0xa8 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1334]

0c 00a1fba8 77aa7a33 000a1604 00000000 000a36f0 ole32!CCtxComChnl::ContextInvoke+0xbb (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\ctxchnl.cxx @ 1204]

0d 00a1fbc4 77b29c89 000cb5d0 00000001 000a36f0 ole32!MTAInvoke+0x18 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\callctrl.cxx @ 1952]

0e 00a1fbf4 77b29a89 000cb588 000a1604 000a36f0 ole32!AppInvoke+0xb5 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1013]

0f 00a1fcb4 77b2a12c 000cd9b8 00000000 000c96f0 ole32!ComInvokeWithLockAndIPID+0x29e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1578]

10 00a1fcf4 77d33721 000cb588 000c96f0 0007a410 ole32!ThreadInvoke+0x1b7 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 3835]

11 00a1fd2c 77d33667 77b29f7b 0007a410 00a1fe0c RPCRT4!DispatchToStubInC+0x84 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\runtime\mtrt\dispatch.c @ 82]

12 00a1fd84 77d33579 00000000 00000000 00a1fe0c RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x100 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 543]

13 00a1fda4 77d48e81 0007a410 00000000 00a1fe0c RPCRT4!RPC_INTERFACE::DispatchToStub+0x5e (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 407]

14 00a1fdd4 77d34bc6 0007a410 0007a3d4 00000000 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xa9 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 795]

15 00a1fe10 77d346c5 000a5530 000826a0 80030001 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0x1c6 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\lpcsvr.cxx @ 2665]

16 00a1fe28 77d422ff 0007a318 00a1fe50 000a5530 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0x10c (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\lpcsvr.cxx @ 867]

17 00a1ff74 77d420d9 77d42528 000826a0 00000000 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x1eb (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\lpcsvr.cxx @ 1175]

18 00a1ff78 77d42528 000826a0 00000000 00000000 RPCRT4!RecvLotsaCallsWrapper+0x9 (FPO: [1,0,0]) (CONV: stdcall) [D:\nt\private\rpc\runtime\mtrt\lpcsvr.cxx @ 37]

19 00a1ffa8 77d424da 00094ce0 00a1ffec 77e887dd RPCRT4!BaseCachedThreadRoutine+0x4f (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 3586]

1a 00a1ffb4 77e887dd 00094d40 00000000 00000000 RPCRT4!ThreadStartRoutine+0x18 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\runtime\mtrt\threads.cxx @ 230]

1b 00a1ffec 00000000 77d424c2 00094d40 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]

 

With private symbols, note the line of code in frame 02 that calls to "SleepEx" (badext.cpp @ 355]). Lining up the source code shows the line as:

 

DWORD doSlow1(EXTENSION_CONTROL_BLOCK *lpEcb, int Variation)

{

char *szVar = new char[10];

char *szVal = new char[10];

getQSVars (lpEcb, szVar, szVal);

int iTime; iTime = atoi( szVal );

iTime *=1000; //convert to ms since we get the time in sec

if (1 == Variation)

{

//Cause iTime delay

DWORD sl = SleepEx( iTime, FALSE ); // Line 355

DWORD dw = NoScenario(lpEcb);

}

}

 


- Microsoft Confidential -