1) Without symbols
a) Looking for the thread running the request.
The thread condidate in a slow response would most likely contain a call to "ZwDelayExecution()" .
In our example, here is the thread making that call.
4 Id: b18.9b8 Suspend: 1 Teb: 7ffdb000 Unfrozen
# ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00 00a1f6f4 012917ec 00007530 00000000 0040c6a8 ntdll!ZwDelayExecution+0xb
01 00a1f70c 012914cd 00e01e90 00000001 00e01f5c badext!HttpExtensionProc+0x4c9
02 00a1f728 65d82b06 00e01e90 00e01f5c 00e01e90 badext!HttpExtensionProc+0x1aa
03 00a1f770 65d823a2 0040a8c8 00e02005 00e01e90 wam!SE_TABLE::ReleaseExtension+0xbe8
04 00a1f7b4 77d339c0 00406c08 00000000 000000a6 wam!SE_TABLE::ReleaseExtension+0x484
05 00a1f7dc 77d93570 65d8223f 00a1f7f8 00000005 RPCRT4!NdrServerInitialize+0x288
06 00a1fa54 77d949ac 000a36f0 000a1604 000cb5d0 RPCRT4!NdrStubCall2+0x570
07 00a1fab8 77b29e7f 000a36f0 000cb5d0 000a1604 RPCRT4!CStdStubBuffer_Invoke+0x9e
08 00a1fafc 77b29d6e 000cb5d0 000c970c 000a2558 ole32!HDC_UserMarshal+0x78a
09 00a1fb44 77aa7ae7 000cb5d0 000c9620 000a36f0 ole32!HDC_UserMarshal+0x679
0a 00a1fba8 77aa7a33 000a1604 00000000 000a36f0 ole32!CoGetCallContext+0x401
0b 00a1fbc4 77b29c89 000cb5d0 00000001 000a36f0 ole32!CoGetCallContext+0x34d
0c 00a1fbf4 77b29a89 000cb588 000a1604 000a36f0 ole32!HDC_UserMarshal+0x594
0d 00a1fcb4 77b2a12c 000cd9b8 00000000 000c96f0 ole32!HDC_UserMarshal+0x394
0e 00a1fcf4 77d33721 000cb588 000c96f0 0007a410 ole32!HDC_UserMarshal+0xa37
0f 00a1fd2c 77d33667 77b29f7b 0007a410 00a1fe0c RPCRT4!I_RpcFree+0x1ff
10 00a1fd84 77d33579 00000000 00000000 00a1fe0c RPCRT4!I_RpcFree+0x145
11 00a1fda4 77d48e81 0007a410 00000000 00a1fe0c RPCRT4!I_RpcFree+0x57
12 00a1fdd4 77d34bc6 0007a410 0007a3d4 00000000 RPCRT4!RpcSsDestroyClientContext+0xa6
13 00a1fe10 77d346c5 000a5530 000826a0 80030001 RPCRT4!NdrCorrelationFree+0x1120
Basically, since only export symbols are used here, we cannot far in analyzing this thread and finding out why this thread is slow in processing the request.
2) With symbols including public symbols of "badExt.dll"
First, we need to add the symbol path(company wide symbols) as well as public symbols of our dll
Let try in this example to use company wide public symbols
a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols ;C:\IISDebugLabs\symbols\Pub a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols ;C :\IISDebugLabs\symbols\Pub
Note: For more information on setting the symbol path for windbg, please refer to the debugger documentation for the .sympath command. This will describe
the "downstream store" feature that lets you download symbols to a symbol server of your own.
Visit also http://www.microsoft.com/whdc/ddk/debugging/symbols.mspx
Here is how the above thread would look like with public symbols.
4 Id: b18.9b8 Suspend: 1 Teb: 7ffdb000 Unfrozen
# ChildEBP RetAddr Args to Child 00 00a1f6d4 77e87617 00000000 00a1f6ec 0040c6d8 ntdll!NtDelayExecution+0xb
01 00a1f6f4 012917ec 00007530 00000000 0040c6a8 KERNEL32!SleepEx+0x32
02 00a1f70c 012914cd 00e01e90 00000001 00e01f5c badext!doSlow1+0x40
03 00a1f728 65d82b06 00e01e90 00e01f5c 00e01e90 badext!HttpExtensionProc+0x1aa
04 00a1f73c 65d82a97 00e01e90 00e01f5c 00e01e90 wam!HSE_APPDLL::ExecuteRequest+0x93
05 00a1f770 65d823a2 0040a8c8 00e02005 00e01e90 wam!WAM::InvokeExtension+0x35
06 00a1f7b4 77d339c0 00406c08 00000000 000000a6 wam!WAM::ProcessRequest+0x240
07 00a1f7dc 77d93570 65d8223f 00a1f7f8 00000005 RPCRT4!Invoke+0x30
08 00a1fa54 77d949ac 000a36f0 000a1604 000cb5d0 RPCRT4!NdrStubCall2+0x63d
09 00a1fab8 77b29e7f 000a36f0 000cb5d0 000a1604 RPCRT4!CStdStubBuffer_Invoke+0xec
0a 00a1fafc 77b29d6e 000cb5d0 000c970c 000a2558 ole32!SyncStubInvoke+0x61
0b 00a1fb44 77aa7ae7 000cb5d0 000c9620 000a36f0 ole32!StubInvoke+0xa8
0c 00a1fba8 77aa7a33 000a1604 00000000 000a36f0 ole32!CCtxComChnl::ContextInvoke+0xbb
0d 00a1fbc4 77b29c89 000cb5d0 00000001 000a36f0 ole32!MTAInvoke+0x18
0e 00a1fbf4 77b29a89 000cb588 000a1604 000a36f0 ole32!AppInvoke+0xb5
0f 00a1fcb4 77b2a12c 000cd9b8 00000000 000c96f0 ole32!ComInvokeWithLockAndIPID+0x29e
10 00a1fcf4 77d33721 000cb588 000c96f0 0007a410 ole32!ThreadInvoke+0x1b7
11 00a1fd2c 77d33667 77b29f7b 0007a410 00a1fe0c RPCRT4!DispatchToStubInC+0x84
12 00a1fd84 77d33579 00000000 00000000 00a1fe0c RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x100
13 00a1fda4 77d48e81 0007a410 00000000 00a1fe0c RPCRT4!RPC_INTERFACE::DispatchToStub+0x5e
14 00a1fdd4 77d34bc6 0007a410 0007a3d4 00000000 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xa9
15 00a1fe10 77d346c5 000a5530 000826a0 80030001 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0x1c6
16 00a1fe28 77d422ff 0007a318 00a1fe50 000a5530 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0x10c
17 00a1ff74 77d420d9 77d42528 000826a0 00000000 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x1eb
18 00a1ff78 77d42528 000826a0 00000000 00000000 RPCRT4!RecvLotsaCallsWrapper+0x9
19 00a1ffa8 77d424da 00094ce0 00a1ffec 77e887dd RPCRT4!BaseCachedThreadRoutine+0x4f
1a 00a1ffb4 77e887dd 00094d40 00000000 00000000 RPCRT4!ThreadStartRoutine+0x18
1b 00a1ffec 00000000 77d424c2 00094d40 00000000 KERNEL32!BaseThreadStart+0x52
This thread shows that the main entry in our ISAPI extension (HttpExtensionProc" is calling the function "doSlow1()" which in turn is calling the win32 API "SleepEx".
The first parameter passed to "sleepEx" is the time interval in Milliseconds (Hex Format) for which thread execution is to be suspended.
To evaluate this parameter, do the following:
0:015> ? 00007530
Evaluate expression: 30000 = 00007530
This shows that the execution of this thread is suspended for 30,000 ms = 30 sec. This is why the response is slow.
To verify that the symbols are lined up properly for our extension, issue this command.
0:006> lmv mbadext
start end module name 06470000 06475000
badext (pdb symbols) C :\IISDebugLabs\symbols\Pub
Loaded symbol image file: C:\IISDebuglabs\www\App\badext.dll Image path: C:\IISDebuglabs\www\App\badext.dll
Timestamp: Wed Jan 29 18:07:14 2003 (3E385EA2)
Checksum: 0000EBCD
File version: 5.0.0.0
Product version: 5.0.0.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
CompanyName: Microsoft Corporation
ProductName: Internet Information Services
InternalName: badEXT.dll
OriginalFilename: badEXT.dll
ProductVersion: 5.00.0984
FileVersion: 5.00.0984
FileDescription: badEXT Test
LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999
3) with Private Symbols "badExt.dll"
First, we need to add the symbol path(company wide symbols) as well as private symbols of our dll a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pri a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pri
The stack backtrace of the thread that should process the request looks like:
4 Id: b18.9b8 Suspend: 1 Teb: 7ffdb000 Unfrozen
# ChildEBP RetAddr Args to Child 00 00a1f6d4 77e87617 00000000 00a1f6ec 0040c6d8 ntdll!NtDelayExecution+0xb (FPO: [2,0,0]) [i386\usrstubs.asm @ 532]
01 00a1f6f4 012917ec 00007530 00000000 0040c6a8 KERNEL32!SleepEx+0x32 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\synch.c @ 1672]
02 00a1f70c 012914cd 00e01e90 00000001 00e01f5c badext!doSlow1+0x40 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\extDBG\badext.cpp @ 355]
03 00a1f728 65d82b06 00e01e90 00e01f5c 00e01e90 badext!HttpExtensionProc+0x1aa (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\extDBG\badext.cpp @ 128]
04 00a1f73c 65d82a97 00e01e90 00e01f5c 00e01e90 wam!HSE_APPDLL::ExecuteRequest+0x93 (FPO: [1,0,3]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\wam\object\isplocal.cxx @ 450]
05 00a1f770 65d823a2 0040a8c8 00e02005 00e01e90 wam!WAM::InvokeExtension+0x35 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\wam\object\wamobj.cxx @ 913]
06 00a1f7b4 77d339c0 00406c08 00000000 000000a6 wam!WAM::ProcessRequest+0x240 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\wam\object\wamobj.cxx @ 643]
07 00a1f7dc 77d93570 65d8223f 00a1f7f8 00000005 RPCRT4!Invoke+0x30 [..\i386\stubless.asm @ 88]
08 00a1fa54 77d949ac 000a36f0 000a1604 000cb5d0 RPCRT4!NdrStubCall2+0x63d (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\ndr20\srvcall.c @ 1724]
09 00a1fab8 77b29e7f 000a36f0 000cb5d0 000a1604 RPCRT4!CStdStubBuffer_Invoke+0xec (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\ndr20\stub.c @ 1571]
0a 00a1fafc 77b29d6e 000cb5d0 000c970c 000a2558 ole32!SyncStubInvoke+0x61 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1136]
0b 00a1fb44 77aa7ae7 000cb5d0 000c9620 000a36f0 ole32!StubInvoke+0xa8 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1334]
0c 00a1fba8 77aa7a33 000a1604 00000000 000a36f0 ole32!CCtxComChnl::ContextInvoke+0xbb (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\ctxchnl.cxx @ 1204]
0d 00a1fbc4 77b29c89 000cb5d0 00000001 000a36f0 ole32!MTAInvoke+0x18 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\callctrl.cxx @ 1952]
0e 00a1fbf4 77b29a89 000cb588 000a1604 000a36f0 ole32!AppInvoke+0xb5 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1013]
0f 00a1fcb4 77b2a12c 000cd9b8 00000000 000c96f0 ole32!ComInvokeWithLockAndIPID+0x29e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1578]
10 00a1fcf4 77d33721 000cb588 000c96f0 0007a410 ole32!ThreadInvoke+0x1b7 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 3835]
11 00a1fd2c 77d33667 77b29f7b 0007a410 00a1fe0c RPCRT4!DispatchToStubInC+0x84 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\runtime\mtrt\dispatch.c @ 82]
12 00a1fd84 77d33579 00000000 00000000 00a1fe0c RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x100 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 543]
13 00a1fda4 77d48e81 0007a410 00000000 00a1fe0c RPCRT4!RPC_INTERFACE::DispatchToStub+0x5e (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 407]
14 00a1fdd4 77d34bc6 0007a410 0007a3d4 00000000 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xa9 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 795]
15 00a1fe10 77d346c5 000a5530 000826a0 80030001 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0x1c6 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\lpcsvr.cxx @ 2665]
16 00a1fe28 77d422ff 0007a318 00a1fe50 000a5530 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0x10c (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\lpcsvr.cxx @ 867]
17 00a1ff74 77d420d9 77d42528 000826a0 00000000 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x1eb (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\lpcsvr.cxx @ 1175]
18 00a1ff78 77d42528 000826a0 00000000 00000000 RPCRT4!RecvLotsaCallsWrapper+0x9 (FPO: [1,0,0]) (CONV: stdcall) [D:\nt\private\rpc\runtime\mtrt\lpcsvr.cxx @ 37]
19 00a1ffa8 77d424da 00094ce0 00a1ffec 77e887dd RPCRT4!BaseCachedThreadRoutine+0x4f (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 3586]
1a 00a1ffb4 77e887dd 00094d40 00000000 00000000 RPCRT4!ThreadStartRoutine+0x18 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\runtime\mtrt\threads.cxx @ 230]
1b 00a1ffec 00000000 77d424c2 00094d40 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]
With private symbols, note the line of code in frame 02 that calls to "SleepEx" (badext.cpp @ 355]). Lining up the source code shows the line as:
DWORD doSlow1(EXTENSION_CONTROL_BLOCK *lpEcb, int Variation)
{
char *szVar = new char[10];
char *szVal = new char[10];
getQSVars (lpEcb, szVar, szVal);
int iTime; iTime = atoi( szVal );
iTime *=1000; //convert to ms since we get the time in sec
if (1 == Variation)
{
//Cause iTime delay
DWORD sl = SleepEx( iTime, FALSE ); // Line 355
DWORD dw = NoScenario(lpEcb);
}
}