1) Without symbols
a) Looking for the "Slow" thread.
The thread condidate in a slow response would most likely contain a call to "ZwDelayExecution()" .
In our example, here is the thread making that call.
22 Id: 458.5bc Suspend: 1 Teb: 7ffa4000 Unfrozen
*** ERROR: Symbol file could not be found. Defaulted to export symbols for c:\IISDEB~1\www\app\badcom.dll -
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
01e8f0cc 02032ec6 00004e20 00000000 0040e0a0 ntdll!ZwDelayExecution+0xb
01e8f0e4 02032a28 00000001 6c733d73 5f31776f badcom!DllUnregisterServer+0x1149
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINNT\system32\OLEAUT32.dll -
01e8f124 779d7b9d 0040bfd8 00408a60 00000000 badcom!DllUnregisterServer+0xcab
01e8f140 77a22c42 0040bfd8 00000024 00000004 OLEAUT32!DispCallFunc+0x15d
01e8f1d0 020325d0 000c0ec4 0040bfd8 00000003 OLEAUT32!ClearCustData+0x588
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINNT\System32\vbscript.dll -
01e8f1f8 6b61dff3 0040bfd8 00000003 6b659388 badcom!DllUnregisterServer+0x853
01e8f274 6b62171a 004085e8 0040bfd8 00000003 vbscript!DllRegisterServer+0x844c
01e8f954 01e8b0a0 0040dc48 00000000 7800ffb8 vbscript!DllRegisterServer+0xbb73
00000000 00000000 00000000 00000000 00000000 0x1e8b0a0
Basically, since only export symbols are used here, we cannot go far in analyzing this thread and finding out why this thread is slow in processing the request.
On the other hand, we need to double check if the call to ZwDelayExecution() is the right one by lining up the public symbols (or private)
2) With symbols including public symbols of "badCom.dll"
First, we need to add the private symbol path(company wide symbols) as well as public symbols of our dll a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pub a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pub
Here is how the above thread would look like with public symbols.
0:022> kvn50
# ChildEBP RetAddr Args to Child
00 01e8f0ac 77e87617 00000000 01e8f0c4 0040e0d0 ntdll!NtDelayExecution+0xb (FPO: [2,0,0])
01 01e8f0cc 02032ec6 00004e20 00000000 0040e0a0 KERNEL32!SleepEx+0x32 (FPO: [Non-Fpo])
02 01e8f0e4 02032a28 00000001 6c733d73 5f31776f badcom!doSlow1+0x3d
03 01e8f124 779d7b9d 0040bfd8 00408a60 00000000 badcom!CScenarios::processSc+0x1de
04 01e8f140 77a22c42 0040bfd8 00000024 00000004 OLEAUT32!DispCallFunc+0x15d
05 01e8f1d0 020325d0 000c0ec4 0040bfd8 00000003 OLEAUT32!CTypeInfo2::Invoke+0x244
06 01e8f1f8 6b61dff3 0040bfd8 00000003 6b659388 badcom!ATL::IDispatchImpl::Invoke+0x41
07 01e8f274 6b62171a 004085e8 0040bfd8 00000003 vbscript!CatchIDispatchInvoke+0x9e
08 01e8f384 6b604038 0009001f 0000023f 004084e0 vbscript!InvokeDispatch+0x330 (FPO: [EBP 0x6b659388] [7,60,4])
09 01e8f3c8 6b603156 004084e0 0040bfd8 00000003 vbscript!GetDispatchDispID+0x1cc (FPO: [EBP 0x01e8f954] [4,3,4])
0a 01e8f970 78001045 00000000 0040b66c 00000009 vbscript!CScriptRuntime::Run+0x3b8d
0b 01e8f998 6b601ef7 00000009 01e8fa8c 00000000 MSVCRT!_nh_malloc+0x10 (FPO: [2,0,0])
0c 01e8f9c8 6b601c5c 00000000 00000000 00000000 vbscript!CScriptRuntime::Init+0x2e1 (FPO: [EBP 0x6b658a10] [6,0,4])
0d 01e8fb08 6b606fc5 004084e0 6b606ef1 004087e0 vbscript!CScriptEntryPoint::Call+0x76 (FPO: [5,58,2])
0e 01e8fb54 749f2e70 01911e90 6b610c9a 00000000 vbscript!CSession::GetGlobalEntryPoint+0x25 (FPO: [3,0,0])
0f 01e8fb6c 749f3af0 01912060 01912060 012d1e90 asp!CResponse::Release+0x18 (FPO: [1,0,1])
10 01e8fb9c 74a22f5b 00000000 74a32c70 77e873a1 asp!CActiveScriptEngine::TryCall+0x15 (FPO: [Non-Fpo])
11 01e8fbe0 749f3a72 00000000 01e8fc28 749f593d asp!CActiveScriptEngine::Call+0x10c (FPO: [Non-Fpo])
12 01e8fbec 749f593d 01e8fc4c 00000000 00000000 asp!CallScriptFunctionOfEngine+0x48 (FPO: [Non-Fpo])
13 01e8fc28 749f54c8 01a11fa0 01591e90 01e8fc94 asp!ExecuteRequest+0x19e (FPO: [Non-Fpo])
14 01e8fc70 749fcf9b 01a11fa0 01591e90 01e8fc94 asp!Execute+0x17e (FPO: [Non-Fpo])
15 01e8fcc0 749fcc37 00000000 000be728 01e8fd58 asp!CHitObj::ViperAsyncCallback+0x452 (FPO: [Non-Fpo])
16 01e8fcd8 787c214a 00000000 77acd455 01b51e90 asp!CViperAsyncRequest::OnCall+0x3f (FPO: [Non-Fpo])
17 01e8fce0 77acd455 01b51e90 77aa71a0 00000000 COMSVCS!STAActivityWorkHelper+0xa (FPO: [1,0,0])
18 01e8fd24 77acd318 00075280 00085850 787c2140 ole32!EnterForCallback+0x6c (FPO: [Non-Fpo])
19 01e8fe50 77acd1dd 00085850 787c2140 01b51e90 ole32!SwitchForCallback+0x12d (FPO: [Non-Fpo])
1a 01e8fe78 77aac61a 00085850 787c2140 01b51e90 ole32!PerformCallback+0x52 (FPO: [Non-Fpo])
1b 01e8fed4 77ad1185 000757c0 787c2140 01b51e90 ole32!CObjectContext::InternalContextCallback+0xf7 (FPO: [Non-Fpo])
1c 01e8fef4 787beff3 000757c0 787c2140 01b51e90 ole32!CObjectContext::DoCallback+0x1c (FPO: [Non-Fpo])
1d 01e8ff24 787c11ad 00010007 00000000 01e8ffec COMSVCS!STAActivityWork::DoWork+0x73 (FPO: [0,4,2])
1e 01e8ffb4 77e887dd 00b1f364 00010007 00000000 COMSVCS!STAThread::STAThreadWorker+0x465 (FPO: [EBP 0x01e8ffec] [1,31,4])
1f 01e8ffec 00000000 787c0d48 00b1f364 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo])
This thread shows that the function "processSc() in the class CScenarios is calling the function "doSlow1()" which in turn is calling the win32 API "SleepEx".
The first parameter passed to "sleepEx" is the time interval in Milliseconds (Hex Format) for which thread execution is to be suspended.
To evaluate this parameter, do the following:
0:022> ? 00004e20
Evaluate expression: 20000 = 00004e20
This shows that the execution of this thread is suspended for 20,000 ms = 20 sec. This is why the response is slow.
To verify that the symbols are lined up properly for our extension, issue this command.
0:020> lmv mbadcom
start end module name
02030000 02038000 badcom (pdb symbols) c:\iisdebuglabs\symbols\pub\badcom.pdb
Loaded symbol image file: badcom.dll
Image path: c:\IISDEB~1\www\app\badcom.dll
Timestamp: Tue Jun 10 22:45:51 2003 (3EE697DF) Checksum: 0000F229
File version: 1.0.0.1
Product version: 1.0.0.1
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
CompanyName:
ProductName: badCOM
Module InternalName: badCOM
OriginalFilename: badCOM.DLL
ProductVersion: 1, 0, 0, 1
FileVersion: 1, 0, 0, 1
FileDescription: badCOM Module
LegalCopyright: Copyright 2003
3) with Private Symbols "badCom.dll"
First, we need to add the private symbol path(company wide symbols) as well as private symbols of our dll a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pri a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pri
The stack backtrace of the above thread would look like:
0:022> kvn50
# ChildEBP RetAddr Args to Child
00 01e8f0ac 77e87617 00000000 01e8f0c4 0040e0d0 ntdll!NtDelayExecution+0xb (FPO: [2,0,0]) [i386\usrstubs.asm @ 532]
01 01e8f0cc 02032ec6 00004e20 00000000 0040e0a0 KERNEL32!SleepEx+0x32 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\synch.c @ 1672]
02 01e8f0e4 02032a28 00000001 6c733d73 5f31776f badcom!doSlow1+0x3d (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\comDBG\scenarios.cpp @ 401]
03 01e8f124 779d7b9d 0040bfd8 00408a60 00000000 badcom!CScenarios::processSc+0x1de (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\comDBG\scenarios.cpp @ 160]
04 01e8f140 77a22c42 0040bfd8 00000024 00000004 OLEAUT32!tPushValJmpTab+0xf5 [D:\OA\src\dispatch\WIN32\i386\invoke.asm @ 340]
05 01e8f1d0 020325d0 000c0ec4 0040bfd8 00000003 OLEAUT32!CTypeInfo2::Invoke+0x244 (CONV: stdcall) [D:\OA\src\typelib2\tiinvoke.cpp @ 629]
06 01e8f1f8 6b61dff3 0040bfd8 00000003 6b659388 badcom!ATL::IDispatchImpl::Invoke+0x41 (CONV: stdcall) [c:\nt\public\sdk\inc\atl30\atlcom.h @ 4093]
07 01e8f274 6b62171a 004085e8 0040bfd8 00000003 vbscript!CatchIDispatchInvoke+0x9e (CONV: stdcall) [E:\vbscript\src\core\callout.h @ 111]
08 01e8f384 6b604038 0009001f 0000023f 004084e0 vbscript!InvokeDispatch+0x330 (FPO: [EBP 0x6b659388] [7,60,4]) (CONV: stdcall) [E:\vbscript\src\vbscript\var.cpp @ 1211]
09 01e8f3c8 6b603156 004084e0 0040bfd8 00000003 vbscript!GetDispatchDispID+0x1cc (FPO: [EBP 0x01e8f954] [4,3,4]) (CONV: stdcall) [E:\vbscript\src\vbscript\var.cpp @ 972]
0a 01e8f970 78001045 00000000 0040b66c 00000009 vbscript!CScriptRuntime::Run+0x3b8d (CONV: thiscall) [E:\vbscript\src\vbscript\interp.cpp @ 3347]
0b 01e8f998 6b601ef7 00000009 01e8fa8c 00000000 MSVCRT!_nh_malloc+0x10 (FPO: [2,0,0]) (CONV: cdecl) [malloc.c @ 113]
0c 01e8f9c8 6b601c5c 00000000 00000000 00000000 vbscript!CScriptRuntime::Init+0x2e1 (FPO: [EBP 0x6b658a10] [6,0,4]) (CONV: thiscall) [E:\vbscript\src\vbscript\interp.cpp @ 904]
0d 01e8fb08 6b606fc5 004084e0 6b606ef1 004087e0 vbscript!CScriptEntryPoint::Call+0x76 (FPO: [5,58,2]) (CONV: thiscall) [E:\vbscript\src\vbscript\entrypnt.cpp @ 40]
0e 01e8fb54 749f2e70 01911e90 6b610c9a 00000000 vbscript!CSession::GetGlobalEntryPoint+0x25 (FPO: [3,0,0]) (CONV: thiscall) [E:\vbscript\src\core\wrapper.cpp @ 485]
0f 01e8fb6c 749f3af0 01912060 01912060 012d1e90 asp!CResponse::Release+0x18 (FPO: [1,0,1]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\response.cpp @ 1716]
10 01e8fb9c 74a22f5b 00000000 74a32c70 77e873a1 asp!CActiveScriptEngine::TryCall+0x15 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\scrptmgr.cpp @ 2040]
11 01e8fbe0 749f3a72 00000000 01e8fc28 749f593d asp!CActiveScriptEngine::Call+0x10c (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\scrptmgr.cpp @ 2005]
12 01e8fbec 749f593d 01e8fc4c 00000000 00000000 asp!CallScriptFunctionOfEngine+0x48 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 1394]
13 01e8fc28 749f54c8 01a11fa0 01591e90 01e8fc94 asp!ExecuteRequest+0x19e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 326]
14 01e8fc70 749fcf9b 01a11fa0 01591e90 01e8fc94 asp!Execute+0x17e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 190]
15 01e8fcc0 749fcc37 00000000 000be728 01e8fd58 asp!CHitObj::ViperAsyncCallback+0x452 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\hitobj.cpp @ 2487]
16 01e8fcd8 787c214a 00000000 77acd455 01b51e90 asp!CViperAsyncRequest::OnCall+0x3f (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\viperint.cpp @ 194]
17 01e8fce0 77acd455 01b51e90 77aa71a0 00000000 COMSVCS!STAActivityWorkHelper+0xa (FPO: [1,0,0]) (CONV: stdcall) [.\stathred.cpp @ 1321]
18 01e8fd24 77acd318 00075280 00085850 787c2140 ole32!EnterForCallback+0x6a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1758]
19 01e8fe50 77acd1dd 00085850 787c2140 01b51e90 ole32!SwitchForCallback+0x12b (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1643]
1a 01e8fe78 77aac61a 00085850 787c2140 01b51e90 ole32!PerformCallback+0x50 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1558]
1b 01e8fed4 77ad1185 000757c0 787c2140 01b51e90 ole32!CObjectContext::InternalContextCallback+0xf5 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\context.cxx @ 3942]
1c 01e8fef4 787beff3 000757c0 787c2140 01b51e90 ole32!CObjectContext::DoCallback+0x1a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\context.cxx @ 3798]
1d 01e8ff24 787c11ad 00010007 00000000 01e8ffec COMSVCS!STAActivityWork::DoWork+0x73 (FPO: [0,4,2]) (CONV: thiscall) [..\inc\STAThred.h @ 340]
1e 01e8ffb4 77e887dd 00b1f364 00010007 00000000 COMSVCS!STAThread::STAThreadWorker+0x465 (FPO: [EBP 0x01e8ffec] [1,31,4]) (CONV: stdcall) [.\stathred.cpp @ 628]
1f 01e8ffec 00000000 787c0d48 00b1f364 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]
With private symbols, note the line of code in frame 02 that calls to "SleepEx" (Scenarios.cpp @ 401]). Lining up the source code shows the line as:
DWORD doSlow1(int Variation)
{
char *szVar = new char[10];
char *szVal = new char[10];
getQSVars (szVar, szVal);
int iTime; iTime = atoi( szVal );
iTime *=1000; //convert to ms since we get the time in sec
if (1 == Variation)
{
//Cause iTime delay
DWORD sl = SleepEx( iTime, FALSE ); // Line 401
}
return S_OK;
}