1) Without symbols
a) Finding the thread(s) causing High CPU using performance Monitor
In performance Monitor:
a- Get the thread ID(s) of the threads averaging high CPU.
b- Convert the thread ID(s) from Decimal to Hexadecimal.
In Windbg:
a- Search for the thread ID(s) in step (b).
In our example:
Looking in Performance Monitor, Process dllhost#1 (assuming it is the only dllhost process you have asside from the system package) is averaging 100% CPU. Adding all threads of that
process, try identify the thread(s) that have the same (or almost) shape as the main process as far as %CPU time is concerned.
Thread ID 324 is the one that hasthe same shape as the process dllhost#1, this means that the process CPU time is taken by this thread.
324 = 0x144
Looking for thread ID 144 in the debugger, it looks like:
0:030> ~~[144 ]kvn50
*** ERROR: Symbol file could not be found. Defaulted to export symbols for c:\IISDEB~1\www\app\badcom.dll -
# ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00 01e4f0f4 020329ee 00000001 70633d73 765f3175 badcom!DllUnregisterServer+0x10d8
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINNT\system32\OLEAUT32.dll -
01 01e4f124 779d7b9d 0040bfd8 00408a60 00000000 badcom!DllUnregisterServer+0xc71
02 01e4f140 77a22c42 0040bfd8 00000024 00000004 OLEAUT32!DispCallFunc+0x15d
03 01e4f1d0 020325d0 000c0884 0040bfd8 00000003 OLEAUT32!ClearCustData+0x588
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINNT\System32\vbscript.dll -
04 01e4f1f8 6b61dff3 0040bfd8 00000003 6b659388 badcom!DllUnregisterServer+0x853
05 01e4f274 6b62171a 004085e8 0040bfd8 00000003 vbscript!DllRegisterServer+0x844c
06 01e4f954 ffffffff 0040b140 00000000 004001a0 vbscript!DllRegisterServer+0xbb73
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Program Files\Common Files\Microsoft Shared\VS7Debug\pdm.dll -
07 77f80000 00000003 00000004 0000ffff 000000b8 0xffffffff
Basically this is enough to tell the customer that the extension "badCom.dll" is the one causing this problem of high CPU usage
b) Finding the thread(s) causing High CPU using "!runaway"
The !runaway debugger extension shows information about the time taken by each thread. Note that this command is used only in a live debug session, you cannot use it if you
are analyzing a crash dump.
0:030> !runaway
User Mode Time
Thread Time
144 0:00:20.0339
710 0:00:00.0050
504 0:00:00.0030
69c 0:00:00.0010
788 0:00:00.0010
5d4 0:00:00.0000
3f8 0:00:00.0000
6d8 0:00:00.0000
4b4 0:00:00.0000
6b4 0:00:00.0000
13c 0:00:00.0000
140 0:00:00.0000
14c 0:00:00.0000
148 0:00:00.0000
7e0 0:00:00.0000
6c8 0:00:00.0000
534 0:00:00.0000
7cc 0:00:00.0000
458 0:00:00.0000
780 0:00:00.0000
5a0 0:00:00.0000
43c 0:00:00.0000
79c 0:00:00.0000
7a0 0:00:00.0000
6c4 0:00:00.0000
678 0:00:00.0000
45c 0:00:00.0000
800 0:00:00.0000
7ac 0:00:00.0000
6d4 0:00:00.0000
7b8 0:00:00.0000
2) With symbols including public symbols of "badCom.dll"
First, we need to add the private symbol path(company wide symbols) as well as public symbols of our dll a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pub a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pub
The stack backtrace of the thread that consumes high CPU looks like:
0:030> ~~[144]kvn50
# ChildEBP RetAddr Args to Child 00 01e4f0f4 020329ee 00000001 70633d73 765f3175 badcom!doCPU1+0x25
01 01e4f124 779d7b9d 0040bfd8 00408a60 00000000 badcom!CScenarios::processSc+0x1a4
02 01e4f140 77a22c42 0040bfd8 00000024 00000004 OLEAUT32!tPushValJmpTab+0xf5 [D:\OA\src\dispatch\WIN32\i386\invoke.asm @ 340]
03 01e4f1d0 020325d0 000c0884 0040bfd8 00000003 OLEAUT32!CTypeInfo2::Invoke+0x244 (CONV: stdcall) [D:\OA\src\typelib2\tiinvoke.cpp @ 629]
04 01e4f1f8 6b61dff3 0040bfd8 00000003 6b659388 badcom!ATL::IDispatchImpl::Invoke+0x41
05 01e4f274 6b62171a 004085e8 0040bfd8 00000003 vbscript!CatchIDispatchInvoke+0x9e (CONV: stdcall) [E:\vbscript\src\core\callout.h @ 111]
06 01e4f384 6b604038 0009001f 0000023f 004084e0 vbscript!InvokeDispatch+0x330 (FPO: [EBP 0x6b659388] [7,60,4]) (CONV: stdcall) [E:\vbscript\src\vbscript\var.cpp @ 1211]
07 01e4f3c8 6b603156 004084e0 0040bfd8 00000003 vbscript!GetDispatchDispID+0x1cc (FPO: [EBP 0x01e4f954] [4,3,4]) (CONV: stdcall) [E:\vbscript\src\vbscript\var.cpp @ 972]
08 01e4f998 6b601ef7 00000009 01e4fa8c 00000000 vbscript!CScriptRuntime::Run+0x3b8d (CONV: thiscall) [E:\vbscript\src\vbscript\interp.cpp @ 3347]
09 01e4f9c8 6b601c5c 00000000 00000000 00000000 vbscript!CScriptRuntime::Init+0x2e1 (FPO: [EBP 0x6b658a10] [6,0,4]) (CONV: thiscall) [E:\vbscript\src\vbscript\interp.cpp @ 904]
0a 01e4fb08 6b606fc5 004084e0 6b606ef1 00408bd8 vbscript!CScriptEntryPoint::Call+0x76 (FPO: [5,58,2]) (CONV: thiscall) [E:\vbscript\src\vbscript\entrypnt.cpp @ 40]
0b 01e4fb54 749f2e70 01911e90 6b610c9a 00000000 vbscript!CSession::GetGlobalEntryPoint+0x25 (FPO: [3,0,0]) (CONV: thiscall) [E:\vbscript\src\core\wrapper.cpp @ 485]
0c 01e4fb6c 749f3af0 01912060 01912060 012d1e90 asp!CResponse::Release+0x18 (FPO: [1,0,1]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\response.cpp @ 1716]
0d 01e4fb9c 74a22f5b 00000000 74a32c70 77e873a1 asp!CActiveScriptEngine::TryCall+0x15 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\scrptmgr.cpp @ 2040]
0e 01e4fbe0 749f3a72 00000000 01e4fc28 749f593d asp!CActiveScriptEngine::Call+0x10c (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\scrptmgr.cpp @ 2005]
0f 01e4fbec 749f593d 01e4fc4c 00000000 00000000 asp!CallScriptFunctionOfEngine+0x48 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 1394]
10 01e4fc28 749f54c8 01a11fa0 01591e90 01e4fc94 asp!ExecuteRequest+0x19e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 326]
11 01e4fc70 749fcf9b 01a11fa0 01591e90 01e4fc94 asp!Execute+0x17e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 190]
12 01e4fcc0 749fcc37 00000000 000bdd10 01e4fd58 asp!CHitObj::ViperAsyncCallback+0x452 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\hitobj.cpp @ 2487]
13 01e4fcd8 787c214a 00000000 77acd455 01b51e90 asp!CViperAsyncRequest::OnCall+0x3f (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\viperint.cpp @ 194]
14 01e4fce0 77acd455 01b51e90 77aa71a0 00000000 COMSVCS!STAActivityWorkHelper+0xa (FPO: [1,0,0]) (CONV: stdcall) [.\stathred.cpp @ 1321]
15 01e4fd24 77acd318 000751d8 00085850 787c2140 ole32!EnterForCallback+0x6a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1758]
16 01e4fe50 77acd1dd 00085850 787c2140 01b51e90 ole32!SwitchForCallback+0x12b (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1643]
17 01e4fe78 77aac61a 00085850 787c2140 01b51e90 ole32!PerformCallback+0x50 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1558]
18 01e4fed4 77ad1185 000757c0 787c2140 01b51e90 ole32!CObjectContext::InternalContextCallback+0xf5 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\context.cxx @ 3942]
19 01e4fef4 787beff3 000757c0 787c2140 01b51e90 ole32!CObjectContext::DoCallback+0x1a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\context.cxx @ 3798]
1a 01e4ff24 787c11ad 00010007 00000000 01e4ffec COMSVCS!STAActivityWork::DoWork+0x73 (FPO: [0,4,2]) (CONV: thiscall) [..\inc\STAThred.h @ 340]
1b 01e4ffb4 77e887dd 00b1f364 00010007 00000000 COMSVCS!STAThread::STAThreadWorker+0x465 (FPO: [EBP 0x01e4ffec] [1,31,4]) (CONV: stdcall) [.\stathred.cpp @ 628]
1c 01e4ffec 00000000 787c0d48 00b1f364 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]
Advanced Technique
Disassembly code
0:021> u eip -6
badcom!doCPU1+1f [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\comDBG\scenarios.cpp @ 356]:
02032e4f 3bc7 cmp eax,edi
02032e51 7706 ja badcom!doCPU1+0x29 (02032e59)
02032e53 ffd6 call esi
02032e55 2bc3 sub eax,ebx
02032e57 ebf6 jmp badcom!doCPU1+0x1f (02032e4f)
02032e59 5f pop edi
02032e5a 5e pop esi
02032e5b 33c0 xor eax,eax
This looks like a loop in the extension code where we jump to 02032e4f any time the value in eax is less than the value in edi (cmp eax,edi ). If the eax is greather than edi, we
will be at 02032e59 which is "pop ebx"
To verify that the symbols are lined up properly for our extension, issue this command.
0:020> lmv mbadcom
start end module name
02030000 02038000 badcom (pdb symbols) c:\iisdebuglabs\symbols\pub\badcom.pdb
Loaded symbol image file: badcom.dll
Image path: c:\IISDEB~1\www\app\badcom.dll
Timestamp: Tue Jun 10 22:45:51 2003 (3EE697DF) Checksum: 0000F229
File version: 1.0.0.1
Product version: 1.0.0.1
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
CompanyName:
ProductName: badCOM
Module InternalName: badCOM
OriginalFilename: badCOM.DLL
ProductVersion: 1, 0, 0, 1
FileVersion: 1, 0, 0, 1
FileDescription: badCOM Module
LegalCopyright: Copyright 2003
3) with Private Symbols "badCom.dll"
First, we need to add the private symbol path(company wide symbols) as well as private symbols of our dll a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pri a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pri
The stack backtrace of the thread that consumes high CPU looks like:
0:030> ~~[144]kvn50
# ChildEBP RetAddr Args to Child 00 01e4f0f4 020329ee 00000001 70633d73 765f3175 badcom!doCPU1+0x25 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\comDBG\scenarios.cpp @ 356]
01 01e4f124 779d7b9d 0040bfd8 00408a60 00000000 badcom!CScenarios::processSc+0x1a4 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\comDBG\scenarios.cpp @ 152]
02 01e4f140 77a22c42 0040bfd8 00000024 00000004 OLEAUT32!tPushValJmpTab+0xf5 [D:\OA\src\dispatch\WIN32\i386\invoke.asm @ 340]
03 01e4f1d0 020325d0 000c0884 0040bfd8 00000003 OLEAUT32!CTypeInfo2::Invoke+0x244 (CONV: stdcall) [D:\OA\src\typelib2\tiinvoke.cpp @ 629]
04 01e4f1f8 6b61dff3 0040bfd8 00000003 6b659388 badcom!ATL::IDispatchImpl::Invoke+0x41 (CONV: stdcall) [c:\nt\public\sdk\inc\atl30\atlcom.h @ 4093]
05 01e4f274 6b62171a 004085e8 0040bfd8 00000003 vbscript!CatchIDispatchInvoke+0x9e (CONV: stdcall) [E:\vbscript\src\core\callout.h @ 111]
06 01e4f384 6b604038 0009001f 0000023f 004084e0 vbscript!InvokeDispatch+0x330 (FPO: [EBP 0x6b659388] [7,60,4]) (CONV: stdcall) [E:\vbscript\src\vbscript\var.cpp @ 1211]
07 01e4f3c8 6b603156 004084e0 0040bfd8 00000003 vbscript!GetDispatchDispID+0x1cc (FPO: [EBP 0x01e4f954] [4,3,4]) (CONV: stdcall) [E:\vbscript\src\vbscript\var.cpp @ 972]
08 01e4f998 6b601ef7 00000009 01e4fa8c 00000000 vbscript!CScriptRuntime::Run+0x3b8d (CONV: thiscall) [E:\vbscript\src\vbscript\interp.cpp @ 3347]
09 01e4f9c8 6b601c5c 00000000 00000000 00000000 vbscript!CScriptRuntime::Init+0x2e1 (FPO: [EBP 0x6b658a10] [6,0,4]) (CONV: thiscall) [E:\vbscript\src\vbscript\interp.cpp @ 904]
0a 01e4fb08 6b606fc5 004084e0 6b606ef1 00408bd8 vbscript!CScriptEntryPoint::Call+0x76 (FPO: [5,58,2]) (CONV: thiscall) [E:\vbscript\src\vbscript\entrypnt.cpp @ 40]
0b 01e4fb54 749f2e70 01911e90 6b610c9a 00000000 vbscript!CSession::GetGlobalEntryPoint+0x25 (FPO: [3,0,0]) (CONV: thiscall) [E:\vbscript\src\core\wrapper.cpp @ 485]
0c 01e4fb6c 749f3af0 01912060 01912060 012d1e90 asp!CResponse::Release+0x18 (FPO: [1,0,1]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\response.cpp @ 1716]
0d 01e4fb9c 74a22f5b 00000000 74a32c70 77e873a1 asp!CActiveScriptEngine::TryCall+0x15 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\scrptmgr.cpp @ 2040]
0e 01e4fbe0 749f3a72 00000000 01e4fc28 749f593d asp!CActiveScriptEngine::Call+0x10c (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\scrptmgr.cpp @ 2005]
0f 01e4fbec 749f593d 01e4fc4c 00000000 00000000 asp!CallScriptFunctionOfEngine+0x48 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 1394]
10 01e4fc28 749f54c8 01a11fa0 01591e90 01e4fc94 asp!ExecuteRequest+0x19e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 326]
11 01e4fc70 749fcf9b 01a11fa0 01591e90 01e4fc94 asp!Execute+0x17e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 190]
12 01e4fcc0 749fcc37 00000000 000bdd10 01e4fd58 asp!CHitObj::ViperAsyncCallback+0x452 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\hitobj.cpp @ 2487]
13 01e4fcd8 787c214a 00000000 77acd455 01b51e90 asp!CViperAsyncRequest::OnCall+0x3f (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\viperint.cpp @ 194]
14 01e4fce0 77acd455 01b51e90 77aa71a0 00000000 COMSVCS!STAActivityWorkHelper+0xa (FPO: [1,0,0]) (CONV: stdcall) [.\stathred.cpp @ 1321]
15 01e4fd24 77acd318 000751d8 00085850 787c2140 ole32!EnterForCallback+0x6a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1758]
16 01e4fe50 77acd1dd 00085850 787c2140 01b51e90 ole32!SwitchForCallback+0x12b (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1643]
17 01e4fe78 77aac61a 00085850 787c2140 01b51e90 ole32!PerformCallback+0x50 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1558]
18 01e4fed4 77ad1185 000757c0 787c2140 01b51e90 ole32!CObjectContext::InternalContextCallback+0xf5 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\context.cxx @ 3942]
19 01e4fef4 787beff3 000757c0 787c2140 01b51e90 ole32!CObjectContext::DoCallback+0x1a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\context.cxx @ 3798]
1a 01e4ff24 787c11ad 00010007 00000000 01e4ffec COMSVCS!STAActivityWork::DoWork+0x73 (FPO: [0,4,2]) (CONV: thiscall) [..\inc\STAThred.h @ 340]
1b 01e4ffb4 77e887dd 00b1f364 00010007 00000000 COMSVCS!STAThread::STAThreadWorker+0x465 (FPO: [EBP 0x01e4ffec] [1,31,4]) (CONV: stdcall) [.\stathred.cpp @ 628]
1c 01e4ffec 00000000 787c0d48 00b1f364 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]
So from the dissasembly code, we noticed that this is a loop in the extension code.