1) Without symbols
a) Finding the thread(s) causing High CPU using performance Monitor
In performance Monitor:
a- Get the thread ID(s) of the threads averaging high CPU.
b- Convert the thread ID(s) from Decimal to Hexadecimal.
In Windbg:
a- Search for the thread ID(s) in step (b).
In our example:
Looking in Performance Monitor, Process dllhost#1 (assuming it is the only dllhost process you have asside from the system package) is averaging 100% CPU. Adding all threads of that
process, try identify the thread(s) that have the same (or almost) shape as the main process as far as %CPU time is concerned.
Thread ID 3060 is the one that hasthe same shape as the process dllhost#1, this means that the process CPU time is taken by this thread.
3060 = 0xbf4
Looking for thread ID bf4 in the debugger(Issue a "~*kv" and then search for " .bf4 ") , it looks like:
12 Id: 330.bf4 Suspend: 1 Teb: 7ffd9000 Unfrozen
*** ERROR: Symbol file could not be found. Defaulted to export symbols for badext.dll -
ChildEBP RetAddr Args to Child
*** ERROR: Symbol file could not be found. Defaulted to export symbols for wam.dll -
0131f728 65d82b06 00e02128 00e021f4 00e02128 badext!HttpExtensionProc+0x170
0131f770 65d823a2 0040a8c8 00e0229d 00e02128 wam!SE_TABLE::ReleaseExtension+0xbe8
0131f7b4 77d339c0 00406c08 00000000 0000009f wam!SE_TABLE::ReleaseExtension+0x484
0131f7dc 77d93570 65d8223f 0131f7f8 00000005 RPCRT4!NdrServerInitialize+0x288
0131fa54 77d949ac 000c7748 0009fd9c 000a0680 RPCRT4!NdrStubCall2+0x570
0131fab8 77b29e7f 000c7748 000a0680 0009fd9c RPCRT4!CStdStubBuffer_Invoke+0x9e
0131fafc 77b29d6e 000a0680 000c8444 000a25b0 ole32!HDC_UserMarshal+0x78a
0131fb44 77aa7ae7 000a0680 000c8358 000c7748 ole32!HDC_UserMarshal+0x679
0131fba8 77aa7a33 0009fd9c 00000000 000c7748 ole32!CoGetCallContext+0x401
0131fbc4 77b29c89 000a0680 00000001 000c7748 ole32!CoGetCallContext+0x34d
0131fbf4 77b29a89 000a0638 0009fd9c 000c7748 ole32!HDC_UserMarshal+0x594
0131fcb4 77b2a12c 000b2d40 00000000 000c8428 ole32!HDC_UserMarshal+0x394
0131fcf4 77d33721 000a0638 000c8428 000ae4e8 ole32!HDC_UserMarshal+0xa37
0131fd2c 77d33667 77b29f7b 000ae4e8 0131fe0c RPCRT4!I_RpcFree+0x1ff
0131fd84 77d33579 00000000 00000000 0131fe0c RPCRT4!I_RpcFree+0x145
0131fda4 77d48e81 000ae4e8 00000000 0131fe0c RPCRT4!I_RpcFree+0x57
0131fdd4 77d34bc6 000ae4e8 000ae4ac 00000000 RPCRT4!RpcSsDestroyClientContext+0xa6
0131fe10 77d346c5 000a4d50 000826a0 80030001 RPCRT4!NdrCorrelationFree+0x1120
0131fe28 77d422ff 000ae3f0 0131fe50 000a4d50 RPCRT4!NdrCorrelationFree+0xc1f
Basically this is enough to tell the customer that the extension "badext.dll" is the one causing this problem of high CPU usage
b) Finding the thread(s) causing High CPU using "!runaway"
The !runaway debugger extension shows information about the time taken by each thread. Note that this command is used only in a live debug session, you cannot use it if you
are analyzing a crash dump.
0:015> !runaway 1
User Mode Time
Thread Time
bf4 0 days 0:00:13.178
a4c 0 days 0:00:00.020
abc 0 days 0:00:00.000
198 0 days 0:00:00.000
b7c 0 days 0:00:00.000
90c 0 days 0:00:00.000
6ac 0 days 0:00:00.000
bc8 0 days 0:00:00.000
498 0 days 0:00:00.000
b94 0 days 0:00:00.000
ab4 0 days 0:00:00.000
b50 0 days 0:00:00.000
978 0 days 0:00:00.000
724 0 days 0:00:00.000
bec 0 days 0:00:00.000
2) With symbols including public symbols of "badExt.dll"
First, we need to add the symbol path(company wide symbols) as well as public symbols of our dll a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pub a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pub
Note: For more information on setting the symbol path for windbg, please refer to the debugger documentation for the .sympath command. This will describe
the "downstream store" feature that lets you download symbols to a symbol server of your own.
The stack backtrace of the thread that consumes high CPU looks like:
12 Id: 330.bf4 Suspend: 1 Teb: 7ffd9000 Unfrozen
ChildEBP RetAddr Args to Child
0131f70c 01291493 00e02128 00000001 00e021f4 badext!doCPU1+0x25
0131f728 65d82b06 00e02128 00e021f4 00e02128 badext!HttpExtensionProc+0x170
0131f73c 65d82a97 00e02128 00e021f4 00e02128 wam!HSE_APPDLL::ExecuteRequest+0x93
0131f770 65d823a2 0040a8c8 00e0229d 00e02128 wam!WAM::InvokeExtension+0x35
0131f7b4 77d339c0 00406c08 00000000 0000009f wam!WAM::ProcessRequest+0x240
0131f7dc 77d93570 65d8223f 0131f7f8 00000005 RPCRT4!Invoke+0x30
0131fa54 77d949ac 000c7748 0009fd9c 000a0680 RPCRT4!NdrStubCall2+0x63d
0131fab8 77b29e7f 000c7748 000a0680 0009fd9c RPCRT4!CStdStubBuffer_Invoke+0xec
0131fafc 77b29d6e 000a0680 000c8444 000a25b0 ole32!SyncStubInvoke+0x61
0131fb44 77aa7ae7 000a0680 000c8358 000c7748 ole32!StubInvoke+0xa8
0131fba8 77aa7a33 0009fd9c 00000000 000c7748 ole32!CCtxComChnl::ContextInvoke+0xbb
0131fbc4 77b29c89 000a0680 00000001 000c7748 ole32!MTAInvoke+0x18
0131fbf4 77b29a89 000a0638 0009fd9c 000c7748 ole32!AppInvoke+0xb5
0131fcb4 77b2a12c 000b2d40 00000000 000c8428 ole32!ComInvokeWithLockAndIPID+0x29e
0131fcf4 77d33721 000a0638 000c8428 000ae4e8 ole32!ThreadInvoke+0x1b7
0131fd2c 77d33667 77b29f7b 000ae4e8 0131fe0c RPCRT4!DispatchToStubInC+0x84
0131fd84 77d33579 00000000 00000000 0131fe0c RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x100
0131fda4 77d48e81 000ae4e8 00000000 0131fe0c RPCRT4!RPC_INTERFACE::DispatchToStub+0x5e ]
0131fdd4 77d34bc6 000ae4e8 000ae4ac 00000000 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xa9
Advanced Technique
Disassembly code
0:003> u eip
badext!doCPU1+1f:
01291770 3bc7 cmp eax,edi
01291772 7706 ja badext!doCPU1+0x29 (0129177a)
01291774 ffd6 call esi
01291776 2bc3 sub eax,ebx
01291778 ebf6 jmp badext!doCPU1+0x1f (01291770)
0129177a 5b pop ebx
0129177b 5e pop esi
0129177c 5f pop edi
This looks like a loop in the extension code where we jump to 01291770 any time the value in eax is less than the value in edi (cmp eax,edi ). If the eax is greather than edi, we
will be at 0129177a which is "pop ebx"
To verify that the symbols are lined up properly for our extension, issue this command.
0:006> lmv mbadext
start end module name 06470000 06475000
badext (pdb symbols) C :\IISDebugLabs\symbols\Pub
Loaded symbol image file: C:\IISDebuglabs\www\App\badext.dll Image path: C:\IISDebuglabs\www\App\badext.dll
Timestamp: Wed Jan 29 18:07:14 2003 (3E385EA2)
Checksum: 0000EBCD
File version: 5.0.0.0
Product version: 5.0.0.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
CompanyName: Microsoft Corporation
ProductName: Internet Information Services
InternalName: badEXT.dll
OriginalFilename: badEXT.dll
ProductVersion: 5.00.0984
FileVersion: 5.00.0984
FileDescription: badEXT Test
LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999
3) with Private Symbols "badExt.dll"
First, we need to add the symbol path(company wide symbols) as well as private symbols of our dll a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pri a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pri
The stack backtrace of the thread that consumes high CPU looks like:
12 Id: 330.bf4 Suspend: 1 Teb: 7ffd9000 Unfrozen
ChildEBP RetAddr Args to Child
0131f70c 01291493 00e02128 00000001 00e021f4 badext!doCPU1+0x25 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\extDBG\badext.cpp @ 314]
0131f728 65d82b06 00e02128 00e021f4 00e02128 badext!HttpExtensionProc+0x170 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\extDBG\badext.cpp @ 120]
0131f73c 65d82a97 00e02128 00e021f4 00e02128 wam!HSE_APPDLL::ExecuteRequest+0x93 (FPO: [1,0,3]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\wam\object\isplocal.cxx @ 450]
0131f770 65d823a2 0040a8c8 00e0229d 00e02128 wam!WAM::InvokeExtension+0x35 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\wam\object\wamobj.cxx @ 913]
0131f7b4 77d339c0 00406c08 00000000 0000009f wam!WAM::ProcessRequest+0x240 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\wam\object\wamobj.cxx @ 643]
0131f7dc 77d93570 65d8223f 0131f7f8 00000005 RPCRT4!Invoke+0x30 [..\i386\stubless.asm @ 88]
0131fa54 77d949ac 000c7748 0009fd9c 000a0680 RPCRT4!NdrStubCall2+0x63d (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\ndr20\srvcall.c @ 1724]
0131fab8 77b29e7f 000c7748 000a0680 0009fd9c RPCRT4!CStdStubBuffer_Invoke+0xec (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\ndr20\stub.c @ 1571]
0131fafc 77b29d6e 000a0680 000c8444 000a25b0 ole32!SyncStubInvoke+0x61 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1136]
0131fb44 77aa7ae7 000a0680 000c8358 000c7748 ole32!StubInvoke+0xa8 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1334]
0131fba8 77aa7a33 0009fd9c 00000000 000c7748 ole32!CCtxComChnl::ContextInvoke+0xbb (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\ctxchnl.cxx @ 1204]
0131fbc4 77b29c89 000a0680 00000001 000c7748 ole32!MTAInvoke+0x18 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\callctrl.cxx @ 1952]
0131fbf4 77b29a89 000a0638 0009fd9c 000c7748 ole32!AppInvoke+0xb5 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1013]
0131fcb4 77b2a12c 000b2d40 00000000 000c8428 ole32!ComInvokeWithLockAndIPID+0x29e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1578]
0131fcf4 77d33721 000a0638 000c8428 000ae4e8 ole32!ThreadInvoke+0x1b7 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 3835]
0131fd2c 77d33667 77b29f7b 000ae4e8 0131fe0c RPCRT4!DispatchToStubInC+0x84 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\runtime\mtrt\dispatch.c @ 82]
0131fd84 77d33579 00000000 00000000 0131fe0c RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x100 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 543]
0131fda4 77d48e81 000ae4e8 00000000 0131fe0c RPCRT4!RPC_INTERFACE::DispatchToStub+0x5e (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 407]
0131fdd4 77d34bc6 000ae4e8 000ae4ac 00000000 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xa9 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 795]