1) Without symbols
a) Finding the thread(s) causing High CPU using performance Monitor
In performance Monitor:
a- Get the thread ID(s) of the threads averaging high CPU.
b- Convert the thread ID(s) from Decimal to Hexadecimal.
In Windbg:
a- Search for the thread ID(s) in step (b).
In our example:
Looking in Performance Monitor, Process inetinfo is averaging 100% CPU. Adding all threads try identify the thread(s) that have the same (or almost) shape
as the main process as far as %CPU time is concerned.
Thread ID 1440 is the one that hasthe same shape as the process inetinfo, this means that the process CPU time is taken by this thread.
1440 = 0x5a0
Looking for thread ID 5a0 in the debugger
0:017> ~~[5a0]kvn
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\IISDebugLabs\www\App\badfil.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINNT\System32\inetsrv\w3svc.dll -
# ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0129fbf4 65f0354f 00000001 00004000 08000002 badfil!HttpFilterProc+0x764
01 0129fc1c 65f034bc 000003e8 00004000 00000002 w3svc!STR::Copy+0x11a
02 0129fc50 65f02cad 0129fedc 0129fedc 00e846d0 w3svc!STR::Copy+0x87
03 0129fca8 65f02ac1 006a9030 0000019d 0129fcd4 w3svc!CLIENT_CONN::OnSessionStartup+0xa69
04 0129fcec 65f029b5 006a9030 0000019d 0129fedc w3svc!CLIENT_CONN::OnSessionStartup+0x87d
05 0129fd2c 65f028f8 0129fea4 0129fedc 0129feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x771
06 0129fd4c 65f027b3 0129fea4 0129fedc 0129feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x6b4
07 0129fec4 65f020d7 0129fedc 011a3568 0129ff20 w3svc!CLIENT_CONN::OnSessionStartup+0x56f
08 0129fee4 65f01b47 00000000 00000000 00000000 w3svc!HTTP_HEADERS::Reset+0x311
09 0129ff08 65f01aa5 0129ff20 77e873b2 011a3568 w3svc+0x1b47
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINNT\System32\inetsrv\ISATQ.DLL -
0a 0129ff4c 6d701443 011a3568 0000019d 00104c40 w3svc+0x1aa5
0b 0129ff80 6d702203 00000000 0000019d 011a356c ISATQ!AtqReadSocket+0x228
0c 77deec86 ece868ff fb6877dc 6477db19 000000a1 ISATQ!AtqWriteSocket+0x495
0d 6aec8b55 00000000 00000000 00000000 00000000 0xece868ff
Basically this is enough to tell the customer that the extension "badFil.dll" is the one causing this problem of high CPU usage
b) Finding the thread(s) causing High CPU using "!runaway"
The !runaway debugger extension shows information about the time taken by each thread. Note that this command is used only in a live debug session, you cannot use it if you
are analyzing a crash dump.
0:017> !runaway 1
User Mode Time
Thread Time
5a0 0:00:27.0990
940 0:00:00.0100
998 0:00:00.0070
970 0:00:00.0030
b08 0:00:00.0000
5f0 0:00:00.0000
b38 0:00:00.0000
ad0 0:00:00.0000
95c 0:00:00.0000
7ac 0:00:00.0000
9bc 0:00:00.0000
9a8 0:00:00.0000
aa0 0:00:00.0000
b5c 0:00:00.0000
45c 0:00:00.0000
b70 0:00:00.0000
980 0:00:00.0000
b14 0:00:00.0000
2) With symbols including public symbols of "badFil.dll"
First, we need to add the symbol path(company wide symbols) as well as symbols of our dll
First, we need to add the public symbol path(company wide symbols) as well as symbols of our dll
a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols;C:\IISDebugLabs\symbols\Pub
and hit Enter b) Using the ".sympath+" command:
0:008> .sympath+ SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols
0:008> .sympath+ C :\IISDebugLabs\symbols\Pub (where public symbols of badFil.dll are located)
Symbol search path is:SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols; C :\IISDebugLabs\symbols\Pub
The stack backtrace of the thread that consumes high CPU looks like:
0:017> ~~[5a0]kvn50
# ChildEBP RetAddr Args to Child
00 0129fbf4 65f0354f 00000001 00004000 08000002 badfil!doCPU1+0x21
01 0129fc1c 65f034bc 000003e8 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d (FPO: [Non-Fpo])
02 0129fc50 65f02cad 0129fedc 0129fedc 00e846d0 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x3e (FPO: [Non-Fpo])
03 0129fca8 65f02ac1 006a9030 0000019d 0129fcd4 w3svc!HTTP_REQUEST::Parse+0x372 (FPO: [Non-Fpo])
04 0129fcec 65f029b5 006a9030 0000019d 0129fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x28 (FPO: [Non-Fpo])
05 0129fd2c 65f028f8 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1e6 (FPO: [Non-Fpo])
06 0129fd4c 65f027b3 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x6c (FPO: [Non-Fpo])
07 0129fec4 65f020d7 0129fedc 011a3568 0129ff20 w3svc!HTTP_REQUEST::DoWork+0x8b (FPO: [Non-Fpo])
08 0129fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa (FPO: [Non-Fpo])
09 0129ff08 65f01aa5 0129ff20 77e873b2 011a3568 w3svc!CreateClient+0x7b (FPO: [Non-Fpo])
0a 0129ff4c 6d701443 011a3568 0000019d 00104c40 w3svc!W3OnConnectEx+0x118 (FPO: [Non-Fpo])
0b 0129ff80 6d702203 00000000 0000019d 011a356c ISATQ!AtqpProcessContext+0x23a (FPO: [Non-Fpo])
0c 0129ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8 (FPO: [EBP 0x0129ffec] [1,4,4])
0d 0129ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo])
Advanced Technique
- Switch to the thread consuning high CPU
0:017> ~~[5a0]s
eax=00007226 ebx=387542e6 ecx=00000000 edx=003875b5 esi=77e8768e edi=00007530
eip=016b1c66 esp=0129f5d4 ebp=0129fbf4 iopl=0 nv up ei ng nz na po cy
cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000287
badfil!doCPU1+21:
016b1c66 ffd6 call esi {KERNEL32!GetTickCount (77e8768e)}
Disassembly code
0:008> u eip
badfil!doCPU1+21
016b1c66 ffd6 call esi
016b1c68 2bc3 sub eax,ebx
016b1c6a ebf6 jmp badfil!doCPU1+0x1d (016b1c62)
016b1c6c 6a01 push 0x1
016b1c6e 58 pop eax
016b1c6f 5f pop edi
016b1c70 5e pop esi
016b1c71 5b pop ebx
Notice the "jmp" instruction to (016b1c62)We need to see what code we have in that address
0:008> u 016b1c62
badfil!doCPU1+1d
016b1c62 3bc7 cmp eax,edi
016b1c64 7706 ja badfil!doCPU1+0x27 (016b1c6c)
016b1c66 ffd6 call esi
016b1c68 2bc3 sub eax,ebx
016b1c6a ebf6 jmp badfil!doCPU1+0x1d (016b1c62)
016b1c6c 6a01 push 0x1
016b1c6e 58 pop eax
016b1c6f 5f pop edi
This looks like a loop in the filter filter at function doCPU1 code where we jump to 016b1c62 any time the value in eax is less than the value in edi (cmp eax,edi ). If the eax is greather than edi, we
will be at 016b1c6c which is "push 0x1 "
3) with Private Symbols "badFil.dll"
Point the debugger to use the private symbols
For company wide internal symbols, use \\symbols\symbols
For the component private symbols, use C:\IISDebugLabs\Symbols\Pri
Verify the component symbols are lined up.
0:008> lmv mbadfil
start end module name
015a0000 015a6000 badfil (pdb symbols) C:\IISDebugLabs\Symbols\Pri
Loaded symbol image file: badfil.dll
Image path: C:\IISDebugLabs\www\App\badfil.dll
Timestamp: Wed May 28 01:11:01 2003 (3ED444E5)
Checksum: 0000D9A0
File version: 5.0.0.0
Product version: 5.0.0.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
CompanyName: Microsoft Corporation
ProductName: Internet Information Services
InternalName: badFIL.dll
OriginalFilename: badFIL.dll
ProductVersion: 5.00.0984
FileVersion: 5.00.0984
FileDescription: badFIL Filter DLL
LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999
The stack backtrace of the thread that consumes high CPU looks like:
0:017> ~~[5a0]kvn50
# ChildEBP RetAddr Args to Child
00 0129fbf4 65f0354f 00000001 00004000 08000002 badfil!doCPU1+0x21 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 649]
01 0129fc1c 65f034bc 000003e8 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d (FPO: [Non-Fpo])
02 0129fc50 65f02cad 0129fedc 0129fedc 00e846d0 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x3e (FPO: [Non-Fpo])
03 0129fca8 65f02ac1 006a9030 0000019d 0129fcd4 w3svc!HTTP_REQUEST::Parse+0x372 (FPO: [Non-Fpo])
04 0129fcec 65f029b5 006a9030 0000019d 0129fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x28 (FPO: [Non-Fpo])
05 0129fd2c 65f028f8 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1e6 (FPO: [Non-Fpo])
06 0129fd4c 65f027b3 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x6c (FPO: [Non-Fpo])
07 0129fec4 65f020d7 0129fedc 011a3568 0129ff20 w3svc!HTTP_REQUEST::DoWork+0x8b (FPO: [Non-Fpo])
08 0129fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa (FPO: [Non-Fpo])
09 0129ff08 65f01aa5 0129ff20 77e873b2 011a3568 w3svc!CreateClient+0x7b (FPO: [Non-Fpo])
0a 0129ff4c 6d701443 011a3568 0000019d 00104c40 w3svc!W3OnConnectEx+0x118 (FPO: [Non-Fpo])
0b 0129ff80 6d702203 00000000 0000019d 011a356c ISATQ!AtqpProcessContext+0x23a (FPO: [Non-Fpo])
0c 0129ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8 (FPO: [EBP 0x0129ffec] [1,4,4])
0d 0129ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo])
So from the dissasembly code, we noticed that this is a loop in the extension code.