Debug Labs


High CPU in Component - Single Thread Loop

This lab will show how to find the thread(s) causing high CPU in the process hosting the web application.

 

1- Lab Setup
	

Note: This lab assumes that you have already installed the necessary lab files to run it.

1- Verify the site is running in medium protection

2- Verify that the process "dllhost.exe" mapped to the out of process pooled application is running and note down its PID

To find the PID of the desired dllhost, do the follwing:

- Open Component Services

- Expand "Component Services" Node

- Expand "Computers" Node

- Expand "My Computer" Node

- Select "COM+ Applications"

- Click "Status View" icon and notice the PID column on the right hand pane

 

3- Run Performace Monitor to monitor the process CPU Utilization.

a- Start Performance Monitor

b- Click the Add button

c- Under "Performance Object", Select "Process"

d- Under "Select Counters from List", select "% Processor Time" and "ID Thread"

e- Under "Select Instances from List", select "dllhost#1"

4- Attach windbg to the dllhost process with the PID found in step 2

a) Using the GUI:

- Click "File"

- Click "Attach to Process"

- Select "dllhost.exe" with the PID found in step 2 and click "OK"

b) Using the command prompt:

- Switch to thr folder where the debugging tools for windows is installed

- Type in "windbg -p <PID>" and hit Enter

5- In windbg command line, type in "g" with no quotation marks and press Enter

 

 

2- Run Scenario

1- Click the "Invoke Extension" button to load the ISAPI Extension in the dllhost process memory space. You should see the ISAPI extension "badExt.dll"among the list of modules loaded in dllhost.exe
2- Click the "High CPU Extension" button to cause dllhost to experience a busy hang.
 In Performance Monitor, notice that "dllhost#1" goes to 100% CPU utilization; Once you verify that, do the following: 
             	a- Press "Ctrl_Break" in WinDbg.        
 	b- In Performance Monitor, right click on your log that is now listed under "Counter Logs" and choose "Stop Log"

					

                                                                           

 

1- Debug Annotation

 

1) Without symbols

 

a) Finding the thread(s) causing High CPU using performance Monitor

 

In performance Monitor:

a- Get the thread ID(s) of the threads averaging high CPU.

b- Convert the thread ID(s) from Decimal to Hexadecimal.

In Windbg:

a- Search for the thread ID(s) in step (b).

 

In our example:

Looking in Performance Monitor, Process dllhost#1 (assuming it is the only dllhost process you have asside from the system package) is averaging 100% CPU. Adding all threads of that

process, try identify the thread(s) that have the same (or almost) shape as the main process as far as %CPU time is concerned. 

 

Thread ID 3060 is the one that hasthe same shape as the process dllhost#1, this means that the process CPU time is taken by this thread.

3060 = 0xbf4

Looking for thread ID bf4 in the debugger(Issue a "~*kv" and then search for " .bf4 ") , it looks like:

 

12 Id: 330.bf4 Suspend: 1 Teb: 7ffd9000 Unfrozen

*** ERROR: Symbol file could not be found. Defaulted to export symbols for badext.dll -

ChildEBP RetAddr Args to Child

*** ERROR: Symbol file could not be found. Defaulted to export symbols for wam.dll -

0131f728 65d82b06 00e02128 00e021f4 00e02128 badext!HttpExtensionProc+0x170

0131f770 65d823a2 0040a8c8 00e0229d 00e02128 wam!SE_TABLE::ReleaseExtension+0xbe8

0131f7b4 77d339c0 00406c08 00000000 0000009f wam!SE_TABLE::ReleaseExtension+0x484

0131f7dc 77d93570 65d8223f 0131f7f8 00000005 RPCRT4!NdrServerInitialize+0x288

0131fa54 77d949ac 000c7748 0009fd9c 000a0680 RPCRT4!NdrStubCall2+0x570

0131fab8 77b29e7f 000c7748 000a0680 0009fd9c RPCRT4!CStdStubBuffer_Invoke+0x9e

0131fafc 77b29d6e 000a0680 000c8444 000a25b0 ole32!HDC_UserMarshal+0x78a

0131fb44 77aa7ae7 000a0680 000c8358 000c7748 ole32!HDC_UserMarshal+0x679

0131fba8 77aa7a33 0009fd9c 00000000 000c7748 ole32!CoGetCallContext+0x401

0131fbc4 77b29c89 000a0680 00000001 000c7748 ole32!CoGetCallContext+0x34d

0131fbf4 77b29a89 000a0638 0009fd9c 000c7748 ole32!HDC_UserMarshal+0x594

0131fcb4 77b2a12c 000b2d40 00000000 000c8428 ole32!HDC_UserMarshal+0x394

0131fcf4 77d33721 000a0638 000c8428 000ae4e8 ole32!HDC_UserMarshal+0xa37

0131fd2c 77d33667 77b29f7b 000ae4e8 0131fe0c RPCRT4!I_RpcFree+0x1ff

0131fd84 77d33579 00000000 00000000 0131fe0c RPCRT4!I_RpcFree+0x145

0131fda4 77d48e81 000ae4e8 00000000 0131fe0c RPCRT4!I_RpcFree+0x57

0131fdd4 77d34bc6 000ae4e8 000ae4ac 00000000 RPCRT4!RpcSsDestroyClientContext+0xa6

0131fe10 77d346c5 000a4d50 000826a0 80030001 RPCRT4!NdrCorrelationFree+0x1120

0131fe28 77d422ff 000ae3f0 0131fe50 000a4d50 RPCRT4!NdrCorrelationFree+0xc1f

 

 

Basically this is enough to tell the customer that the extension "badext.dll" is the one causing this problem of high CPU usage

 

b) Finding the thread(s) causing High CPU using "!runaway"

   

The !runaway debugger extension shows information about the time taken by each thread. Note that this command is used only in a live debug session, you cannot use it if you

are analyzing a crash dump.

 

0:015> !runaway 1

User Mode Time

Thread Time

bf4 0 days 0:00:13.178

a4c 0 days 0:00:00.020

abc 0 days 0:00:00.000

198 0 days 0:00:00.000

b7c 0 days 0:00:00.000

90c 0 days 0:00:00.000

6ac 0 days 0:00:00.000

bc8 0 days 0:00:00.000

498 0 days 0:00:00.000

b94 0 days 0:00:00.000

ab4 0 days 0:00:00.000

b50 0 days 0:00:00.000

978 0 days 0:00:00.000

724 0 days 0:00:00.000

bec 0 days 0:00:00.000

 

 

2) With symbols including public symbols of "badExt.dll"

 

 

First, we need to add the symbol path(company wide symbols) as well as public symbols of our dll

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pub and hit Enter

b) Using the ".sympath+" command:

.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pub

 

 

Note: For more information on setting the symbol path for windbg, please refer to the debugger documentation for the .sympath command. This will describe

the "downstream store" feature that lets you download symbols to a symbol server of your own.

The stack backtrace of the thread that consumes high CPU looks like:

 

12 Id: 330.bf4 Suspend: 1 Teb: 7ffd9000 Unfrozen

ChildEBP RetAddr Args to Child

0131f70c 01291493 00e02128 00000001 00e021f4 badext!doCPU1+0x25

0131f728 65d82b06 00e02128 00e021f4 00e02128 badext!HttpExtensionProc+0x170

0131f73c 65d82a97 00e02128 00e021f4 00e02128 wam!HSE_APPDLL::ExecuteRequest+0x93

0131f770 65d823a2 0040a8c8 00e0229d 00e02128 wam!WAM::InvokeExtension+0x35

0131f7b4 77d339c0 00406c08 00000000 0000009f wam!WAM::ProcessRequest+0x240

0131f7dc 77d93570 65d8223f 0131f7f8 00000005 RPCRT4!Invoke+0x30

0131fa54 77d949ac 000c7748 0009fd9c 000a0680 RPCRT4!NdrStubCall2+0x63d

0131fab8 77b29e7f 000c7748 000a0680 0009fd9c RPCRT4!CStdStubBuffer_Invoke+0xec

0131fafc 77b29d6e 000a0680 000c8444 000a25b0 ole32!SyncStubInvoke+0x61

0131fb44 77aa7ae7 000a0680 000c8358 000c7748 ole32!StubInvoke+0xa8

0131fba8 77aa7a33 0009fd9c 00000000 000c7748 ole32!CCtxComChnl::ContextInvoke+0xbb

0131fbc4 77b29c89 000a0680 00000001 000c7748 ole32!MTAInvoke+0x18

0131fbf4 77b29a89 000a0638 0009fd9c 000c7748 ole32!AppInvoke+0xb5

0131fcb4 77b2a12c 000b2d40 00000000 000c8428 ole32!ComInvokeWithLockAndIPID+0x29e

0131fcf4 77d33721 000a0638 000c8428 000ae4e8 ole32!ThreadInvoke+0x1b7

0131fd2c 77d33667 77b29f7b 000ae4e8 0131fe0c RPCRT4!DispatchToStubInC+0x84

0131fd84 77d33579 00000000 00000000 0131fe0c RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x100

0131fda4 77d48e81 000ae4e8 00000000 0131fe0c RPCRT4!RPC_INTERFACE::DispatchToStub+0x5e ]

0131fdd4 77d34bc6 000ae4e8 000ae4ac 00000000 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xa9

 

 

Advanced Technique

 

Disassembly code

 

0:003> u eip

badext!doCPU1+1f:

01291770 3bc7 cmp eax,edi

01291772 7706 ja badext!doCPU1+0x29 (0129177a)

01291774 ffd6 call esi

01291776 2bc3 sub eax,ebx

01291778 ebf6 jmp badext!doCPU1+0x1f (01291770)

0129177a 5b pop ebx

0129177b 5e pop esi

0129177c 5f pop edi

 

This looks like a loop in the extension code where we jump to 01291770 any time the value in eax is less than the value in edi (cmp eax,edi ). If the eax is greather than edi, we

will be at 0129177a which is "pop ebx"

 

To verify that the symbols are lined up properly for our extension, issue this command.

 

0:006> lmv mbadext

start end module name 06470000 06475000

badext (pdb symbols) C :\IISDebugLabs\symbols\Pub

Loaded symbol image file: C:\IISDebuglabs\www\App\badext.dll Image path: C:\IISDebuglabs\www\App\badext.dll

Timestamp: Wed Jan 29 18:07:14 2003 (3E385EA2)

Checksum: 0000EBCD

File version: 5.0.0.0

Product version: 5.0.0.0

File flags: 0 (Mask 3F)

File OS: 40004 NT Win32

File type: 2.0 Dll

File date: 00000000.00000000

CompanyName: Microsoft Corporation

ProductName: Internet Information Services

InternalName: badEXT.dll

OriginalFilename: badEXT.dll

ProductVersion: 5.00.0984

FileVersion: 5.00.0984

FileDescription: badEXT Test

LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999

3) with Private Symbols "badExt.dll"

 

First, we need to add the symbol path(company wide symbols) as well as private symbols of our dll

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pri and hit Enter

b) Using the ".sympath+" command:

.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pri

 

The stack backtrace of the thread that consumes high CPU looks like: 

 

 

12 Id: 330.bf4 Suspend: 1 Teb: 7ffd9000 Unfrozen

ChildEBP RetAddr Args to Child

0131f70c 01291493 00e02128 00000001 00e021f4 badext!doCPU1+0x25 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\extDBG\badext.cpp @ 314]

0131f728 65d82b06 00e02128 00e021f4 00e02128 badext!HttpExtensionProc+0x170 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\extDBG\badext.cpp @ 120]

0131f73c 65d82a97 00e02128 00e021f4 00e02128 wam!HSE_APPDLL::ExecuteRequest+0x93 (FPO: [1,0,3]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\wam\object\isplocal.cxx @ 450]

0131f770 65d823a2 0040a8c8 00e0229d 00e02128 wam!WAM::InvokeExtension+0x35 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\wam\object\wamobj.cxx @ 913]

0131f7b4 77d339c0 00406c08 00000000 0000009f wam!WAM::ProcessRequest+0x240 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\wam\object\wamobj.cxx @ 643]

0131f7dc 77d93570 65d8223f 0131f7f8 00000005 RPCRT4!Invoke+0x30 [..\i386\stubless.asm @ 88]

0131fa54 77d949ac 000c7748 0009fd9c 000a0680 RPCRT4!NdrStubCall2+0x63d (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\ndr20\srvcall.c @ 1724]

0131fab8 77b29e7f 000c7748 000a0680 0009fd9c RPCRT4!CStdStubBuffer_Invoke+0xec (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\ndr20\stub.c @ 1571]

0131fafc 77b29d6e 000a0680 000c8444 000a25b0 ole32!SyncStubInvoke+0x61 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1136]

0131fb44 77aa7ae7 000a0680 000c8358 000c7748 ole32!StubInvoke+0xa8 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1334]

0131fba8 77aa7a33 0009fd9c 00000000 000c7748 ole32!CCtxComChnl::ContextInvoke+0xbb (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\ctxchnl.cxx @ 1204]

0131fbc4 77b29c89 000a0680 00000001 000c7748 ole32!MTAInvoke+0x18 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\callctrl.cxx @ 1952]

0131fbf4 77b29a89 000a0638 0009fd9c 000c7748 ole32!AppInvoke+0xb5 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1013]

0131fcb4 77b2a12c 000b2d40 00000000 000c8428 ole32!ComInvokeWithLockAndIPID+0x29e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 1578]

0131fcf4 77d33721 000a0638 000c8428 000ae4e8 ole32!ThreadInvoke+0x1b7 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\channelb.cxx @ 3835]

0131fd2c 77d33667 77b29f7b 000ae4e8 0131fe0c RPCRT4!DispatchToStubInC+0x84 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\rpc\runtime\mtrt\dispatch.c @ 82]

0131fd84 77d33579 00000000 00000000 0131fe0c RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x100 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 543]

0131fda4 77d48e81 000ae4e8 00000000 0131fe0c RPCRT4!RPC_INTERFACE::DispatchToStub+0x5e (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 407]

0131fdd4 77d34bc6 000ae4e8 000ae4ac 00000000 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xa9 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\rpc\runtime\mtrt\hndlsvr.cxx @ 795]

So from the dissasembly code, we noticed that this is a loop in the extension code.
Lining up the code (use ".srcpath+" debugger command to add the source path of the extension code), line 314 looks like:
 
DWORD doCPU1(EXTENSION_CONTROL_BLOCK *lpEcb, int Variation) 
{ 
	if (1 == Variation) 
	{ 
		DWORD dwStartTime = GetTickCount(); 
		while( GetTickCount() - dwStartTime <= 30000 )   //Line 314
		{ 
		} 
	} 
	return HSE_STATUS_SUCCESS; 
} 
 
 

 


- Microsoft Confidential -