Debug Labs


High CPU in Component - Single Thread Loop

This lab will show how to find the thread(s) causing high CPU in the process hosting the web application.

 

1- Lab Setup
	

Note: This lab assumes that you have already installed the necessary lab files to run it.

1- Verify the site is running in medium protection

2- Verify that the process "dllhost.exe" mapped to the out of process pooled application is running and note down its PID

To find the PID of the desired dllhost, do the follwing:

- Open Component Services

- Expand "Component Services" Node

- Expand "Computers" Node

- Expand "My Computer" Node

- Select "COM+ Applications"

- Click "Status View" icon and notice the PID column on the right hand pane

 

3- Run Performace Monitor to monitor the process CPU Utilization.

a- Start Performance Monitor

b- Click the Add button

c- Under "Performance Object", Select "Process"

d- Under "Select Counters from List", select "% Processor Time" and "ID Thread"

e- Under "Select Instances from List", select "dllhost#1"

4- Attach windbg to the dllhost process with the PID found in step 2

a) Using the GUI:

- Click "File"

- Click "Attach to Process"

- Select "dllhost.exe" with the PID found in step 2 and click "OK"

b) Using the command prompt:

- Switch to thr folder where the debugging tools for windows is installed

- Type in "windbg -p <PID>" and hit Enter

5- In windbg command line, type in "g" with no quotation marks and press Enter

 

 

2- Run Scenario

1- Click the "Invoke Component" button to load the COM Component in the dllhost process memory space. You should see the COM Component "badCom.dll"among the list of modules loaded in dllhost.exe
2- Click the "High CPU Component" button to cause dllhost to experience a busy hang.
 In Performance Monitor, notice that "dllhost#1" goes to 100% CPU utilization; Once you verify that, do the following: 
             	a- Press "Ctrl_Break" in WinDbg.        
 	b- In Performance Monitor, right click on your log that is now listed under "Counter Logs" and choose "Stop Log"

					

                                                                           

 

1- Debug Annotation

 

1) Without symbols

 

a) Finding the thread(s) causing High CPU using performance Monitor

 

In performance Monitor:

a- Get the thread ID(s) of the threads averaging high CPU.

b- Convert the thread ID(s) from Decimal to Hexadecimal.

In Windbg:

a- Search for the thread ID(s) in step (b).

 

In our example:

Looking in Performance Monitor, Process dllhost#1 (assuming it is the only dllhost process you have asside from the system package) is averaging 100% CPU. Adding all threads of that

process, try identify the thread(s) that have the same (or almost) shape as the main process as far as %CPU time is concerned. 

 

Thread ID 324 is the one that hasthe same shape as the process dllhost#1, this means that the process CPU time is taken by this thread.

324 = 0x144

 

Looking for thread ID 144 in the debugger, it looks like:

 

0:030> ~~[144 ]kvn50

*** ERROR: Symbol file could not be found. Defaulted to export symbols for c:\IISDEB~1\www\app\badcom.dll -

# ChildEBP RetAddr Args to Child

WARNING: Stack unwind information not available. Following frames may be wrong.

00 01e4f0f4 020329ee 00000001 70633d73 765f3175 badcom!DllUnregisterServer+0x10d8

*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINNT\system32\OLEAUT32.dll -

01 01e4f124 779d7b9d 0040bfd8 00408a60 00000000 badcom!DllUnregisterServer+0xc71

02 01e4f140 77a22c42 0040bfd8 00000024 00000004 OLEAUT32!DispCallFunc+0x15d

03 01e4f1d0 020325d0 000c0884 0040bfd8 00000003 OLEAUT32!ClearCustData+0x588

*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINNT\System32\vbscript.dll -

04 01e4f1f8 6b61dff3 0040bfd8 00000003 6b659388 badcom!DllUnregisterServer+0x853

05 01e4f274 6b62171a 004085e8 0040bfd8 00000003 vbscript!DllRegisterServer+0x844c

06 01e4f954 ffffffff 0040b140 00000000 004001a0 vbscript!DllRegisterServer+0xbb73

*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Program Files\Common Files\Microsoft Shared\VS7Debug\pdm.dll -

07 77f80000 00000003 00000004 0000ffff 000000b8 0xffffffff

 

 

Basically this is enough to tell the customer that the extension "badCom.dll" is the one causing this problem of high CPU usage

 

b) Finding the thread(s) causing High CPU using "!runaway"

   

The !runaway debugger extension shows information about the time taken by each thread. Note that this command is used only in a live debug session, you cannot use it if you

are analyzing a crash dump.

 

0:030> !runaway

User Mode Time

Thread Time

144 0:00:20.0339

710 0:00:00.0050

504 0:00:00.0030

69c 0:00:00.0010

788 0:00:00.0010

5d4 0:00:00.0000

3f8 0:00:00.0000

6d8 0:00:00.0000

4b4 0:00:00.0000

6b4 0:00:00.0000

13c 0:00:00.0000

140 0:00:00.0000

14c 0:00:00.0000

148 0:00:00.0000

7e0 0:00:00.0000

6c8 0:00:00.0000

534 0:00:00.0000

7cc 0:00:00.0000

458 0:00:00.0000

780 0:00:00.0000

5a0 0:00:00.0000

43c 0:00:00.0000

79c 0:00:00.0000

7a0 0:00:00.0000

6c4 0:00:00.0000

678 0:00:00.0000

45c 0:00:00.0000

800 0:00:00.0000

7ac 0:00:00.0000

6d4 0:00:00.0000

7b8 0:00:00.0000

 

 

 

2) With symbols including public symbols of "badCom.dll"

 

 

First, we need to add the private symbol path(company wide symbols) as well as public symbols of our dll

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pub and hit Enter

b) Using the ".sympath+" command:

.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pub

The stack backtrace of the thread that consumes high CPU looks like:

 

0:030> ~~[144]kvn50

# ChildEBP RetAddr Args to Child 00 01e4f0f4 020329ee 00000001 70633d73 765f3175 badcom!doCPU1+0x25

01 01e4f124 779d7b9d 0040bfd8 00408a60 00000000 badcom!CScenarios::processSc+0x1a4

02 01e4f140 77a22c42 0040bfd8 00000024 00000004 OLEAUT32!tPushValJmpTab+0xf5 [D:\OA\src\dispatch\WIN32\i386\invoke.asm @ 340]

03 01e4f1d0 020325d0 000c0884 0040bfd8 00000003 OLEAUT32!CTypeInfo2::Invoke+0x244 (CONV: stdcall) [D:\OA\src\typelib2\tiinvoke.cpp @ 629]

04 01e4f1f8 6b61dff3 0040bfd8 00000003 6b659388 badcom!ATL::IDispatchImpl::Invoke+0x41

05 01e4f274 6b62171a 004085e8 0040bfd8 00000003 vbscript!CatchIDispatchInvoke+0x9e (CONV: stdcall) [E:\vbscript\src\core\callout.h @ 111]

06 01e4f384 6b604038 0009001f 0000023f 004084e0 vbscript!InvokeDispatch+0x330 (FPO: [EBP 0x6b659388] [7,60,4]) (CONV: stdcall) [E:\vbscript\src\vbscript\var.cpp @ 1211]

07 01e4f3c8 6b603156 004084e0 0040bfd8 00000003 vbscript!GetDispatchDispID+0x1cc (FPO: [EBP 0x01e4f954] [4,3,4]) (CONV: stdcall) [E:\vbscript\src\vbscript\var.cpp @ 972]

08 01e4f998 6b601ef7 00000009 01e4fa8c 00000000 vbscript!CScriptRuntime::Run+0x3b8d (CONV: thiscall) [E:\vbscript\src\vbscript\interp.cpp @ 3347]

09 01e4f9c8 6b601c5c 00000000 00000000 00000000 vbscript!CScriptRuntime::Init+0x2e1 (FPO: [EBP 0x6b658a10] [6,0,4]) (CONV: thiscall) [E:\vbscript\src\vbscript\interp.cpp @ 904]

0a 01e4fb08 6b606fc5 004084e0 6b606ef1 00408bd8 vbscript!CScriptEntryPoint::Call+0x76 (FPO: [5,58,2]) (CONV: thiscall) [E:\vbscript\src\vbscript\entrypnt.cpp @ 40]

0b 01e4fb54 749f2e70 01911e90 6b610c9a 00000000 vbscript!CSession::GetGlobalEntryPoint+0x25 (FPO: [3,0,0]) (CONV: thiscall) [E:\vbscript\src\core\wrapper.cpp @ 485]

0c 01e4fb6c 749f3af0 01912060 01912060 012d1e90 asp!CResponse::Release+0x18 (FPO: [1,0,1]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\response.cpp @ 1716]

0d 01e4fb9c 74a22f5b 00000000 74a32c70 77e873a1 asp!CActiveScriptEngine::TryCall+0x15 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\scrptmgr.cpp @ 2040]

0e 01e4fbe0 749f3a72 00000000 01e4fc28 749f593d asp!CActiveScriptEngine::Call+0x10c (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\scrptmgr.cpp @ 2005]

0f 01e4fbec 749f593d 01e4fc4c 00000000 00000000 asp!CallScriptFunctionOfEngine+0x48 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 1394]

10 01e4fc28 749f54c8 01a11fa0 01591e90 01e4fc94 asp!ExecuteRequest+0x19e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 326]

11 01e4fc70 749fcf9b 01a11fa0 01591e90 01e4fc94 asp!Execute+0x17e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 190]

12 01e4fcc0 749fcc37 00000000 000bdd10 01e4fd58 asp!CHitObj::ViperAsyncCallback+0x452 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\hitobj.cpp @ 2487]

13 01e4fcd8 787c214a 00000000 77acd455 01b51e90 asp!CViperAsyncRequest::OnCall+0x3f (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\viperint.cpp @ 194]

14 01e4fce0 77acd455 01b51e90 77aa71a0 00000000 COMSVCS!STAActivityWorkHelper+0xa (FPO: [1,0,0]) (CONV: stdcall) [.\stathred.cpp @ 1321]

15 01e4fd24 77acd318 000751d8 00085850 787c2140 ole32!EnterForCallback+0x6a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1758]

16 01e4fe50 77acd1dd 00085850 787c2140 01b51e90 ole32!SwitchForCallback+0x12b (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1643]

17 01e4fe78 77aac61a 00085850 787c2140 01b51e90 ole32!PerformCallback+0x50 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1558]

18 01e4fed4 77ad1185 000757c0 787c2140 01b51e90 ole32!CObjectContext::InternalContextCallback+0xf5 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\context.cxx @ 3942]

19 01e4fef4 787beff3 000757c0 787c2140 01b51e90 ole32!CObjectContext::DoCallback+0x1a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\context.cxx @ 3798]

1a 01e4ff24 787c11ad 00010007 00000000 01e4ffec COMSVCS!STAActivityWork::DoWork+0x73 (FPO: [0,4,2]) (CONV: thiscall) [..\inc\STAThred.h @ 340]

1b 01e4ffb4 77e887dd 00b1f364 00010007 00000000 COMSVCS!STAThread::STAThreadWorker+0x465 (FPO: [EBP 0x01e4ffec] [1,31,4]) (CONV: stdcall) [.\stathred.cpp @ 628]

1c 01e4ffec 00000000 787c0d48 00b1f364 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]

 

 

 

Advanced Technique

 

Disassembly code

 

0:021> u eip -6

badcom!doCPU1+1f [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\comDBG\scenarios.cpp @ 356]:

02032e4f 3bc7 cmp eax,edi

02032e51 7706 ja badcom!doCPU1+0x29 (02032e59)

02032e53 ffd6 call esi

02032e55 2bc3 sub eax,ebx

02032e57 ebf6 jmp badcom!doCPU1+0x1f (02032e4f)

02032e59 5f pop edi

02032e5a 5e pop esi

02032e5b 33c0 xor eax,eax

 

This looks like a loop in the extension code where we jump to 02032e4f any time the value in eax is less than the value in edi (cmp eax,edi ). If the eax is greather than edi, we

will be at 02032e59 which is "pop ebx"

 

 

To verify that the symbols are lined up properly for our extension, issue this command.

 

 

0:020> lmv mbadcom

start end module name

02030000 02038000 badcom (pdb symbols) c:\iisdebuglabs\symbols\pub\badcom.pdb

Loaded symbol image file: badcom.dll

Image path: c:\IISDEB~1\www\app\badcom.dll

Timestamp: Tue Jun 10 22:45:51 2003 (3EE697DF) Checksum: 0000F229

File version: 1.0.0.1

Product version: 1.0.0.1

File flags: 0 (Mask 3F)

File OS: 4 Unknown Win32

File type: 2.0 Dll

File date: 00000000.00000000

CompanyName:

ProductName: badCOM

Module InternalName: badCOM

OriginalFilename: badCOM.DLL

ProductVersion: 1, 0, 0, 1

FileVersion: 1, 0, 0, 1

FileDescription: badCOM Module

LegalCopyright: Copyright 2003

 

3) with Private Symbols "badCom.dll"

 

First, we need to add the private symbol path(company wide symbols) as well as private symbols of our dll

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pri and hit Enter

b) Using the ".sympath+" command:

.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pri

 

 

The stack backtrace of the thread that consumes high CPU looks like: 

 

 

0:030> ~~[144]kvn50

# ChildEBP RetAddr Args to Child 00 01e4f0f4 020329ee 00000001 70633d73 765f3175 badcom!doCPU1+0x25 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\comDBG\scenarios.cpp @ 356]

01 01e4f124 779d7b9d 0040bfd8 00408a60 00000000 badcom!CScenarios::processSc+0x1a4 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\comDBG\scenarios.cpp @ 152]

02 01e4f140 77a22c42 0040bfd8 00000024 00000004 OLEAUT32!tPushValJmpTab+0xf5 [D:\OA\src\dispatch\WIN32\i386\invoke.asm @ 340]

03 01e4f1d0 020325d0 000c0884 0040bfd8 00000003 OLEAUT32!CTypeInfo2::Invoke+0x244 (CONV: stdcall) [D:\OA\src\typelib2\tiinvoke.cpp @ 629]

04 01e4f1f8 6b61dff3 0040bfd8 00000003 6b659388 badcom!ATL::IDispatchImpl::Invoke+0x41 (CONV: stdcall) [c:\nt\public\sdk\inc\atl30\atlcom.h @ 4093]

05 01e4f274 6b62171a 004085e8 0040bfd8 00000003 vbscript!CatchIDispatchInvoke+0x9e (CONV: stdcall) [E:\vbscript\src\core\callout.h @ 111]

06 01e4f384 6b604038 0009001f 0000023f 004084e0 vbscript!InvokeDispatch+0x330 (FPO: [EBP 0x6b659388] [7,60,4]) (CONV: stdcall) [E:\vbscript\src\vbscript\var.cpp @ 1211]

07 01e4f3c8 6b603156 004084e0 0040bfd8 00000003 vbscript!GetDispatchDispID+0x1cc (FPO: [EBP 0x01e4f954] [4,3,4]) (CONV: stdcall) [E:\vbscript\src\vbscript\var.cpp @ 972]

08 01e4f998 6b601ef7 00000009 01e4fa8c 00000000 vbscript!CScriptRuntime::Run+0x3b8d (CONV: thiscall) [E:\vbscript\src\vbscript\interp.cpp @ 3347]

09 01e4f9c8 6b601c5c 00000000 00000000 00000000 vbscript!CScriptRuntime::Init+0x2e1 (FPO: [EBP 0x6b658a10] [6,0,4]) (CONV: thiscall) [E:\vbscript\src\vbscript\interp.cpp @ 904]

0a 01e4fb08 6b606fc5 004084e0 6b606ef1 00408bd8 vbscript!CScriptEntryPoint::Call+0x76 (FPO: [5,58,2]) (CONV: thiscall) [E:\vbscript\src\vbscript\entrypnt.cpp @ 40]

0b 01e4fb54 749f2e70 01911e90 6b610c9a 00000000 vbscript!CSession::GetGlobalEntryPoint+0x25 (FPO: [3,0,0]) (CONV: thiscall) [E:\vbscript\src\core\wrapper.cpp @ 485]

0c 01e4fb6c 749f3af0 01912060 01912060 012d1e90 asp!CResponse::Release+0x18 (FPO: [1,0,1]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\response.cpp @ 1716]

0d 01e4fb9c 74a22f5b 00000000 74a32c70 77e873a1 asp!CActiveScriptEngine::TryCall+0x15 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\scrptmgr.cpp @ 2040]

0e 01e4fbe0 749f3a72 00000000 01e4fc28 749f593d asp!CActiveScriptEngine::Call+0x10c (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\scrptmgr.cpp @ 2005]

0f 01e4fbec 749f593d 01e4fc4c 00000000 00000000 asp!CallScriptFunctionOfEngine+0x48 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 1394]

10 01e4fc28 749f54c8 01a11fa0 01591e90 01e4fc94 asp!ExecuteRequest+0x19e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 326]

11 01e4fc70 749fcf9b 01a11fa0 01591e90 01e4fc94 asp!Execute+0x17e (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\exec.cpp @ 190]

12 01e4fcc0 749fcc37 00000000 000bdd10 01e4fd58 asp!CHitObj::ViperAsyncCallback+0x452 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\cmp\asp\hitobj.cpp @ 2487]

13 01e4fcd8 787c214a 00000000 77acd455 01b51e90 asp!CViperAsyncRequest::OnCall+0x3f (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\cmp\asp\viperint.cpp @ 194]

14 01e4fce0 77acd455 01b51e90 77aa71a0 00000000 COMSVCS!STAActivityWorkHelper+0xa (FPO: [1,0,0]) (CONV: stdcall) [.\stathred.cpp @ 1321]

15 01e4fd24 77acd318 000751d8 00085850 787c2140 ole32!EnterForCallback+0x6a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1758]

16 01e4fe50 77acd1dd 00085850 787c2140 01b51e90 ole32!SwitchForCallback+0x12b (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1643]

17 01e4fe78 77aac61a 00085850 787c2140 01b51e90 ole32!PerformCallback+0x50 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\crossctx.cxx @ 1558]

18 01e4fed4 77ad1185 000757c0 787c2140 01b51e90 ole32!CObjectContext::InternalContextCallback+0xf5 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\context.cxx @ 3942]

19 01e4fef4 787beff3 000757c0 787c2140 01b51e90 ole32!CObjectContext::DoCallback+0x1a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\ole32\com\dcomrem\context.cxx @ 3798]

1a 01e4ff24 787c11ad 00010007 00000000 01e4ffec COMSVCS!STAActivityWork::DoWork+0x73 (FPO: [0,4,2]) (CONV: thiscall) [..\inc\STAThred.h @ 340]

1b 01e4ffb4 77e887dd 00b1f364 00010007 00000000 COMSVCS!STAThread::STAThreadWorker+0x465 (FPO: [EBP 0x01e4ffec] [1,31,4]) (CONV: stdcall) [.\stathred.cpp @ 628]

1c 01e4ffec 00000000 787c0d48 00b1f364 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]

 

 

So from the dissasembly code, we noticed that this is a loop in the extension code.

Lining up the code (use ".srcpath+" debugger command to add the source path of the extension code), line 356 looks like:
 
  DWORD doCPU1(int Variation) 
{ 
	if (1 == Variation) 
	{ 
		DWORD dwStartTime = GetTickCount(); 
		while( GetTickCount() - dwStartTime <= 30000 )   //Line 356
		{ 
		} 
	} 
	return S_OK; 
} 
 
 


- Microsoft Confidential -