Debug Labs


High CPU in Component - Single Thread Loop

This lab will show how to find the thread(s) causing high CPU in the process hosting the web application.

 

1- Lab Setup
	

Note: This lab assumes that you have already installed the necessary lab files to run it.

1-Install the ISAPI Filter

 

- Right Click the Server Name

- Click "Properties"

- In the "WWW Service" Master Properties, click "Edit"

- In the "ISAPI Filters" tab, click "Add"

- In the "Filter Name" type in "BadFilter"

- In the "Executable", browse to C:\IISDebugLabs\www\App\badfil.dll

- Click "Apply" and click "Ok" all teh way out.

 

2- Restart the IIS Admin service

- Right Click the Server Name

- Click "Restart IIS" and Click "Ok"

 

 

3- Run Performace Monitor to monitor the process CPU Utilization.

a- Start Performance Monitor

b- Click the Add button

c- Under "Performance Object", Select "Process"

d- Under "Select Counters from List", select "% Processor Time" and "ID Thread"

e- Under "Select Instances from List", select "inetinfo"

 

 

4- Attach the debugger to inetinfo.exe

a) Using the GUI:

- Click "File"

- Click "Attach to Process"

- Select "inetinfo.exe" and click "OK"

b) Using the command prompt

- Switch to the folder where the debuggers is installed

- Type in the command "windbg.exe -pn inetinfo.exe" with no quotes

 

5- In windbg command line, type in "g" with no quotation marks and press Enter

 

 

2- Run Scenario

1- Click the "High CPU Filter" button to cause inetinfo to experience a busy hang.
 In Performance Monitor, notice that inetinfo goes to 100% CPU utilization; Once you verify that, do the following: 
             	a- Press "Ctrl_Break" in WinDbg.        
 	b- In Performance Monitor, right click on your log that is now listed under "Counter Logs" and choose "Stop Log"

					

                

 

1- Debug Annotation

 

1) Without symbols

 

a) Finding the thread(s) causing High CPU using performance Monitor

 

In performance Monitor:

a- Get the thread ID(s) of the threads averaging high CPU.

b- Convert the thread ID(s) from Decimal to Hexadecimal.

In Windbg:

a- Search for the thread ID(s) in step (b).

 

In our example:

Looking in Performance Monitor, Process inetinfo is averaging 100% CPU. Adding all threads try identify the thread(s) that have the same (or almost) shape

as the main process as far as %CPU time is concerned. 

 

Thread ID 1440 is the one that hasthe same shape as the process inetinfo, this means that the process CPU time is taken by this thread.

1440 = 0x5a0

Looking for thread ID 5a0 in the debugger

 

 

0:017> ~~[5a0]kvn

*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\IISDebugLabs\www\App\badfil.dll -

*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINNT\System32\inetsrv\w3svc.dll -

# ChildEBP RetAddr Args to Child

WARNING: Stack unwind information not available. Following frames may be wrong.

00 0129fbf4 65f0354f 00000001 00004000 08000002 badfil!HttpFilterProc+0x764

01 0129fc1c 65f034bc 000003e8 00004000 00000002 w3svc!STR::Copy+0x11a

02 0129fc50 65f02cad 0129fedc 0129fedc 00e846d0 w3svc!STR::Copy+0x87

03 0129fca8 65f02ac1 006a9030 0000019d 0129fcd4 w3svc!CLIENT_CONN::OnSessionStartup+0xa69

04 0129fcec 65f029b5 006a9030 0000019d 0129fedc w3svc!CLIENT_CONN::OnSessionStartup+0x87d

05 0129fd2c 65f028f8 0129fea4 0129fedc 0129feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x771

06 0129fd4c 65f027b3 0129fea4 0129fedc 0129feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x6b4

07 0129fec4 65f020d7 0129fedc 011a3568 0129ff20 w3svc!CLIENT_CONN::OnSessionStartup+0x56f

08 0129fee4 65f01b47 00000000 00000000 00000000 w3svc!HTTP_HEADERS::Reset+0x311

09 0129ff08 65f01aa5 0129ff20 77e873b2 011a3568 w3svc+0x1b47

*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINNT\System32\inetsrv\ISATQ.DLL -

0a 0129ff4c 6d701443 011a3568 0000019d 00104c40 w3svc+0x1aa5

0b 0129ff80 6d702203 00000000 0000019d 011a356c ISATQ!AtqReadSocket+0x228

0c 77deec86 ece868ff fb6877dc 6477db19 000000a1 ISATQ!AtqWriteSocket+0x495

0d 6aec8b55 00000000 00000000 00000000 00000000 0xece868ff

 

 

 

Basically this is enough to tell the customer that the extension "badFil.dll" is the one causing this problem of high CPU usage

 

b) Finding the thread(s) causing High CPU using "!runaway"

   

The !runaway debugger extension shows information about the time taken by each thread. Note that this command is used only in a live debug session, you cannot use it if you

are analyzing a crash dump.

 

0:017> !runaway 1

User Mode Time

Thread Time

5a0 0:00:27.0990

940 0:00:00.0100

998 0:00:00.0070

970 0:00:00.0030

b08 0:00:00.0000

5f0 0:00:00.0000

b38 0:00:00.0000

ad0 0:00:00.0000

95c 0:00:00.0000

7ac 0:00:00.0000

9bc 0:00:00.0000

9a8 0:00:00.0000

aa0 0:00:00.0000

b5c 0:00:00.0000

45c 0:00:00.0000

b70 0:00:00.0000

980 0:00:00.0000

b14 0:00:00.0000

 

 

2) With symbols including public symbols of "badFil.dll"

 

First, we need to add the symbol path(company wide symbols) as well as symbols of our dll

 

 

First, we need to add the public symbol path(company wide symbols) as well as symbols of our dll

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols;C:\IISDebugLabs\symbols\Pub and hit Enter

b) Using the ".sympath+" command:

0:008> .sympath+ SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols

Symbol search path is: SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols

 

0:008> .sympath+ C :\IISDebugLabs\symbols\Pub (where public symbols of badFil.dll are located)

Symbol search path is:SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols; C :\IISDebugLabs\symbols\Pub

The stack backtrace of the thread that consumes high CPU looks like:

 

0:017> ~~[5a0]kvn50

# ChildEBP RetAddr Args to Child

00 0129fbf4 65f0354f 00000001 00004000 08000002 badfil!doCPU1+0x21

01 0129fc1c 65f034bc 000003e8 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d (FPO: [Non-Fpo])

02 0129fc50 65f02cad 0129fedc 0129fedc 00e846d0 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x3e (FPO: [Non-Fpo])

03 0129fca8 65f02ac1 006a9030 0000019d 0129fcd4 w3svc!HTTP_REQUEST::Parse+0x372 (FPO: [Non-Fpo])

04 0129fcec 65f029b5 006a9030 0000019d 0129fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x28 (FPO: [Non-Fpo])

05 0129fd2c 65f028f8 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1e6 (FPO: [Non-Fpo])

06 0129fd4c 65f027b3 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x6c (FPO: [Non-Fpo])

07 0129fec4 65f020d7 0129fedc 011a3568 0129ff20 w3svc!HTTP_REQUEST::DoWork+0x8b (FPO: [Non-Fpo])

08 0129fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa (FPO: [Non-Fpo])

09 0129ff08 65f01aa5 0129ff20 77e873b2 011a3568 w3svc!CreateClient+0x7b (FPO: [Non-Fpo])

0a 0129ff4c 6d701443 011a3568 0000019d 00104c40 w3svc!W3OnConnectEx+0x118 (FPO: [Non-Fpo])

0b 0129ff80 6d702203 00000000 0000019d 011a356c ISATQ!AtqpProcessContext+0x23a (FPO: [Non-Fpo])

0c 0129ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8 (FPO: [EBP 0x0129ffec] [1,4,4])

0d 0129ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo])

 

 

 

Advanced Technique

 

- Switch to the thread consuning high CPU

 

0:017> ~~[5a0]s

eax=00007226 ebx=387542e6 ecx=00000000 edx=003875b5 esi=77e8768e edi=00007530

eip=016b1c66 esp=0129f5d4 ebp=0129fbf4 iopl=0 nv up ei ng nz na po cy

cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000287

badfil!doCPU1+21:

016b1c66 ffd6 call esi {KERNEL32!GetTickCount (77e8768e)}  

 

Disassembly code

 

0:008> u eip

badfil!doCPU1+21

016b1c66 ffd6 call esi

016b1c68 2bc3 sub eax,ebx

016b1c6a ebf6 jmp badfil!doCPU1+0x1d (016b1c62)

016b1c6c 6a01 push 0x1

016b1c6e 58 pop eax

016b1c6f 5f pop edi

016b1c70 5e pop esi

016b1c71 5b pop ebx

 

Notice the "jmp" instruction to (016b1c62)We need to see what code we have in that address

 

0:008> u 016b1c62

badfil!doCPU1+1d

016b1c62 3bc7 cmp eax,edi

016b1c64 7706 ja badfil!doCPU1+0x27 (016b1c6c)

016b1c66 ffd6 call esi

016b1c68 2bc3 sub eax,ebx

016b1c6a ebf6 jmp badfil!doCPU1+0x1d (016b1c62)

016b1c6c 6a01 push 0x1

016b1c6e 58 pop eax

016b1c6f 5f pop edi

 

This looks like a loop in the filter filter at function doCPU1 code where we jump to 016b1c62 any time the value in eax is less than the value in edi (cmp eax,edi ). If the eax is greather than edi, we

will be at 016b1c6c which is "push 0x1 "

 

3) with Private Symbols "badFil.dll"

 

Point the debugger to use the private symbols

For company wide internal symbols, use \\symbols\symbols

For the component private symbols, use C:\IISDebugLabs\Symbols\Pri

 

Verify the component symbols are lined up.

 

0:008> lmv mbadfil

start end module name

015a0000 015a6000 badfil (pdb symbols) C:\IISDebugLabs\Symbols\Pri

Loaded symbol image file: badfil.dll

Image path: C:\IISDebugLabs\www\App\badfil.dll

Timestamp: Wed May 28 01:11:01 2003 (3ED444E5)

Checksum: 0000D9A0

File version: 5.0.0.0

Product version: 5.0.0.0

File flags: 0 (Mask 3F)

File OS: 40004 NT Win32

File type: 2.0 Dll

File date: 00000000.00000000

CompanyName: Microsoft Corporation

ProductName: Internet Information Services

InternalName: badFIL.dll

OriginalFilename: badFIL.dll

ProductVersion: 5.00.0984

FileVersion: 5.00.0984

FileDescription: badFIL Filter DLL

LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999

The stack backtrace of the thread that consumes high CPU looks like: 

 

 

0:017> ~~[5a0]kvn50

# ChildEBP RetAddr Args to Child

00 0129fbf4 65f0354f 00000001 00004000 08000002 badfil!doCPU1+0x21 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 649]

01 0129fc1c 65f034bc 000003e8 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d (FPO: [Non-Fpo])

02 0129fc50 65f02cad 0129fedc 0129fedc 00e846d0 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x3e (FPO: [Non-Fpo])

03 0129fca8 65f02ac1 006a9030 0000019d 0129fcd4 w3svc!HTTP_REQUEST::Parse+0x372 (FPO: [Non-Fpo])

04 0129fcec 65f029b5 006a9030 0000019d 0129fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x28 (FPO: [Non-Fpo])

05 0129fd2c 65f028f8 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1e6 (FPO: [Non-Fpo])

06 0129fd4c 65f027b3 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x6c (FPO: [Non-Fpo])

07 0129fec4 65f020d7 0129fedc 011a3568 0129ff20 w3svc!HTTP_REQUEST::DoWork+0x8b (FPO: [Non-Fpo])

08 0129fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa (FPO: [Non-Fpo])

09 0129ff08 65f01aa5 0129ff20 77e873b2 011a3568 w3svc!CreateClient+0x7b (FPO: [Non-Fpo])

0a 0129ff4c 6d701443 011a3568 0000019d 00104c40 w3svc!W3OnConnectEx+0x118 (FPO: [Non-Fpo])

0b 0129ff80 6d702203 00000000 0000019d 011a356c ISATQ!AtqpProcessContext+0x23a (FPO: [Non-Fpo])

0c 0129ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8 (FPO: [EBP 0x0129ffec] [1,4,4])

0d 0129ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo])

 

So from the dissasembly code, we noticed that this is a loop in the extension code.

Lining up the code (use ".srcpath+" debugger command to add the source path of the extension code), line 649 looks like:
 
  DWORD doCPU1(int Variation) 
{ 
	if (1 == Variation) 
	{ 
		DWORD dwStartTime = GetTickCount(); 
		while( GetTickCount() - dwStartTime <= 30000 )   //Line 649
		{ 
		} 
	} 
	return 1; 
} 
 
 


- Microsoft Confidential -