Debug Labs


Slow Response in Component

This lab will demonstrate a slow response in an ISAPI filter.

 

1- Lab Setup
	

Note: This lab assumes that you have already installed the necessary lab files to run it.

1- Install the ISAPI Filter

- Right Click the Server Name

- Click "Properties"

- In the "WWW Service" Master Properties, click "Edit"

- In the "ISAPI Filters" tab, click "Add"

- In the "Filter Name" type in "BadFilter"

- In the "Executable", browse to C:\IISDebugLabs\www\App\badfil.dll

- Click "Apply" and click "Ok" all teh way out

2- Restart the IIS Admin service

- Right Click the Server Name

- Click "Restart IIS" and Click "Ok"

3- Attach windbg to inetinfo.exe

a) Using the GUI

- Click "File"

- Click "Attach to Process"

- Select "inetinfo.exe" and click "OK"

b) Using the command prompt

-Type in "windng -pn inetinfo.exe" without quotation marks and hit "Enter"

 

5- In windbg command line, type in "g" with no quotation marks and press Enter

 

 

2- Run Scenario

1- Click the "Cause XXs Delay" button to cause a slow response
 During that period of time, Press "Ctrl-Break" in Winddbg to break into inetinfo. 

					

                                                                           

1- Debug Annotation

 

1) Without symbols

 

a) Looking for the thread running the request.

 

Issue a " ~*kv " in the windbg command line to display the call stack of all running threads

 

To find the thread running the request, look for threads containing calls to "w3svc.dll" and "badfil.dll".

 

8 Id: 698.a78 Suspend: 1 Teb: 7ffd5000 Unfrozen

*** ERROR: Symbol file could not be found. Defaulted to export symbols for badfil.dll -

ChildEBP RetAddr Args to Child

WARNING: Stack unwind information not available. Following frames may be wrong.

04f3f5c8 05e51cd8 00002710 00000000 070c4ff0 ntdll!ZwDelayExecution+0xb

*** ERROR: Symbol file could not be found. Defaulted to export symbols for w3svc.dll -

04f3fbf4 65f0354f 00000001 00004000 08000002 badfil!HttpFilterProc+0x7d6

04f3fc1c 65f034bc 000003f5 00004000 00000005 w3svc!STR::Copy+0x11a

04f3fc50 65f02cad 04f3fedc 04f3fedc 0711a410 w3svc!STR::Copy+0x87

04f3fca8 65f02ac1 06208e78 00000147 04f3fcd4 w3svc!CLIENT_CONN::OnSessionStartup+0xa69

04f3fcec 65f029b5 06208e78 00000147 04f3fedc w3svc!CLIENT_CONN::OnSessionStartup+0x87d

04f3fd2c 65f028f8 04f3fea4 04f3fedc 04f3feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x771

04f3fd4c 65f027b3 04f3fea4 04f3fedc 04f3feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x6b4

04f3fec4 65f020d7 04f3fedc 04d7ceb0 04f3ff20 w3svc!CLIENT_CONN::OnSessionStartup+0x56f

04f3fee4 65f01b47 00000000 00000000 00000000 w3svc!HTTP_HEADERS::Reset+0x311

04f3ff08 65f01aa5 04f3ff20 77e873b2 04d7ceb0 w3svc+0x1b47

04f3ff4c 6d701443 04d7ceb0 00000147 064187c0 w3svc+0x1aa5

04f3ff80 6d702203 00000000 00000147 04d7ceb4 ISATQ!AtqReadSocket+0x228

77deec86 ece868ff fb6877dc 6477db19 000000a1 ISATQ!AtqWriteSocket+0x495

6aec8b55 00000000 00000000 00000000 00000000 0xece868ff

 

Basically, since only export symbols are used here, we cannot go far in analyzing this thread and finding out why this thread is slow in processing the request.

   

 

2) With symbols including public symbols of "badFil.dll"

 

First, we need to add the public symbol path(company wide symbols) as well as symbols of our dll

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols;C:\IISDebugLabs\symbols\Pub and hit Enter

b) Using the ".sympath+" command:

 

0:008> .sympath+ SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols

Symbol search path is: SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols

 

0:008> .sympath+ C :\IISDebugLabs\symbols\Pub (where public symbols of badFil.dll are located)

Symbol search path is:SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols; C :\IISDebugLabs\symbols\Pub

 

Note: For more information on setting the symbol path for windbg, please refer to the debugger documentation for the .sympath command. This will describe

the "downstream store" feature that lets you download symbols to a symbol server of your own.

 

Here is how the above thread would look like with public symbols.

 

 

8 Id: 698.a78 Suspend: 1 Teb: 7ffd5000 Unfrozen

ChildEBP RetAddr Args to Child

04f3f5a8 77e87617 00000000 04f3f5c0 062aeff0 ntdll!NtDelayExecution+0xb

04f3f5c8 05e51cd8 00002710 00000000 070c4ff0 KERNEL32!SleepEx+0x32

04f3fbf4 65f0354f 00000001 00004000 08000002 badfil!doSlow1+0x3b

04f3fc1c 65f034bc 000003f5 00004000 00000005 w3svc!FILTER_LIST::NotifyFilters+0x9d

04f3fc50 65f02cad 04f3fedc 04f3fedc 0711a410 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x4c

04f3fca8 65f02ac1 06208e78 00000147 04f3fcd4 w3svc!HTTP_REQUEST::Parse+0x380

04f3fcec 65f029b5 06208e78 00000147 04f3fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x36

04f3fd2c 65f028f8 04f3fea4 04f3fedc 04f3feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4

04f3fd4c 65f027b3 04f3fea4 04f3fedc 04f3feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a

04f3fec4 65f020d7 04f3fedc 04d7ceb0 04f3ff20 w3svc!HTTP_REQUEST::DoWork+0x99

04f3fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa

04f3ff08 65f01aa5 04f3ff20 77e873b2 04d7ceb0 w3svc!CreateClient+0x7b

04f3ff4c 6d701443 04d7ceb0 00000147 064187c0 w3svc!W3OnConnectEx+0x118

04f3ff80 6d702203 00000000 00000147 04d7ceb4 ISATQ!AtqpProcessContext+0x23a

04f3ffb4 77e887dd abcdef01 00000000 048af8a4 ISATQ!AtqPoolThread+0x1a8

04f3ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52

 

This thread shows that the call to "badfil!doSlow1" was to act on a notification. The notification is "OnPreProcHeader". You could identify that by the call to

" w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters". Up in the stack, the function "doSlow1" in badfil.dll is calling the win32 API "SleepEx". which in turn calls

"DelayExecution" of ntdll.dll to postpone thread execution.

The first parameter passed to "sleepEx" is the time interval in Milliseconds (Hex Format) for which thread execution is to be suspended.

To evaluate this parameter, do the following:

 

 

0:008> ? 00002710

Evaluate expression: 10000 = 00002710

 

 

This shows that the execution of this thread is suspended for 10,000 ms = 10 sec. This is why the response is slow.  

 

To verify that the symbols are lined up properly for our extension, issue this command.

 

 

0:008> lmv mbadfil

start end module name

015a0000 015a6000 badfil (pdb symbols) C:\IISDebugLabs\Symbols\Pri

Loaded symbol image file: badfil.dll

Image path: C:\IISDebugLabs\www\App\badfil.dll

Timestamp: Wed May 28 01:11:01 2003 (3ED444E5)

Checksum: 0000D9A0

File version: 5.0.0.0

Product version: 5.0.0.0

File flags: 0 (Mask 3F)

File OS: 40004 NT Win32

File type: 2.0 Dll

File date: 00000000.00000000

CompanyName: Microsoft Corporation

ProductName: Internet Information Services

InternalName: badFIL.dll

OriginalFilename: badFIL.dll

ProductVersion: 5.00.0984

FileVersion: 5.00.0984

FileDescription: badFIL Filter DLL

LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999

 

3) with Private Symbols "badExt.dll"

 

 

Point the debugger to use the private symbols

For company wide internal symbols, use \\symbols\symbols

For the component private symbols, use C:\IISDebugLabs\Symbols\Pri

 

The stack backtrace of the thread that should process the request looks like:

 

 

 

8 Id: 698.a78 Suspend: 1 Teb: 7ffd5000 Unfrozen

# ChildEBP RetAddr Args to Child

00 04f3f5a8 77e87617 00000000 04f3f5c0 062aeff0 ntdll!NtDelayExecution+0xb (FPO: [2,0,0]) [i386\usrstubs.asm @ 532]

01 04f3f5c8 05e51cd8 00002710 00000000 070c4ff0 KERNEL32!SleepEx+0x32 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\synch.c @ 1672]

02 04f3fbf4 65f0354f 00000001 00004000 08000002 badfil!doSlow1+0x3b (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 694]

03 04f3fc1c 65f034bc 000003f5 00004000 00000005 w3svc!FILTER_LIST::NotifyFilters+0x9d (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpfilt.cxx @ 968]

04 04f3fc50 65f02cad 04f3fedc 04f3fedc 0711a410 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x4c (FPO: [Non-Fpo]) (CONV: thiscall) [.\filter.hxx @ 473]

05 04f3fca8 65f02ac1 06208e78 00000147 04f3fcd4 w3svc!HTTP_REQUEST::Parse+0x380 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 1497]

06 04f3fcec 65f029b5 06208e78 00000147 04f3fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x36 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 1289]

07 04f3fd2c 65f028f8 04f3fea4 04f3fedc 04f3feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 634]

08 04f3fd4c 65f027b3 04f3fea4 04f3fedc 04f3feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 253]

09 04f3fec4 65f020d7 04f3fedc 04d7ceb0 04f3ff20 w3svc!HTTP_REQUEST::DoWork+0x99 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 639]

0a 04f3fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]

0b 04f3ff08 65f01aa5 04f3ff20 77e873b2 04d7ceb0 w3svc!CreateClient+0x7b (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310]

0c 04f3ff4c 6d701443 04d7ceb0 00000147 064187c0 w3svc!W3OnConnectEx+0x118 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]

0d 04f3ff80 6d702203 00000000 00000147 04d7ceb4 ISATQ!AtqpProcessContext+0x23a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]

0e 04f3ffb4 77e887dd abcdef01 00000000 048af8a4 ISATQ!AtqPoolThread+0x1a8 (FPO: [EBP 0x04f3ffec] [1,4,4]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127]

0f 04f3ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]

 

 

With private symbols, note the line of code in frame 02 that calls to "SleepEx" (badfil.cpp @ 694]). Lining up the source code shows the line as:

 

DWORD doSlow1(int Variation)

{

char *szVar = new char[10];

char *szVal = new char[10];

getQSVars (lpEcb, szVar, szVal);

int iTime; iTime = atoi( szVal );

iTime *=1000; //convert to ms since we get the time in sec

if (1 == Variation)

{

//Cause iTime delay

DWORD sl = SleepEx( iTime, FALSE ); // Line 694

DWORD dw = NoScenario(lpEcb);

}

}

 


- Microsoft Confidential -