Debug Labs


 Stack Overflow/Recursion in Component

This lab will demonstrate a stack overflow caused by a recursive function in the filter

 

1- Lab Setup
	

Note: This lab assumes that you have already installed the necessary lab files to run it.

1-Install the ISAPI Filter

 

- Right Click the Server Name

- Click "Properties"

- In the "WWW Service" Master Properties, click "Edit"

- In the "ISAPI Filters" tab, click "Add"

- In the "Filter Name" type in "BadFilter"

- In the "Executable", browse to C:\IISDebugLabs\www\App\badfil.dll

- Click "Apply" and click "Ok" all teh way out.

 

2- Restart the IIS Admin service

- Right Click the Server Name

- Click "Restart IIS" and Click "Ok"

 

3- Attach the debugger to inetinfo.exe

a) Using the GUI:

- Click "File"

- Click "Attach to Process"

- Select "inetinfo.exe" and click "OK"

b) Using the command prompt

-Type in "windng -pn inetinfo.exe" without quotation marks and hit "Enter"

 

4- In windbg command line, type in "g" with no quotation marks and press Enter

 

 

2- Run Scenario

1- Click the "Cause Stack Overflow" button the first time to cause a 1st chance Stack Overflow in inetinfo.exe.
 Note that the debugger breaks and diplays 
	a- Comments about the exception
b- The value of the registers
c- The module where the SOV happened
d- The assembly line that caused the SOV
2- In the WinDbg command line, type in "g" and press "Enter", a 2nd chance SOV is now encountered 

                 

 

1- Debug Annotation

 

1) Without symbols

 

a) The faulting Stack

 

The stack backtrace of the thread that SOV'd would look like:

 

0:007> kvn50

*** ERROR: Symbol file could not be found. Defaulted to export symbols for w3svc.dll -

# ChildEBP RetAddr Args to Child

WARNING: Stack unwind information not available. Following frames may be wrong.

00 0129fbf4 65f0354f 00000001 00004000 08000002 badfil!HttpFilterProc+0xb10

01 0129fc1c 65f034bc 000003e8 00004000 00000002 w3svc!STR::Copy+0x11a

02 0129fc50 65f02cad 0129fedc 0129fedc 00e846a0 w3svc!STR::Copy+0x87

03 0129fca8 65f02ac1 006a9030 0000019a 0129fcd4 w3svc!CLIENT_CONN::OnSessionStartup+0xa69

04 0129fcec 65f029b5 006a9030 0000019a 0129fedc w3svc!CLIENT_CONN::OnSessionStartup+0x87d

05 0129fd2c 65f028f8 0129fea4 0129fedc 0129feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x771

06 0129fd4c 65f027b3 0129fea4 0129fedc 0129feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x6b4

07 0129fec4 65f020d7 0129fedc 011a3568 0129ff20 w3svc!CLIENT_CONN::OnSessionStartup+0x56f

08 0129fee4 65f01b47 00000000 00000000 00000000 w3svc!HTTP_HEADERS::Reset+0x311

09 0129ff08 65f01aa5 0129ff20 77e873b2 011a3568 w3svc+0x1b47

*** ERROR: Symbol file could not be found. Defaulted to export symbols for ISATQ.DLL -

0a 0129ff4c 6d701443 011a3568 0000019a 00104be0 w3svc+0x1aa5

0b 0129ff80 6d702203 00000000 0000019a 011a356c ISATQ!AtqReadSocket+0x228

0c 77deec86 ece868ff fb6877dc 6477db19 000000a1 ISATQ!AtqWriteSocket+0x495

0d 6aec8b55 00000000 00000000 00000000 00000000 0xece868ff

 

 

It's obvious that in this stack backtrace, the culprit is "badFil.dll". We don't know the function yet because we still don't have

the symbols lined up. So it is enough to tell the customer to contact the dll vendor to fix it. If it is an in-house component and the customer would like help

from Microsoft, they should open an advisory case to troubleshoot the issue further.

 

0:007> .lastevent

Last event: 684.a18: Stack overflow - code c00000fd (!!! second chance !!!)

 

This shows that this thread (ID = a18) basically encountered a second chance stack overflow.

 

b) The regiters values

 

0:007> r

eax=0010c540 ebx=00000003 ecx=0129ffdc edx=0010c540 esi=00070000 edi=0010c540

eip=016b2012 esp=01263000 ebp=0129fbf4 iopl=0 nv up ei pl zr na po nc

cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246

badfil!HttpFilterProc+b10:

016b2012 e8f3ffffff call badfil!HttpFilterProc+0xb08 (016b200a)

 

c) The disassembly code

 

0:007> u eip

badfil!HttpFilterProc+b10:

016b2012 e8f3ffffff call badfil!HttpFilterProc+0xb08 (016b200a)

016b2017 c20800 ret 0x8

016b201a 8b44240c mov eax,[esp+0xc]

016b201e 85c0 test eax,eax

016b2020 740e jz badfil!HttpFilterProc+0xb2e (016b2030)

016b2022 8b542404 mov edx,[esp+0x4]

016b2026 8a4c2408 mov cl,[esp+0x8]

016b202a 48 dec eax

 

 

2) With symbols including public symbols of "badFil.dll"

 

First, we need to add the public symbol path(company wide symbols) as well as symbols of our dll

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols;C:\IISDebugLabs\symbols\Pub and hit Enter

b) Using the ".sympath+" command:

0:008> .sympath+ SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols

Symbol search path is: SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols

 

0:008> .sympath+ C :\IISDebugLabs\symbols\Pub (where public symbols of badFil.dll are located)

Symbol search path is:SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols; C :\IISDebugLabs\symbols\Pub

 

Note: For more information on setting the symbol path for windbg, please refer to the debugger documentation for the .sympath command. This will describe

the "downstream store" feature that lets you download symbols to a symbol server of your own.

 

The stack backtrace of the thread that SOV'd would look like:

 

0:007> kvn50

# ChildEBP RetAddr Args to Child

00 0129fbf4 65f0354f 00000001 00004000 08000002 badfil!FindChar+0x8

01 0129fc1c 65f034bc 000003e8 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d

02 0129fc50 65f02cad 0129fedc 0129fedc 00e846a0 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x4c

03 0129fca8 65f02ac1 006a9030 0000019a 0129fcd4 w3svc!HTTP_REQUEST::Parse+0x380

04 0129fcec 65f029b5 006a9030 0000019a 0129fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x36

05 0129fd2c 65f028f8 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4

06 0129fd4c 65f027b3 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a

07 0129fec4 65f020d7 0129fedc 011a3568 0129ff20 w3svc!HTTP_REQUEST::DoWork+0x99

08 0129fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa

09 0129ff08 65f01aa5 0129ff20 77e873b2 011a3568 w3svc!CreateClient+0x7b

0a 0129ff4c 6d701443 011a3568 0000019a 00104be0 w3svc!W3OnConnectEx+0x118

0b 0129ff80 6d702203 00000000 0000019a 011a356c ISATQ!AtqpProcessContext+0x23a

0c 0129ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8

0d 0129ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52

 

Now, with the public symbols lined up, we see that the last call that SOV'd was to the function "FindChar()".

Advanced Technique

To see the stack overflow caused by this thread, let issue the command "dds esp" to display the raw stack at the stack pointer region.

 

0:007> dds esp

01263000 0010c540

01263004 00000078

01263008 016b2017 badfil!FindChar+0xd

0126300c 0010c540

01263010 00000078

01263014 016b2017 badfil!FindChar+0xd

01263018 0010c540

0126301c 00000078

01263020 016b2017 badfil!FindChar+0xd

01263024 0010c540

01263028 00000078

0126302c 016b2017 badfil!FindChar+0xd

01263030 0010c540

01263034 00000078

01263038 016b2017 badfil!FindChar+0xd

0126303c 0010c540

01263040 00000078

01263044 016b2017 badfil!FindChar+0xd

01263048 0010c540

0126304c 00000078

01263050 016b2017 badfil!FindChar+0xd

01263054 0010c540

01263058 00000078

0126305c 016b2017 badfil!FindChar+0xd

01263060 0010c540

01263064 00000078

01263068 016b2017 badfil!FindChar+0xd

0126306c 0010c540

01263070 00000078

01263074 016b2017 badfil!FindChar+0xd

01263078 0010c540

0126307c 00000078

 

To see the stack size, let issue the command "!teb" that displays information about the thread environment block

 

0:007> !teb

TEB at 7FFD6000

ExceptionList: 129ffdc

Stack Base: 12a0000

Stack Limit: 1261000

SubSystemTib: 0

FiberData: 1e00

ArbitraryUser: 0

Self: 7ffd6000

EnvironmentPtr: 0

ClientId: 684.a18

Real ClientId: 684.a18

RpcHandle: 0

Tls Storage: 0

PEB Address: 7ffdf000

LastErrorValue: 1413

LastStatusValue: c0000100

Count Owned Locks: 0

HardErrorsMode: 0

 

The current stack size is:

 

0:007> ? 12a0000-1261000

Evaluate expression: 258048 = 0003f000

 

0:007> ?(258048 /1024)

Evaluate expression: 594 = 00000252

 

Which is 252 KB

 

In IIS4 and 5, each thread has a stack of a maximum size of 256k. Please refer to teh following KB:

PRB: Stack Size Now 256 K in Internet Information Server 4.0 WGID:246

ID: 189888.KB.EN-US CREATED: 1998-07-21 MODIFIED: 2001-07-16

 

 

3) with Private Symbols "badFil.dll"

 

 

Point the debugger to use the private symbols

For company wide internal symbols, use \\symbols\symbols

For the component private symbols, use C:\IISDebugLabs\Symbols\Pri

 

Verify the component symbols are lined up.

 

0:008> lmv mbadfil

start end module name

015a0000 015a6000 badfil (pdb symbols) C:\IISDebugLabs\Symbols\Pri

Loaded symbol image file: badfil.dll

Image path: C:\IISDebugLabs\www\App\badfil.dll

Timestamp: Wed May 28 01:11:01 2003 (3ED444E5)

Checksum: 0000D9A0

File version: 5.0.0.0

Product version: 5.0.0.0

File flags: 0 (Mask 3F)

File OS: 40004 NT Win32

File type: 2.0 Dll

File date: 00000000.00000000

CompanyName: Microsoft Corporation

ProductName: Internet Information Services

InternalName: badFIL.dll

OriginalFilename: badFIL.dll

ProductVersion: 5.00.0984

FileVersion: 5.00.0984

FileDescription: badFIL Filter DLL

LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999

 

 

The stack backtrace of the thread that AV'd would look like:

 

0:007> kvn50

# ChildEBP RetAddr Args to Child

00 0129fbf4 65f0354f 00000001 00004000 08000002 badfil!FindChar+0x8 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 952]

01 0129fc1c 65f034bc 000003e8 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpfilt.cxx @ 968]

02 0129fc50 65f02cad 0129fedc 0129fedc 00e846a0 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x4c (FPO: [Non-Fpo]) (CONV: thiscall) [.\filter.hxx @ 473]

03 0129fca8 65f02ac1 006a9030 0000019a 0129fcd4 w3svc!HTTP_REQUEST::Parse+0x380 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 1497]

04 0129fcec 65f029b5 006a9030 0000019a 0129fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x36 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 1289]

05 0129fd2c 65f028f8 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 634]

06 0129fd4c 65f027b3 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 253]

07 0129fec4 65f020d7 0129fedc 011a3568 0129ff20 w3svc!HTTP_REQUEST::DoWork+0x99 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 639]

08 0129fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]

09 0129ff08 65f01aa5 0129ff20 77e873b2 011a3568 w3svc!CreateClient+0x7b (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310]

0a 0129ff4c 6d701443 011a3568 0000019a 00104be0 w3svc!W3OnConnectEx+0x118 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]

0b 0129ff80 6d702203 00000000 0000019a 011a356c ISATQ!AtqpProcessContext+0x23a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]

0c 0129ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8 (FPO: [EBP 0x0129ffec] [1,4,4]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127]

0d 0129ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]

 

 

As you can see, badFil.dll SOV'd while acting on the "PreProcHeader" notification, we know that because

of frame 02 (w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters). It is helpful sometimes to exact information like what request

caused this filter to AV.

In this special case, using the debugger "kp" command will show the exact request.

 

 

0:007> kpn

# ChildEBP RetAddr

00 0129fbf4 65f0354f badfil!FindChar(char * sz = 0x65f0354f "???", char c = 1 '')+0x8 [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 952]

01 0129fc1c 65f034bc w3svc!FILTER_LIST::NotifyFilters(class HTTP_FILTER * pFilter = 0x000003e8, unsigned long NotificationType = 0x4000, struct _HTTP_FILTER_CONTEXT * phfc = 0x00000002, void * NotificationData = 0x0129fc3c, int * pfFinished = 0x0129fedc, int fNotifyAll = 0)+0x9d [D:\nt\private\inet\iis\svcs\w3\server\httpfilt.cxx @ 968]

02 0129fc50 65f02cad w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters(int * pfFinished = 0x0129fedc)+0x4c [.\filter.hxx @ 473]

03 0129fca8 65f02ac1 w3svc!HTTP_REQUEST::Parse(char * pchRequest = 0x006a9030 "GET /IISDebugLabs/site/fil_so1_v1?sf=so1_v1 HTTP/1.1???", unsigned long cbData = 0x19a, unsigned long * pcbExtraData = 0x0129fcd4, int * pfHandled = 0x0129fce8, int * pfFinished = 0x0129fedc)+0x380 [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 1497]

04 0129fcec 65f029b5 w3svc!HTTP_REQ_BASE::OnCompleteRequest(char * pchRequest = 0x006a9030 "GET /IISDebugLabs/site/fil_so1_v1?sf=so1_v1 HTTP/1.1???", unsigned long cbData = 0x19a, int * pfFinished = 0x0129fedc, int * pfContinueProcessingRequest = 0x0129feb8)+0x36 [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 1289]

05 0129fd2c 65f028f8 w3svc!HTTP_REQ_BASE::UnWrapRequest(int * pfCompleteRequest = 0x0129fea4, int * pfFinished = 0x0129fedc, int * pfContinueProcessingRequest = 0x0129feb8)+0x1f4 [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 634]

06 0129fd4c 65f027b3 w3svc!HTTP_REQ_BASE::OnFillClientReq(int * pfCompleteRequest = 0x0129fea4, int * pfFinished = 0x0129fedc, int * pfContinueProcessingRequest = 0x0129feb8)+0x7a [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 253]

07 0129fec4 65f020d7 w3svc!HTTP_REQUEST::DoWork(int * pfFinished = 0x0129fedc)+0x99 [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 639]

08 0129fee4 65f01b47 w3svc!CLIENT_CONN::DoWork(unsigned long BytesWritten = 0, unsigned long CompletionStatus = 0, int fIOCompletion = 0)+0x1aa [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]

09 0129ff08 65f01aa5 w3svc!CreateClient(struct _CLIENT_CONN_PARAMS * ClientParam = 0x0129ff20)+0x7b [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310]

0a 0129ff4c 6d701443 w3svc!W3OnConnectEx(void * patqContext = 0x011a3568, unsigned long cbWritten = 0x19a, unsigned long err = 0x104be0, struct _OVERLAPPED * lpo = 0x00e83bf0)+0x118 [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]

0b 0129ff80 6d702203 ISATQ!AtqpProcessContext(struct ATQ_CONTEXT * pAtqContext = 0x00000000, unsigned long cbWritten = 0x19a, struct _OVERLAPPED * lpo = 0x011a356c, int fRet = 1)+0x23a [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]

0c 0129ffb4 77e887dd ISATQ!AtqPoolThread(unsigned long * param = 0xece868ff)+0x1a8 [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127]

0d 0129ffec 00000000 KERNEL32!BaseThreadStart( * lpStartAddress = 0x6d7091f7, void * lpParameter = 0xabcdef01)+0x52 [D:\nt\private\windows\base\client\support.c @ 460]

 

Notice in frame 00 the line of code as well as the name and location of the source code. Going back to the source code in badFil.cpp:

 

Line 952 looks like

 

char* FindChar(char* sz, char c)

{

char* ch = NULL;

{

ch = sz++;

sz = FindChar( ch, c ); //Line 952

}

return sz;

}

As you see this recursive function will never stop execution becuase there is no stopping condition. In real life

this kind of problems would happen if the stopping condition is never met for some reason. 

 


- Microsoft Confidential -