1) Without symbols
a) The faulting Stack
The stack backtrace of the thread that SOV'd would look like:
0:007> kvn50
*** ERROR: Symbol file could not be found. Defaulted to export symbols for w3svc.dll -
# ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0129fbf4 65f0354f 00000001 00004000 08000002 badfil!HttpFilterProc+0xb10
01 0129fc1c 65f034bc 000003e8 00004000 00000002 w3svc!STR::Copy+0x11a
02 0129fc50 65f02cad 0129fedc 0129fedc 00e846a0 w3svc!STR::Copy+0x87
03 0129fca8 65f02ac1 006a9030 0000019a 0129fcd4 w3svc!CLIENT_CONN::OnSessionStartup+0xa69
04 0129fcec 65f029b5 006a9030 0000019a 0129fedc w3svc!CLIENT_CONN::OnSessionStartup+0x87d
05 0129fd2c 65f028f8 0129fea4 0129fedc 0129feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x771
06 0129fd4c 65f027b3 0129fea4 0129fedc 0129feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x6b4
07 0129fec4 65f020d7 0129fedc 011a3568 0129ff20 w3svc!CLIENT_CONN::OnSessionStartup+0x56f
08 0129fee4 65f01b47 00000000 00000000 00000000 w3svc!HTTP_HEADERS::Reset+0x311
09 0129ff08 65f01aa5 0129ff20 77e873b2 011a3568 w3svc+0x1b47
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ISATQ.DLL -
0a 0129ff4c 6d701443 011a3568 0000019a 00104be0 w3svc+0x1aa5
0b 0129ff80 6d702203 00000000 0000019a 011a356c ISATQ!AtqReadSocket+0x228
0c 77deec86 ece868ff fb6877dc 6477db19 000000a1 ISATQ!AtqWriteSocket+0x495
0d 6aec8b55 00000000 00000000 00000000 00000000 0xece868ff
It's obvious that in this stack backtrace, the culprit is "badFil.dll". We don't know the function yet because we still don't have
the symbols lined up. So it is enough to tell the customer to contact the dll vendor to fix it. If it is an in-house component and the customer would like helpfrom Microsoft, they should open an advisory case to troubleshoot the issue further.
0:007> .lastevent
Last event: 684.a18: Stack overflow - code c00000fd (!!! second chance !!!)
This shows that this thread (ID = a18) basically encountered a second chance stack overflow.
b) The regiters values
0:007> r
eax=0010c540 ebx=00000003 ecx=0129ffdc edx=0010c540 esi=00070000 edi=0010c540
eip=016b2012 esp=01263000 ebp=0129fbf4 iopl=0 nv up ei pl zr na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246
badfil!HttpFilterProc+b10:
016b2012 e8f3ffffff call badfil!HttpFilterProc+0xb08 (016b200a)
c) The disassembly code
0:007> u eip
badfil!HttpFilterProc+b10:
016b2012 e8f3ffffff call badfil!HttpFilterProc+0xb08 (016b200a)
016b2017 c20800 ret 0x8
016b201a 8b44240c mov eax,[esp+0xc]
016b201e 85c0 test eax,eax
016b2020 740e jz badfil!HttpFilterProc+0xb2e (016b2030)
016b2022 8b542404 mov edx,[esp+0x4]
016b2026 8a4c2408 mov cl,[esp+0x8]
016b202a 48 dec eax
2) With symbols including public symbols of "badFil.dll"
First, we need to add the public symbol path(company wide symbols) as well as symbols of our dll
a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols;C:\IISDebugLabs\symbols\Pub and hit Enter
b) Using the ".sympath+" command:
0:008> .sympath+ SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols
0:008> .sympath+ C :\IISDebugLabs\symbols\Pub (where public symbols of badFil.dll are located)
Symbol search path is:SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols; C :\IISDebugLabs\symbols\Pub
Note: For more information on setting the symbol path for windbg, please refer to the debugger documentation for the .sympath command. This will describe
the "downstream store" feature that lets you download symbols to a symbol server of your own.
The stack backtrace of the thread that SOV'd would look like:
0:007> kvn50
# ChildEBP RetAddr Args to Child
00 0129fbf4 65f0354f 00000001 00004000 08000002 badfil!FindChar+0x8
01 0129fc1c 65f034bc 000003e8 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d
02 0129fc50 65f02cad 0129fedc 0129fedc 00e846a0 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x4c
03 0129fca8 65f02ac1 006a9030 0000019a 0129fcd4 w3svc!HTTP_REQUEST::Parse+0x380
04 0129fcec 65f029b5 006a9030 0000019a 0129fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x36
05 0129fd2c 65f028f8 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4
06 0129fd4c 65f027b3 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a
07 0129fec4 65f020d7 0129fedc 011a3568 0129ff20 w3svc!HTTP_REQUEST::DoWork+0x99
08 0129fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa
09 0129ff08 65f01aa5 0129ff20 77e873b2 011a3568 w3svc!CreateClient+0x7b
0a 0129ff4c 6d701443 011a3568 0000019a 00104be0 w3svc!W3OnConnectEx+0x118
0b 0129ff80 6d702203 00000000 0000019a 011a356c ISATQ!AtqpProcessContext+0x23a
0c 0129ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8
0d 0129ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52
Now, with the public symbols lined up, we see that the last call that SOV'd was to the function "FindChar()".
Advanced Technique
To see the stack overflow caused by this thread, let issue the command "dds esp" to display the raw stack at the stack pointer region.
0:007> dds esp
01263000 0010c540
01263004 00000078
01263008 016b2017 badfil!FindChar+0xd
0126300c 0010c540
01263010 00000078
01263014 016b2017 badfil!FindChar+0xd
01263018 0010c540
0126301c 00000078
01263020 016b2017 badfil!FindChar+0xd
01263024 0010c540
01263028 00000078
0126302c 016b2017 badfil!FindChar+0xd
01263030 0010c540
01263034 00000078
01263038 016b2017 badfil!FindChar+0xd
0126303c 0010c540
01263040 00000078
01263044 016b2017 badfil!FindChar+0xd
01263048 0010c540
0126304c 00000078
01263050 016b2017 badfil!FindChar+0xd
01263054 0010c540
01263058 00000078
0126305c 016b2017 badfil!FindChar+0xd
01263060 0010c540
01263064 00000078
01263068 016b2017 badfil!FindChar+0xd
0126306c 0010c540
01263070 00000078
01263074 016b2017 badfil!FindChar+0xd
01263078 0010c540
0126307c 00000078
To see the stack size, let issue the command "!teb" that displays information about the thread environment block
0:007> !teb
TEB at 7FFD6000
ExceptionList: 129ffdc
Stack Base: 12a0000
Stack Limit: 1261000
SubSystemTib: 0
FiberData: 1e00
ArbitraryUser: 0
Self: 7ffd6000
EnvironmentPtr: 0
ClientId: 684.a18
Real ClientId: 684.a18
RpcHandle: 0
Tls Storage: 0
PEB Address: 7ffdf000
LastErrorValue: 1413
LastStatusValue: c0000100
Count Owned Locks: 0
HardErrorsMode: 0
The current stack size is:
0:007> ? 12a0000-1261000
Evaluate expression: 258048 = 0003f000
0:007> ?(258048 /1024)
Evaluate expression: 594 = 00000252
Which is 252 KB
In IIS4 and 5, each thread has a stack of a maximum size of 256k. Please refer to teh following KB:
PRB: Stack Size Now 256 K in Internet Information Server 4.0 WGID:246
ID: 189888.KB.EN-US CREATED: 1998-07-21 MODIFIED: 2001-07-16
3) with Private Symbols "badFil.dll"
Point the debugger to use the private symbols
For company wide internal symbols, use \\symbols\symbols
For the component private symbols, use C:\IISDebugLabs\Symbols\Pri
Verify the component symbols are lined up.
0:008> lmv mbadfil
start end module name
015a0000 015a6000 badfil (pdb symbols) C:\IISDebugLabs\Symbols\Pri
Loaded symbol image file: badfil.dll
Image path: C:\IISDebugLabs\www\App\badfil.dll
Timestamp: Wed May 28 01:11:01 2003 (3ED444E5)
Checksum: 0000D9A0
File version: 5.0.0.0
Product version: 5.0.0.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
CompanyName: Microsoft Corporation
ProductName: Internet Information Services
InternalName: badFIL.dll
OriginalFilename: badFIL.dll
ProductVersion: 5.00.0984
FileVersion: 5.00.0984
FileDescription: badFIL Filter DLL
LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999
The stack backtrace of the thread that AV'd would look like:
0:007> kvn50
# ChildEBP RetAddr Args to Child
00 0129fbf4 65f0354f 00000001 00004000 08000002 badfil!FindChar+0x8 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 952]
01 0129fc1c 65f034bc 000003e8 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpfilt.cxx @ 968]
02 0129fc50 65f02cad 0129fedc 0129fedc 00e846a0 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x4c (FPO: [Non-Fpo]) (CONV: thiscall) [.\filter.hxx @ 473]
03 0129fca8 65f02ac1 006a9030 0000019a 0129fcd4 w3svc!HTTP_REQUEST::Parse+0x380 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 1497]
04 0129fcec 65f029b5 006a9030 0000019a 0129fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x36 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 1289]
05 0129fd2c 65f028f8 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 634]
06 0129fd4c 65f027b3 0129fea4 0129fedc 0129feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 253]
07 0129fec4 65f020d7 0129fedc 011a3568 0129ff20 w3svc!HTTP_REQUEST::DoWork+0x99 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 639]
08 0129fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]
09 0129ff08 65f01aa5 0129ff20 77e873b2 011a3568 w3svc!CreateClient+0x7b (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310]
0a 0129ff4c 6d701443 011a3568 0000019a 00104be0 w3svc!W3OnConnectEx+0x118 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]
0b 0129ff80 6d702203 00000000 0000019a 011a356c ISATQ!AtqpProcessContext+0x23a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]
0c 0129ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8 (FPO: [EBP 0x0129ffec] [1,4,4]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127]
0d 0129ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]
As you can see, badFil.dll SOV'd while acting on the "PreProcHeader" notification, we know that because
of frame 02 (w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters). It is helpful sometimes to exact information like what request
caused this filter to AV.
In this special case, using the debugger "kp" command will show the exact request.
0:007> kpn
# ChildEBP RetAddr
00 0129fbf4 65f0354f badfil!FindChar(char * sz = 0x65f0354f "???", char c = 1 '')+0x8 [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 952]
01 0129fc1c 65f034bc w3svc!FILTER_LIST::NotifyFilters(class HTTP_FILTER * pFilter = 0x000003e8, unsigned long NotificationType = 0x4000, struct _HTTP_FILTER_CONTEXT * phfc = 0x00000002, void * NotificationData = 0x0129fc3c, int * pfFinished = 0x0129fedc, int fNotifyAll = 0)+0x9d [D:\nt\private\inet\iis\svcs\w3\server\httpfilt.cxx @ 968]
02 0129fc50 65f02cad w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters(int * pfFinished = 0x0129fedc)+0x4c [.\filter.hxx @ 473]
03 0129fca8 65f02ac1 w3svc!HTTP_REQUEST::Parse(char * pchRequest = 0x006a9030 "GET /IISDebugLabs/site/fil_so1_v1?sf=so1_v1 HTTP/1.1???", unsigned long cbData = 0x19a, unsigned long * pcbExtraData = 0x0129fcd4, int * pfHandled = 0x0129fce8, int * pfFinished = 0x0129fedc)+0x380 [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 1497]
04 0129fcec 65f029b5 w3svc!HTTP_REQ_BASE::OnCompleteRequest(char * pchRequest = 0x006a9030 "GET /IISDebugLabs/site/fil_so1_v1?sf=so1_v1 HTTP/1.1???", unsigned long cbData = 0x19a, int * pfFinished = 0x0129fedc, int * pfContinueProcessingRequest = 0x0129feb8)+0x36 [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 1289]
05 0129fd2c 65f028f8 w3svc!HTTP_REQ_BASE::UnWrapRequest(int * pfCompleteRequest = 0x0129fea4, int * pfFinished = 0x0129fedc, int * pfContinueProcessingRequest = 0x0129feb8)+0x1f4 [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 634]
06 0129fd4c 65f027b3 w3svc!HTTP_REQ_BASE::OnFillClientReq(int * pfCompleteRequest = 0x0129fea4, int * pfFinished = 0x0129fedc, int * pfContinueProcessingRequest = 0x0129feb8)+0x7a [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 253]
07 0129fec4 65f020d7 w3svc!HTTP_REQUEST::DoWork(int * pfFinished = 0x0129fedc)+0x99 [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 639]
08 0129fee4 65f01b47 w3svc!CLIENT_CONN::DoWork(unsigned long BytesWritten = 0, unsigned long CompletionStatus = 0, int fIOCompletion = 0)+0x1aa [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]
09 0129ff08 65f01aa5 w3svc!CreateClient(struct _CLIENT_CONN_PARAMS * ClientParam = 0x0129ff20)+0x7b [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310]
0a 0129ff4c 6d701443 w3svc!W3OnConnectEx(void * patqContext = 0x011a3568, unsigned long cbWritten = 0x19a, unsigned long err = 0x104be0, struct _OVERLAPPED * lpo = 0x00e83bf0)+0x118 [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]
0b 0129ff80 6d702203 ISATQ!AtqpProcessContext(struct ATQ_CONTEXT * pAtqContext = 0x00000000, unsigned long cbWritten = 0x19a, struct _OVERLAPPED * lpo = 0x011a356c, int fRet = 1)+0x23a [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]
0c 0129ffb4 77e887dd ISATQ!AtqPoolThread(unsigned long * param = 0xece868ff)+0x1a8 [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127]
0d 0129ffec 00000000 KERNEL32!BaseThreadStart( * lpStartAddress = 0x6d7091f7, void * lpParameter = 0xabcdef01)+0x52 [D:\nt\private\windows\base\client\support.c @ 460]
Notice in frame 00 the line of code as well as the name and location of the source code. Going back to the source code in badFil.cpp:
Line 952 looks like
char* FindChar(char* sz, char c)
{
char* ch = NULL;
{
ch = sz++;
sz = FindChar( ch, c ); //Line 952
}
return sz;
}
As you see this recursive function will never stop execution becuase there is no stopping condition. In real life
this kind of problems would happen if the stopping condition is never met for some reason.