1) Without symbols
a) The faulting Stack
The stack backtrace of the thread that AV'd would look like:
0:008>kvn50
*** ERROR: Symbol file could not be found. Defaulted to export symbols for badfil.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for w3svc.dll
# ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0118fbf4 65f0354f 00000001 00004000 08000002 badfil!HttpFilterProc+0x539
01 0118fc1c 65f034bc 000003e8 00004000 00000002 w3svc!STR::Copy+0x11a
02 0118fc50 65f02cad 0118fedc 0118fedc 00e74618 w3svc!STR::Copy+0x87
03 0118fca8 65f02ac1 006a8fb8 0000019b 0118fcd4 w3svc!CLIENT_CONN::OnSessionStartup+0xa69
04 0118fcec 65f029b5 006a8fb8 0000019b 0118fedc w3svc!CLIENT_CONN::OnSessionStartup+0x87d
05 0118fd2c 65f028f8 0118fea4 0118fedc 0118feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x771
06 0118fd4c 65f027b3 0118fea4 0118fedc 0118feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x6b4
07 0118fec4 65f020d7 0118fedc 01093568 0118ff20 w3svc!CLIENT_CONN::OnSessionStartup+0x56f
08 0118fee4 65f01b47 00000000 00000000 00000000 w3svc!HTTP_HEADERS::Reset+0x311
09 0118ff08 65f01aa5 0118ff20 77e873b2 01093568 w3svc+0x1b47
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ISATQ.DLL -
0a 0118ff4c 6d701443 01093568 0000019b 00104cd0 w3svc+0x1aa5
0b 0118ff80 6d702203 00000000 0000019b 0109356c ISATQ!AtqReadSocket+0x228
0c 77deec86 ece868ff fb6877dc 6477db19 000000a1 ISATQ!AtqWriteSocket+0x495
0d 6aec8b55 00000000 00000000 00000000 00000000 0xece868ff
In this stack backtrace, it is clear that the culprit is "badFil.dll". So it is enough to tell the customer to contact the dll vendor to fix it. If it is an in-house component and the customer would like help
from Microsoft, they should open an advisory case to troubleshoot the issue further.
b) The regiters values
0:008> r eax=00000000 ebx=00000003 ecx=00000000 edx=7803b012 esi=78002041 edi=00e70978
eip=015a1a3b esp=0118f5e0 ebp=0118fbf4 iopl=0 nv up ei pl zr na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246
badfil!HttpFilterProc+539: 015a1a3b
66c700d307 mov word ptr [eax],0x7d3 ds:0023:00000000=????
c) The disassmbly code
0:008> u eip
badfil!HttpFilterProc+539:
015a1a3b 66c700d307 mov word ptr [eax],0x7d3
015a1a40 6a01 push 0x1
015a1a42 58 pop eax
015a1a43 c20400 ret 0x4
015a1a46 837c240401 cmp dword ptr [esp+0x4],0x1
015a1a4b 7508 jnz badfil!HttpFilterProc+0x553 (015a1a55)
015a1a4d 6a00 push 0x0
015a1a4f ff1524105a01 call dword ptr [badfil+0x1024 (015a1024)]
2) With symbols including public symbols of "badFil.dll"
First, we need to add the public symbol path(company wide symbols) as well as symbols of our dll
a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols;C:\IISDebugLabs\symbols\Pub and hit Enter
b) Using the ".sympath+" command:
0:008> .sympath+ SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols
0:008> .sympath+ C :\IISDebugLabs\symbols\Pub (where public symbols of badFil.dll are located)
Symbol search path is:SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols; C :\IISDebugLabs\symbols\Pub
Note: For more information on setting the symbol path for windbg, please refer to the debugger documentation for the .sympath command. This will describe
the "downstream store" feature that lets you download symbols to a symbol server of your own.
Visit also http://www.microsoft.com/whdc/ddk/debugging/symbols.mspx
The stack backtrace of the thread that AV'd would look like:
0:008>kvn50
# ChildEBP RetAddr Args to Child
00 0118fbf4 65f0354f 00000001 00004000 08000002 badfil!doAV1+0x9
01 0118fc1c 65f034bc 000003e8 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d
02 0118fc50 65f02cad 0118fedc 0118fedc 00e74618 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x4c
03 0118fca8 65f02ac1 006a8fb8 0000019b 0118fcd4 w3svc!HTTP_REQUEST::Parse+0x380
04 0118fcec 65f029b5 006a8fb8 0000019b 0118fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x36
05 0118fd2c 65f028f8 0118fea4 0118fedc 0118feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4
06 0118fd4c 65f027b3 0118fea4 0118fedc 0118feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a
07 0118fec4 65f020d7 0118fedc 01093568 0118ff20 w3svc!HTTP_REQUEST::DoWork+0x99
08 0118fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa
09 0118ff08 65f01aa5 0118ff20 77e873b2 01093568 w3svc!CreateClient+0x7b
0a 0118ff4c 6d701443 01093568 0000019b 00104cd0 w3svc!W3OnConnectEx+0x118
0b 0118ff80 6d702203 00000000 0000019b 0109356c ISATQ!AtqpProcessContext+0x23a
0c 0118ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8
0d 0118ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52
Now, with the public symbols lined up, we see that the last call that AV'd was to the function "doAV1" that was called in reponse to a notification
"PreProcHeader".
Again, In this stack backtrace, it is clear that the culprit is "badFil.dll". So it is enough to tell the customer to contact the dll vendor to fix it. If it is an in-house component and the customer would like help
from Microsoft, they should open an advisory case to troubleshoot the issue further.
3) with Private Symbols "badFil.dll"
Point the debugger to use the private symbols
For company wide internal symbols, use \\symbols\symbols
For the component private symbols, use C:\IISDebugLabs\Symbols\Pri
Verify the component symbols are linet up.
0:008> lmv mbadfil
start end module name
015a0000 015a6000 badfil (pdb symbols) C:\IISDebugLabs\Symbols\Pri
Loaded symbol image file: badfil.dll
Image path: C:\IISDebugLabs\www\App\badfil.dll
Timestamp: Wed May 28 01:11:01 2003 (3ED444E5)
Checksum: 0000D9A0
File version: 5.0.0.0
Product version: 5.0.0.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
CompanyName: Microsoft Corporation
ProductName: Internet Information Services
InternalName: badFIL.dll
OriginalFilename: badFIL.dll
ProductVersion: 5.00.0984
FileVersion: 5.00.0984
FileDescription: badFIL Filter DLL
LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999
The stack backtrace of the thread that AV'd would look like:
0:008>kvn50
# ChildEBP RetAddr Args to Child
00 0118fbf4 65f0354f 00000001 00004000 08000002 badfil!doAV1+0x9 (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 512]
01 0118fc1c 65f034bc 000003e8 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpfilt.cxx @ 968]
02 0118fc50 65f02cad 0118fedc 0118fedc 00e74618 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x4c (FPO: [Non-Fpo]) (CONV: thiscall) [.\filter.hxx @ 473]
03 0118fca8 65f02ac1 006a8fb8 0000019b 0118fcd4 w3svc!HTTP_REQUEST::Parse+0x380 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 1497]
04 0118fcec 65f029b5 006a8fb8 0000019b 0118fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x36 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 1289]
05 0118fd2c 65f028f8 0118fea4 0118fedc 0118feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 634]
06 0118fd4c 65f027b3 0118fea4 0118fedc 0118feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 253]
07 0118fec4 65f020d7 0118fedc 01093568 0118ff20 w3svc!HTTP_REQUEST::DoWork+0x99 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 639]
08 0118fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]
09 0118ff08 65f01aa5 0118ff20 77e873b2 01093568 w3svc!CreateClient+0x7b (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310]
0a 0118ff4c 6d701443 01093568 0000019b 00104cd0 w3svc!W3OnConnectEx+0x118 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]
0b 0118ff80 6d702203 00000000 0000019b 0109356c ISATQ!AtqpProcessContext+0x23a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]
0c 0118ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8 (FPO: [EBP 0x0118ffec] [1,4,4]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127]
0d 0118ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]
As mentioned above, badFil.dll AV'd while acting on the "PreProcHeader" notification, we know that because
of frame 02 (w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters). It is helpful sometimes to exact information like what request
caused this filter to AV.
In this special case, using the debugger "kp" command will show the exact request.
0:008> kpn #
ChildEBP RetAddr
00 0118fbf4 65f0354f badfil!doAV1(int Variation = 0x65f0354f)+0x9 [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 512]
01 0118fc1c 65f034bc w3svc!FILTER_LIST::NotifyFilters(HTTP_FILTER* pFilter = 000003e8 , unsigned long NotificationType = 0x4000, _HTTP_FILTER_CONTEXT* phfc = 00000002 , void* NotificationData = 0118fc3c , int* pfFinished = 0118fedc , int fNotifyAll = 0)+0x9d [D:\nt\private\inet\iis\svcs\w3\server\httpfilt.cxx @ 968]
02 0118fc50 65f02cad w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters(int* pfFinished = 0118fedc )+0x4c [.\filter.hxx @ 473]
03 0118fca8 65f02ac1 w3svc!HTTP_REQUEST::Parse(char* pchRequest = 006a8fb8 "GET /IISDebugLabs/Site/fil_av1.htm?sf=av1_v1 HTTP/1.1???", unsigned long cbData = 0x19b, unsigned long* pcbExtraData = 0118fcd4 , int* pfHandled = 0118fce8 , int* pfFinished = 0118fedc )+0x380 [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 1497]
04 0118fcec 65f029b5 w3svc!HTTP_REQ_BASE::OnCompleteRequest(char* pchRequest = 006a8fb8 "GET /IISDebugLabs/Site/fil_av1.htm?sf=av1_v1 HTTP/1.1???", unsigned long cbData = 0x19b, int* pfFinished = 0118fedc , int* pfContinueProcessingRequest = 0118feb8 )+0x36 [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 1289]
05 0118fd2c 65f028f8 w3svc!HTTP_REQ_BASE::UnWrapRequest(int* pfCompleteRequest = 0118fea4 , int* pfFinished = 0118fedc , int* pfContinueProcessingRequest = 0118feb8 )+0x1f4 [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 634]
06 0118fd4c 65f027b3 w3svc!HTTP_REQ_BASE::OnFillClientReq(int* pfCompleteRequest = 0118fea4 , int* pfFinished = 0118fedc , int* pfContinueProcessingRequest = 0118feb8 )+0x7a [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 253]
07 0118fec4 65f020d7 w3svc!HTTP_REQUEST::DoWork(int* pfFinished = 0118fedc )+0x99 [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 639]
08 0118fee4 65f01b47 w3svc!CLIENT_CONN::DoWork(unsigned long BytesWritten = 0, unsigned long CompletionStatus = 0, int fIOCompletion = 0)+0x1aa [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]
09 0118ff08 65f01aa5 w3svc!CreateClient(_CLIENT_CONN_PARAMS* ClientParam = 0118ff20 )+0x7b [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310] 0a 0118ff4c 6d701443 w3svc!W3OnConnectEx(void* patqContext = 01093568 , unsigned long cbWritten = 0x19b, unsigned long err = 0x104cd0, _OVERLAPPED* lpo = 00e73bf0 )+0x118 [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]
0b 0118ff80 6d702203 ISATQ!AtqpProcessContext(ATQ_CONTEXT* pAtqContext = 00000000 , unsigned long cbWritten = 0x19b, _OVERLAPPED* lpo = 0109356c , int fRet = 1)+0x23a [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]
0c 0118ffb4 77e887dd ISATQ!AtqPoolThread(unsigned long* param = abcdef01 )+0x1a8 [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127]
0d 0118ffec 00000000 KERNEL32!BaseThreadStart(* lpStartAddress = 6d7091f7 , void* lpParameter = abcdef01 )+0x52 [D:\nt\private\windows\base\client\support.c @ 460]
Notice in frame OO the line of code as well as the name and location of the source code. Going back to the source code in badFil.cpp:
Line 512 looks like
DWORD doAV1(int Variation)
{
if (1 == Variation)
{
SYSTEMTIME* pSysTime = NULL;
pSysTime->wYear = 2003; // av right here. --Line 512:
}
return 1;
}