1) Without symbols
a) The faulting Stack
The stack backtrace of the thread that AV'd would look like:
0:008> kvn50
*** ERROR: Symbol file could not be found. Defaulted to export symbols for badfil.dll -
# ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0118f5d4 015a1a55 00000000 015a1670 00000001 KERNEL32!GetSystemTime+0x34
*** ERROR: Symbol file could not be found. Defaulted to export symbols for w3svc.dll -
01 0118fbf4 65f0354f 00000001 00004000 08000002 badfil!HttpFilterProc+0x553
02 0118fc1c 65f034bc 000003e9 00004000 00000002 w3svc!STR::Copy+0x11a
03 0118fc50 65f02cad 0118fedc 0118fedc 00e746a0 w3svc!STR::Copy+0x87
04 0118fca8 65f02ac1 006a8fb8 0000019b 0118fcd4 w3svc!CLIENT_CONN::OnSessionStartup+0xa69
05 0118fcec 65f029b5 006a8fb8 0000019b 0118fedc w3svc!CLIENT_CONN::OnSessionStartup+0x87d
06 0118fd2c 65f028f8 0118fea4 0118fedc 0118feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x771
07 0118fd4c 65f027b3 0118fea4 0118fedc 0118feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x6b4
08 0118fec4 65f020d7 0118fedc 01093568 0118ff20 w3svc!CLIENT_CONN::OnSessionStartup+0x56f
09 0118fee4 65f01b47 00000000 00000000 00000000 w3svc!HTTP_HEADERS::Reset+0x311
0a 0118ff08 65f01aa5 0118ff20 77e873b2 01093568 w3svc+0x1b47
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ISATQ.DLL -
0b 0118ff4c 6d701443 01093568 0000019b 00104be0 w3svc+0x1aa5
0c 0118ff80 6d702203 00000000 0000019b 0109356c ISATQ!AtqReadSocket+0x228
0d 77deec86 ece868ff fb6877dc 6477db19 000000a1 ISATQ!AtqWriteSocket+0x495
0e 6aec8b55 00000000 00000000 00000000 00000000 0xece868ff
In this stack backtrace, notice that the AV occured in "kernel32.dll". Does this mean that the culprit is "kernel32.dll" ? Actually No!
It is most likely because of some bad parameters passed to an API call in kernel32. to invistigate this further, we need to line up the symbols properly to see
what function AV'd in frame 00.
b) The regiters values
0:008> r
eax=00000000 ebx=00000003 ecx=000007d3 edx=0000001a esi=78002041 edi=00e77820
eip=77e87b14 esp=0118f5bc ebp=0118f5d4 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206
KERNEL32!GetSystemTime+34:
77e87b14 668908 mov [eax],cx ds:0023:00000000=????
c) The disassmbly code
0:008> u eip
KERNEL32!GetSystemTime+34:
77e87b14 668908 mov [eax],cx
77e87b17 668b4dea mov cx,[ebp-0x16]
77e87b1b 66894802 mov [eax+0x2],cx
77e87b1f 668b4df6 mov cx,[ebp-0xa]
77e87b23 66894804 mov [eax+0x4],cx
77e87b27 668b4dec mov cx,[ebp-0x14]
77e87b2b 66894806 mov [eax+0x6],cx
77e87b2f 668b4dee mov cx,[ebp-0x12]
2) With symbols including public symbols of "badFil.dll"
First, we need to add the public symbol path(company wide symbols) as well as symbols of our dll
a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols;C:\IISDebugLabs\symbols\Pub and hit Enter
b) Using the ".sympath+" command:
0:008> .sympath+ SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols
0:008> .sympath+ C :\IISDebugLabs\symbols\Pub (where public symbols of badFil.dll are located)
Symbol search path is:SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols; C :\IISDebugLabs\symbols\Pub
Note: For more information on setting the symbol path for windbg, please refer to the debugger documentation for the .sympath command. This will describe
the "downstream store" feature that lets you download symbols to a symbol server of your own.
The stack backtrace of the thread that AV'd would look like:
0:008> kvn50
# ChildEBP RetAddr Args to Child
00 0118f5d4 015a1a55 00000000 015a1670 00000001 KERNEL32!GetSystemTime+0x34
01 0118fbf4 65f0354f 00000001 00004000 08000002 badfil!doAV2+0xf
02 0118fc1c 65f034bc 000003e9 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d
03 0118fc50 65f02cad 0118fedc 0118fedc 00e746a0 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x4c
04 0118fca8 65f02ac1 006a8fb8 0000019b 0118fcd4 w3svc!HTTP_REQUEST::Parse+0x380
05 0118fcec 65f029b5 006a8fb8 0000019b 0118fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x36
06 0118fd2c 65f028f8 0118fea4 0118fedc 0118feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4
07 0118fd4c 65f027b3 0118fea4 0118fedc 0118feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a
08 0118fec4 65f020d7 0118fedc 01093568 0118ff20 w3svc!HTTP_REQUEST::DoWork+0x99
09 0118fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa
0a 0118ff08 65f01aa5 0118ff20 77e873b2 01093568 w3svc!CreateClient+0x7b
0b 0118ff4c 6d701443 01093568 0000019b 00104be0 w3svc!W3OnConnectEx+0x118
0c 0118ff80 6d702203 00000000 0000019b 0109356c ISATQ!AtqpProcessContext+0x23a
0d 0118ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8
0e 0118ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52
Now, with the public symbols lined up, we see that the last call that AV'd was to the function "GetSystemTime" in "kernel32.dll". this API was called from
the function "doAV2" in "badFil.dll".
Looking in MSDN for the API "GetSystemTime" shows:
The GetSystemTime function retrieves the current system date and time. The system time is expressed in Coordinated Universal Time (UTC).
void GetSystemTime(
LPSYSTEMTIME lpSystemTime
);
So it looks like the GetSystemTime API takes as a parameter a pointer to a SYSTEMTIME structure to receive
the current system date and time which could not be NULL.
Advanced Technique
You may want to look closely at the "GetSystemTime" function.
The source code for this function is shown below:
VOID WINAPI GetSystemTime( LPSYSTEMTIME lpSystemTime)
..... .
....
.....
RtlTimeToTimeFields(&SystemTime,&TimeFields);
lpSystemTime->wYear = TimeFields.Year ;
lpSystemTime->wMonth = TimeFields.Month ;
lpSystemTime->wDayOfWeek = TimeFields.Weekday ;
lpSystemTime->wDay = TimeFields.Day ;
lpSystemTime->wHour = TimeFields.Hour ;
lpSystemTime->wMinute = TimeFields.Minute ;
lpSystemTime->wSecond = TimeFields.Second ;
lpSystemTime->wMilliseconds = TimeFields.Milliseconds; }
The first and only parameter to this function is a long pointer to the system time (LPSYSTEMTIME).
Back to the dump file, this first parameter is NULL as shown below:
00 0118f5d4 015a1a55 00000000 015a1670 00000001 KERNEL32!GetSystemTime+0x34
Since "lpSystemTime" is NULL, any attempt to access its members will result in an AV.
Based on this, we can say that the culprit is "badext.dll". So it is enough to tell the customer to contact the dll vendor to fix it. If it is an in-house component and the customer would like help
from Microsoft, they should open an advisory case to troubleshoot the issue further.
Again, In this stack backtrace, it is clear that the culprit is "badFil.dll". So it is enough to tell the customer to contact the dll vendor to fix it. If it is an in-house component and the customer would like help
from Microsoft, they should open an advisory case to troubleshoot the issue further.
3) with Private Symbols "badFil.dll"
Point the debugger to use the private symbols
For company wide internal symbols, use \\symbols\symbols
For the component private symbols, use C:\IISDebugLabs\Symbols\Pri
Verify the component symbols are lined up.
0:008> lmv mbadfil
start end module name
015a0000 015a6000 badfil (pdb symbols) C:\IISDebugLabs\Symbols\Pri
Loaded symbol image file: badfil.dll
Image path: C:\IISDebugLabs\www\App\badfil.dll
Timestamp: Wed May 28 01:11:01 2003 (3ED444E5)
Checksum: 0000D9A0
File version: 5.0.0.0
Product version: 5.0.0.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
CompanyName: Microsoft Corporation
ProductName: Internet Information Services
InternalName: badFIL.dll
OriginalFilename: badFIL.dll
ProductVersion: 5.00.0984
FileVersion: 5.00.0984
FileDescription: badFIL Filter DLL
LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999
The stack backtrace of the thread that AV'd would look like:
0:008> kvn50
# ChildEBP RetAddr Args to Child
00 0118f5d4 015a1a55 00000000 015a1670 00000001 KERNEL32!GetSystemTime+0x34 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\datetime.c @ 177]
01 0118fbf4 65f0354f 00000001 00004000 08000002 badfil!doAV2+0xf (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 526]
02 0118fc1c 65f034bc 000003e9 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpfilt.cxx @ 968]
03 0118fc50 65f02cad 0118fedc 0118fedc 00e746a0 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x4c (FPO: [Non-Fpo]) (CONV: thiscall) [.\filter.hxx @ 473]
04 0118fca8 65f02ac1 006a8fb8 0000019b 0118fcd4 w3svc!HTTP_REQUEST::Parse+0x380 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 1497]
05 0118fcec 65f029b5 006a8fb8 0000019b 0118fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x36 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 1289]
06 0118fd2c 65f028f8 0118fea4 0118fedc 0118feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 634]
07 0118fd4c 65f027b3 0118fea4 0118fedc 0118feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 253]
08 0118fec4 65f020d7 0118fedc 01093568 0118ff20 w3svc!HTTP_REQUEST::DoWork+0x99 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 639]
09 0118fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]
0a 0118ff08 65f01aa5 0118ff20 77e873b2 01093568 w3svc!CreateClient+0x7b (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310]
0b 0118ff4c 6d701443 01093568 0000019b 00104be0 w3svc!W3OnConnectEx+0x118 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]
0c 0118ff80 6d702203 00000000 0000019b 0109356c ISATQ!AtqpProcessContext+0x23a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]
0d 0118ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8 (FPO: [EBP 0x0118ffec] [1,4,4]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127]
0e 0118ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]
As mentioned above, badFil.dll AV'd while acting on the "PreProcHeader" notification, we know that because
of frame 02 (w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters). It is helpful sometimes to exact information like what request
caused this filter to AV.
In this special case, using the debugger "kp" command will show the exact request.
0:008> kpn #
ChildEBP RetAddr 00 0118f5d4 015a1a55 KERNEL32!GetSystemTime(struct _SYSTEMTIME * lpSystemTime = 0x00000000)+0x34 [D:\nt\private\windows\base\client\datetime.c @ 177]
01 0118fbf4 65f0354f badfil!doAV2(int Variation = 1710241103)+0xf [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 526]
02 0118fc1c 65f034bc w3svc!FILTER_LIST::NotifyFilters(class HTTP_FILTER * pFilter = 0x000003e9, unsigned long NotificationType = 0x4000, struct _HTTP_FILTER_CONTEXT * phfc = 0x00000002, void * NotificationData = 0x0118fc3c, int * pfFinished = 0x0118fedc, int fNotifyAll = 0)+0x9d [D:\nt\private\inet\iis\svcs\w3\server\httpfilt.cxx @ 968]
03 0118fc50 65f02cad w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters(int * pfFinished = 0x0118fedc)+0x4c [.\filter.hxx @ 473]
04 0118fca8 65f02ac1 w3svc!HTTP_REQUEST::Parse(char * pchRequest = 0x006a8fb8 "GET /IISDebugLabs/Site/fil_av2.htm?sf=av2_v1 HTTP/1.1???", unsigned long cbData = 0x19b, unsigned long * pcbExtraData = 0x0118fcd4, int * pfHandled = 0x0118fce8, int * pfFinished = 0x0118fedc)+0x380 [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 1497]
05 0118fcec 65f029b5 w3svc!HTTP_REQ_BASE::OnCompleteRequest(char * pchRequest = 0x006a8fb8 "GET /IISDebugLabs/Site/fil_av2.htm?sf=av2_v1 HTTP/1.1???", unsigned long cbData = 0x19b, int * pfFinished = 0x0118fedc, int * pfContinueProcessingRequest = 0x0118feb8)+0x36 [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 1289]
06 0118fd2c 65f028f8 w3svc!HTTP_REQ_BASE::UnWrapRequest(int * pfCompleteRequest = 0x0118fea4, int * pfFinished = 0x0118fedc, int * pfContinueProcessingRequest = 0x0118feb8)+0x1f4 [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 634]
07 0118fd4c 65f027b3 w3svc!HTTP_REQ_BASE::OnFillClientReq(int * pfCompleteRequest = 0x0118fea4, int * pfFinished = 0x0118fedc, int * pfContinueProcessingRequest = 0x0118feb8)+0x7a [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 253]
08 0118fec4 65f020d7 w3svc!HTTP_REQUEST::DoWork(int * pfFinished = 0x0118fedc)+0x99 [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 639] 09 0118fee4 65f01b47 w3svc!CLIENT_CONN::DoWork(unsigned long BytesWritten = 0, unsigned long CompletionStatus = 0, int fIOCompletion = 0)+0x1aa [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]
0a 0118ff08 65f01aa5 w3svc!CreateClient(struct _CLIENT_CONN_PARAMS * ClientParam = 0x0118ff20)+0x7b [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310] 0b 0118ff4c 6d701443 w3svc!W3OnConnectEx(void * patqContext = 0x01093568, unsigned long cbWritten = 0x19b, unsigned long err = 0x104be0, struct _OVERLAPPED * lpo = 0x00e73bf0)+0x118 [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]
0c 0118ff80 6d702203 ISATQ!AtqpProcessContext(struct ATQ_CONTEXT * pAtqContext = 0x00000000, unsigned long cbWritten = 0x19b, struct _OVERLAPPED * lpo = 0x0109356c, int fRet = 1)+0x23a [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]
0d 0118ffb4 77e887dd ISATQ!AtqPoolThread(unsigned long * param = 0xece868ff)+0x1a8 [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127] 0e 0118ffec 00000000 KERNEL32!BaseThreadStart( * lpStartAddress = 0x6d7091f7, void * lpParameter = 0xabcdef01)+0x52 [D:\nt\private\windows\base\client\support.c @ 460]
0e 0118ffec 00000000 KERNEL32!BaseThreadStart( * lpStartAddress = 0x6d7091f7, void * lpParameter = 0xabcdef01)+0x52 [D:\nt\private\windows\base\client\support.c @ 460]
Notice in frame 01 the line of code as well as the name and location of the source code. Going back to the source code in badFil.cpp:
Line 526 looks like
DWORD doAV2(int Variation)
{
if (1 == Variation)
{
SYSTEMTIME* pSysTime = NULL;
GetSystemTime( pSysTime ); // Line 526 would av in GetSystemTime
}
return 1;
}