Debug Labs


Access Violation in External Code

This lab will demonstrate an Access Violation in External Code. In this scenarion, an AV in "kernel32.dll"

 

1- Lab Setup
	

Note: This lab assumes that you have already installed the necessary lab files to run it.

1-Install the ISAPI Filter

 

- Right Click the Server Name

- Click "Properties"

- In the "WWW Service" Master Properties, click "Edit"

- In the "ISAPI Filters" tab, click "Add"

- In the "Filter Name" type in "BadFilter"

- In the "Executable", browse to C:\IISDebugLabs\www\App\badfil.dll

- Click "Apply" and click "Ok" all teh way out.

 

2- Restart the IIS Admin service

 

- Right Click the Server Name

- Click "Restart IIS" and Click "Ok"

 

3- Attach the debugger to inetinfo.exe

a) Using the GUI:

- Click "File"

- Click "Attach to Process"

- Select "inetinfo.exe" and click "OK"

b) Using the command prompt

- Switch to the folder where the debuggers is installed

- Type in the command "windbg.exe -pn inetinfo.exe" with no quotes

 

4- In windbg command line, type in "g" with no quotation marks and press Enter

 

 

2- Run Scenario

1- Click the "AV Filter" button the first time to cause a 1st chance AV in inetinfo.exe.
 Note that the debugger breaks and diplays 
	a- Comments about the exception
b- The value of the registers
c- The module where the AV happened
d- The assembly line that caused the AV
2- In the WinDbg command line, type in "g" and press "Enter", a 2nd chance AV is now encountered 

                 

 

1- Debug Annotation

 

1) Without symbols

 

a) The faulting Stack

 

The stack backtrace of the thread that AV'd would look like:

 

0:008> kvn50

*** ERROR: Symbol file could not be found. Defaulted to export symbols for badfil.dll -

# ChildEBP RetAddr Args to Child

WARNING: Stack unwind information not available. Following frames may be wrong.

00 0118f5d4 015a1a55 00000000 015a1670 00000001 KERNEL32!GetSystemTime+0x34

*** ERROR: Symbol file could not be found. Defaulted to export symbols for w3svc.dll -

01 0118fbf4 65f0354f 00000001 00004000 08000002 badfil!HttpFilterProc+0x553

02 0118fc1c 65f034bc 000003e9 00004000 00000002 w3svc!STR::Copy+0x11a

03 0118fc50 65f02cad 0118fedc 0118fedc 00e746a0 w3svc!STR::Copy+0x87

04 0118fca8 65f02ac1 006a8fb8 0000019b 0118fcd4 w3svc!CLIENT_CONN::OnSessionStartup+0xa69

05 0118fcec 65f029b5 006a8fb8 0000019b 0118fedc w3svc!CLIENT_CONN::OnSessionStartup+0x87d

06 0118fd2c 65f028f8 0118fea4 0118fedc 0118feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x771

07 0118fd4c 65f027b3 0118fea4 0118fedc 0118feb8 w3svc!CLIENT_CONN::OnSessionStartup+0x6b4

08 0118fec4 65f020d7 0118fedc 01093568 0118ff20 w3svc!CLIENT_CONN::OnSessionStartup+0x56f

09 0118fee4 65f01b47 00000000 00000000 00000000 w3svc!HTTP_HEADERS::Reset+0x311

0a 0118ff08 65f01aa5 0118ff20 77e873b2 01093568 w3svc+0x1b47

*** ERROR: Symbol file could not be found. Defaulted to export symbols for ISATQ.DLL -

0b 0118ff4c 6d701443 01093568 0000019b 00104be0 w3svc+0x1aa5

0c 0118ff80 6d702203 00000000 0000019b 0109356c ISATQ!AtqReadSocket+0x228

0d 77deec86 ece868ff fb6877dc 6477db19 000000a1 ISATQ!AtqWriteSocket+0x495

0e 6aec8b55 00000000 00000000 00000000 00000000 0xece868ff

 

 

In this stack backtrace, notice that the AV occured in "kernel32.dll". Does this mean that the culprit is "kernel32.dll" ? Actually No!

It is most likely because of some bad parameters passed to an API call in kernel32. to invistigate this further, we need to line up the symbols properly to see

what function AV'd in frame 00.

 

b) The regiters values

 

0:008> r

eax=00000000 ebx=00000003 ecx=000007d3 edx=0000001a esi=78002041 edi=00e77820

eip=77e87b14 esp=0118f5bc ebp=0118f5d4 iopl=0 nv up ei pl nz na po nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206

KERNEL32!GetSystemTime+34:

77e87b14 668908 mov [eax],cx ds:0023:00000000=????

 

c) The disassmbly code

 

0:008> u eip

KERNEL32!GetSystemTime+34:

77e87b14 668908 mov [eax],cx

77e87b17 668b4dea mov cx,[ebp-0x16]

77e87b1b 66894802 mov [eax+0x2],cx

77e87b1f 668b4df6 mov cx,[ebp-0xa]

77e87b23 66894804 mov [eax+0x4],cx

77e87b27 668b4dec mov cx,[ebp-0x14]

77e87b2b 66894806 mov [eax+0x6],cx

77e87b2f 668b4dee mov cx,[ebp-0x12]

 

2) With symbols including public symbols of "badFil.dll"

 

First, we need to add the public symbol path(company wide symbols) as well as symbols of our dll

a) Using the GUI:

- Click "File"

- Click "Symbol File Path"

- Type in SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols;C:\IISDebugLabs\symbols\Pub and hit Enter

b) Using the ".sympath+" command:

0:008> .sympath+ SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols

Symbol search path is: SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols

 

0:008> .sympath+ C :\IISDebugLabs\symbols\Pub (where public symbols of badFil.dll are located)

Symbol search path is:SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols; C :\IISDebugLabs\symbols\Pub

 

Note: For more information on setting the symbol path for windbg, please refer to the debugger documentation for the .sympath command. This will describe

the "downstream store" feature that lets you download symbols to a symbol server of your own.

 

The stack backtrace of the thread that AV'd would look like:

 

0:008> kvn50

# ChildEBP RetAddr Args to Child

00 0118f5d4 015a1a55 00000000 015a1670 00000001 KERNEL32!GetSystemTime+0x34

01 0118fbf4 65f0354f 00000001 00004000 08000002 badfil!doAV2+0xf

02 0118fc1c 65f034bc 000003e9 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d

03 0118fc50 65f02cad 0118fedc 0118fedc 00e746a0 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x4c

04 0118fca8 65f02ac1 006a8fb8 0000019b 0118fcd4 w3svc!HTTP_REQUEST::Parse+0x380

05 0118fcec 65f029b5 006a8fb8 0000019b 0118fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x36

06 0118fd2c 65f028f8 0118fea4 0118fedc 0118feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4

07 0118fd4c 65f027b3 0118fea4 0118fedc 0118feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a

08 0118fec4 65f020d7 0118fedc 01093568 0118ff20 w3svc!HTTP_REQUEST::DoWork+0x99

09 0118fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa

0a 0118ff08 65f01aa5 0118ff20 77e873b2 01093568 w3svc!CreateClient+0x7b

0b 0118ff4c 6d701443 01093568 0000019b 00104be0 w3svc!W3OnConnectEx+0x118

0c 0118ff80 6d702203 00000000 0000019b 0109356c ISATQ!AtqpProcessContext+0x23a

0d 0118ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8

0e 0118ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52

 

 

 

 

Now, with the public symbols lined up, we see that the last call that AV'd was to the function "GetSystemTime" in "kernel32.dll". this API was called from

the function "doAV2" in "badFil.dll".

 

Looking in MSDN for the API "GetSystemTime" shows:

 

GetSystemTime

The GetSystemTime function retrieves the current system date and time. The system time is expressed in Coordinated Universal Time (UTC).

 
void GetSystemTime(
  LPSYSTEMTIME lpSystemTime
);

 

So it looks like the GetSystemTime API takes as a parameter a pointer to a SYSTEMTIME structure to receive

the current system date and time which could not be NULL.

 

Advanced Technique

 

You may want to look closely at the "GetSystemTime" function.

The source code for this function is shown below:

VOID WINAPI GetSystemTime( LPSYSTEMTIME lpSystemTime)

..... .

....

.....

RtlTimeToTimeFields(&SystemTime,&TimeFields);

lpSystemTime->wYear = TimeFields.Year ;

lpSystemTime->wMonth = TimeFields.Month ;

lpSystemTime->wDayOfWeek = TimeFields.Weekday ;

lpSystemTime->wDay = TimeFields.Day ;

lpSystemTime->wHour = TimeFields.Hour ;

lpSystemTime->wMinute = TimeFields.Minute ;

lpSystemTime->wSecond = TimeFields.Second ;

lpSystemTime->wMilliseconds = TimeFields.Milliseconds; }

 

The first and only parameter to this function is a long pointer to the system time (LPSYSTEMTIME).

Back to the dump file, this first parameter is NULL as shown below:

00 0118f5d4 015a1a55 00000000 015a1670 00000001 KERNEL32!GetSystemTime+0x34

 

Since "lpSystemTime" is NULL, any attempt to access its members will result in an AV.

 

Based on this, we can say that the culprit is "badext.dll". So it is enough to tell the customer to contact the dll vendor to fix it. If it is an in-house component and the customer would like help

from Microsoft, they should open an advisory case to troubleshoot the issue further.

Again, In this stack backtrace, it is clear that the culprit is "badFil.dll". So it is enough to tell the customer to contact the dll vendor to fix it. If it is an in-house component and the customer would like help

from Microsoft, they should open an advisory case to troubleshoot the issue further.

 

3) with Private Symbols "badFil.dll"

 

 

Point the debugger to use the private symbols

For company wide internal symbols, use \\symbols\symbols

For the component private symbols, use C:\IISDebugLabs\Symbols\Pri

 

Verify the component symbols are lined up.

 

0:008> lmv mbadfil

start end module name

015a0000 015a6000 badfil (pdb symbols) C:\IISDebugLabs\Symbols\Pri

Loaded symbol image file: badfil.dll

Image path: C:\IISDebugLabs\www\App\badfil.dll

Timestamp: Wed May 28 01:11:01 2003 (3ED444E5)

Checksum: 0000D9A0

File version: 5.0.0.0

Product version: 5.0.0.0

File flags: 0 (Mask 3F)

File OS: 40004 NT Win32

File type: 2.0 Dll

File date: 00000000.00000000

CompanyName: Microsoft Corporation

ProductName: Internet Information Services

InternalName: badFIL.dll

OriginalFilename: badFIL.dll

ProductVersion: 5.00.0984

FileVersion: 5.00.0984

FileDescription: badFIL Filter DLL

LegalCopyright: Copyright (C) Microsoft Corp. 1981-1999

 

 

The stack backtrace of the thread that AV'd would look like:

 

 

 

0:008> kvn50

# ChildEBP RetAddr Args to Child

00 0118f5d4 015a1a55 00000000 015a1670 00000001 KERNEL32!GetSystemTime+0x34 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\datetime.c @ 177]

01 0118fbf4 65f0354f 00000001 00004000 08000002 badfil!doAV2+0xf (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 526]

02 0118fc1c 65f034bc 000003e9 00004000 00000002 w3svc!FILTER_LIST::NotifyFilters+0x9d (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpfilt.cxx @ 968]

03 0118fc50 65f02cad 0118fedc 0118fedc 00e746a0 w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters+0x4c (FPO: [Non-Fpo]) (CONV: thiscall) [.\filter.hxx @ 473]

04 0118fca8 65f02ac1 006a8fb8 0000019b 0118fcd4 w3svc!HTTP_REQUEST::Parse+0x380 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 1497]

05 0118fcec 65f029b5 006a8fb8 0000019b 0118fedc w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x36 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 1289]

06 0118fd2c 65f028f8 0118fea4 0118fedc 0118feb8 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 634]

07 0118fd4c 65f027b3 0118fea4 0118fedc 0118feb8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 253]

08 0118fec4 65f020d7 0118fedc 01093568 0118ff20 w3svc!HTTP_REQUEST::DoWork+0x99 (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 639]

09 0118fee4 65f01b47 00000000 00000000 00000000 w3svc!CLIENT_CONN::DoWork+0x1aa (FPO: [Non-Fpo]) (CONV: thiscall) [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]

0a 0118ff08 65f01aa5 0118ff20 77e873b2 01093568 w3svc!CreateClient+0x7b (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310]

0b 0118ff4c 6d701443 01093568 0000019b 00104be0 w3svc!W3OnConnectEx+0x118 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]

0c 0118ff80 6d702203 00000000 0000019b 0109356c ISATQ!AtqpProcessContext+0x23a (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]

0d 0118ffb4 77e887dd abcdef01 00000000 00012047 ISATQ!AtqPoolThread+0x1a8 (FPO: [EBP 0x0118ffec] [1,4,4]) (CONV: stdcall) [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127]

0e 0118ffec 00000000 6d7091f7 abcdef01 00000000 KERNEL32!BaseThreadStart+0x52 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 460]

 

As mentioned above, badFil.dll AV'd while acting on the "PreProcHeader" notification, we know that because

of frame 02 (w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters). It is helpful sometimes to exact information like what request

caused this filter to AV.

In this special case, using the debugger "kp" command will show the exact request.

 

 

0:008> kpn #

ChildEBP RetAddr 00 0118f5d4 015a1a55 KERNEL32!GetSystemTime(struct _SYSTEMTIME * lpSystemTime = 0x00000000)+0x34 [D:\nt\private\windows\base\client\datetime.c @ 177]

01 0118fbf4 65f0354f badfil!doAV2(int Variation = 1710241103)+0xf [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\filDBG\badfil.cpp @ 526]

02 0118fc1c 65f034bc w3svc!FILTER_LIST::NotifyFilters(class HTTP_FILTER * pFilter = 0x000003e9, unsigned long NotificationType = 0x4000, struct _HTTP_FILTER_CONTEXT * phfc = 0x00000002, void * NotificationData = 0x0118fc3c, int * pfFinished = 0x0118fedc, int fNotifyAll = 0)+0x9d [D:\nt\private\inet\iis\svcs\w3\server\httpfilt.cxx @ 968]

03 0118fc50 65f02cad w3svc!HTTP_FILTER::NotifyPreProcHeaderFilters(int * pfFinished = 0x0118fedc)+0x4c [.\filter.hxx @ 473]

04 0118fca8 65f02ac1 w3svc!HTTP_REQUEST::Parse(char * pchRequest = 0x006a8fb8 "GET /IISDebugLabs/Site/fil_av2.htm?sf=av2_v1 HTTP/1.1???", unsigned long cbData = 0x19b, unsigned long * pcbExtraData = 0x0118fcd4, int * pfHandled = 0x0118fce8, int * pfFinished = 0x0118fedc)+0x380 [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 1497]

05 0118fcec 65f029b5 w3svc!HTTP_REQ_BASE::OnCompleteRequest(char * pchRequest = 0x006a8fb8 "GET /IISDebugLabs/Site/fil_av2.htm?sf=av2_v1 HTTP/1.1???", unsigned long cbData = 0x19b, int * pfFinished = 0x0118fedc, int * pfContinueProcessingRequest = 0x0118feb8)+0x36 [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 1289]

06 0118fd2c 65f028f8 w3svc!HTTP_REQ_BASE::UnWrapRequest(int * pfCompleteRequest = 0x0118fea4, int * pfFinished = 0x0118fedc, int * pfContinueProcessingRequest = 0x0118feb8)+0x1f4 [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 634]

07 0118fd4c 65f027b3 w3svc!HTTP_REQ_BASE::OnFillClientReq(int * pfCompleteRequest = 0x0118fea4, int * pfFinished = 0x0118fedc, int * pfContinueProcessingRequest = 0x0118feb8)+0x7a [D:\nt\private\inet\iis\svcs\w3\server\httpio.cxx @ 253]

08 0118fec4 65f020d7 w3svc!HTTP_REQUEST::DoWork(int * pfFinished = 0x0118fedc)+0x99 [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 639] 09 0118fee4 65f01b47 w3svc!CLIENT_CONN::DoWork(unsigned long BytesWritten = 0, unsigned long CompletionStatus = 0, int fIOCompletion = 0)+0x1aa [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]

0a 0118ff08 65f01aa5 w3svc!CreateClient(struct _CLIENT_CONN_PARAMS * ClientParam = 0x0118ff20)+0x7b [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310] 0b 0118ff4c 6d701443 w3svc!W3OnConnectEx(void * patqContext = 0x01093568, unsigned long cbWritten = 0x19b, unsigned long err = 0x104be0, struct _OVERLAPPED * lpo = 0x00e73bf0)+0x118 [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]

0c 0118ff80 6d702203 ISATQ!AtqpProcessContext(struct ATQ_CONTEXT * pAtqContext = 0x00000000, unsigned long cbWritten = 0x19b, struct _OVERLAPPED * lpo = 0x0109356c, int fRet = 1)+0x23a [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]

0d 0118ffb4 77e887dd ISATQ!AtqPoolThread(unsigned long * param = 0xece868ff)+0x1a8 [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127] 0e 0118ffec 00000000 KERNEL32!BaseThreadStart( * lpStartAddress = 0x6d7091f7, void * lpParameter = 0xabcdef01)+0x52 [D:\nt\private\windows\base\client\support.c @ 460]

0e 0118ffec 00000000 KERNEL32!BaseThreadStart( * lpStartAddress = 0x6d7091f7, void * lpParameter = 0xabcdef01)+0x52 [D:\nt\private\windows\base\client\support.c @ 460]

 

Notice in frame 01 the line of code as well as the name and location of the source code. Going back to the source code in badFil.cpp:

 

Line 526 looks like

 

DWORD doAV2(int Variation)

{

if (1 == Variation)

{

SYSTEMTIME* pSysTime = NULL;

GetSystemTime( pSysTime ); // Line 526 would av in GetSystemTime

}

return 1;

}

 

 


- Microsoft Confidential -