1) Without symbols
a) The faulting Stack
The stack backtrace of the thread that caused the Stack Overflow would look like:
0:000> kvn50
# ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0006ff70 01001daf 00000001 00232468 00232d78 badcgi+0x1bc2
*** ERROR: Symbol file could not be found. Defaulted to export symbols for KERNEL32.dll -
01 0006ffc0 77e9ca90 77f82b95 77f83920 7ffdf000 badcgi+0x1daf
02 0006fff0 00000000 01001cb0 00000000 000000c8 KERNEL32!CreateProcessW+0x15a6
Keep in mind that we are here debugging a custom CGI application that misbehaved. The web application relying on this
CGI will eventually not run properly. So, in this stack backtrace, it is clear that the culprit is "badCgi.exe".
So it is enough to tell the customer to contact the CGI vendor to fix it. If it is an in-house component and the customer
would like help from Microsoft, they should open an advisory case to troubleshoot the issue further.
0:000> .lastevent Last event:
8b8.45c: Stack overflow - code c00000fd (!!! second chance !!!)
b) The regiters values
0:000> r
eax=000723d0 ebx=00000003 ecx=0006ffb0 edx=00070608 esi=00070000 edi=000723d0
eip=01001bc2 esp=00033000 ebp=0006ff70 iopl=0 nv up ei ng nz ac po nc
cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000296
badcgi+1bc2:
01001bc2 ff742408 push dword ptr [esp+0x8] ss:0023:00033008=00000078
c) The disassmbly code
0:000> u eip
badcgi+1bc2:
01001bc2 ff742408 push dword ptr [esp+0x8]
01001bc6 ff742408 push dword ptr [esp+0x8]
01001bca e8f3ffffff call badcgi+0x1bc2 (01001bc2)
01001bcf c20800 ret 0x8
01001bd2 8b44240c mov eax,[esp+0xc]
01001bd6 85c0 test eax,eax
01001bd8 740e jz badcgi+0x1be8 (01001be8)
01001bda 8b542404 mov edx,[esp+0x4]
2) With symbols including public symbols of "badCgi.exe"
First, we need to add the symbol path(company wide symbols) as well as symbols of our exe
0:000> .sympath+ SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols
0:000> .sympath+ C :\IISDebugLabs\symbols\Pub (where public symbols of badFil.dll are located)
Symbol search path is:SRV*C:\LabSymbols*http://msdl.microsoft.com/download/symbols; C :\IISDebugLabs\symbols\Pub
The stack backtrace of the thread that caused the Stack Overflow would look like:
0:000> kvn50
# ChildEBP RetAddr Args to Child
00 0006ff70 01001daf 00000001 00232468 00232d78 badcgi!FindChar
01 0006ffc0 77e9ca90 77f82b95 77f83920 7ffdf000 badcgi!mainCRTStartup+0xff
02 0006fff0 00000000 01001cb0 00000000 000000c8 KERNEL32!BaseProcessStart+0x3d
So basically here, we can confirm that the call to the function "FindChar()" caused the stack overflow. This enough to tell
the customer that the problem is coming from the function "FindChar()" in the custom CGI executable.
Advanced Technique
To see the stack overflow caused by this thread, let issue the command "dds esp" to display the raw stack at the stack pointer region.
0:000> dds esp
00033000 01001bcf badcgi!FindChar+0xd
00033004 000723d0
00033008 00000078
0003300c 01001bcf badcgi!FindChar+0xd
00033010 000723d0
00033014 00000078
00033018 01001bcf badcgi!FindChar+0xd
0003301c 000723d0
00033020 00000078
00033024 01001bcf badcgi!FindChar+0xd
00033028 000723d0
0003302c 00000078
00033030 01001bcf badcgi!FindChar+0xd
00033034 000723d0
00033038 00000078
0003303c 01001bcf badcgi!FindChar+0xd
00033040 000723d0
00033044 00000078
00033048 01001bcf badcgi!FindChar+0xd
0003304c 000723d0
00033050 00000078
00033054 01001bcf badcgi!FindChar+0xd
00033058 000723d0
0003305c 00000078
00033060 01001bcf badcgi!FindChar+0xd
00033064 000723d0
00033068 00000078
0003306c 01001bcf badcgi!FindChar+0xd
00033070 000723d0
00033074 00000078
00033078 01001bcf badcgi!FindChar+0xd
0003307c 000723d0
In IIS4 and 5, each thread has a stack of a maximum size of 256k. Please refer to the following KB:
PRB: Stack Size Now 256 K in Internet Information Server 4.0 WGID:246 ID: 189888.
KB.EN-US CREATED: 1998-07-21 MODIFIED: 2001-07-16
We will be using the !teb command (Thread Environment Variable) to check the stack size.
0:000> !teb
TEB at 7FFDE000
ExceptionList: 6ffb0
Stack Base: 70000
Stack Limit: 31000
SubSystemTib: 0
FiberData: 1e00
ArbitraryUser: 0
Self: 7ffde000
EnvironmentPtr: 0
ClientId: 8b8.45c
Real ClientId: 8b8.45c
RpcHandle: 0
Tls Storage: 0
PEB Address: 7ffdf000
LastErrorValue: 0
LastStatusValue: c0000034
Count Owned Locks:0
HardErrorsMode: 0
Finding the current size:
0:000> ?(70000 - 31000)
Evaluate expression: 258048 = 0003f000
0:000> ? (258048 /1024)
Evaluate expression: 594 = 00000252
This shows that the current size now is 252 KB which is almost the limit
3) With Private Symbols "badCgi,exe"
First, we need to add the private symbol path(company wide symbols) as well as public symbols of our exe
a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pri a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pri
0:000> kvn50
# ChildEBP RetAddr Args to Child
00 0006ff70 01001daf 00000001 00232468 00232d78 badcgi!FindChar (CONV: stdcall) [C:\build\nT\PRIVATE\inet\iis\svcs\w3\gateways\cgiDBG\badcgi.cpp @ 639]
01 0006ffc0 77e9ca90 77f82b95 77f83920 7ffdf000 badcgi!mainCRTStartup+0xff (CONV: cdecl) [crtexe.c @ 338]
02 0006fff0 00000000 01001cb0 00000000 000000c8 KERNEL32!BaseProcessStart+0x3d (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 504]
Looking at the source code:
char* FindChar(char* sz, char c)
{
char* ch = NULL;
{
ch = sz++;
sz = FindChar( ch, c ); //Line 369
}
return sz;
}
As you see this recursive function will never stop execution becuase there is no stopping condition.
In real life this kind of problems would happen if the stopping condition is never met for some reason.
Advanced Technique
Remember that our web application is running in the IIS out of process pooled application (dllhost.exe).
The CGI executable is loaded in its own memory space and it is basically spawned by "inetinfo.exe". So for all http requests involving directly the the CGI
without passing via another extension will be processed by both inetinfo and the CGI executable without passing via dllhost. (for more information, please
refer to the class docs Chapter:...).
To get a feeling on how the CGI processing is made, let do the following:
1) In the command prompt, switch to the folder where the debugger is installed and run the following command:
c:\ debuggers>windbg -pn inetinfo.exe
2) In WinDbg, let set a breakpoint in CreateProcess function of Kernel32
In the windbg command line, type in "bp KERNEL32!CreateProcessA"
3) In Windbg command line, type in "g" and hit "Enter"
4) Click the "AV CGI" button in this page.
Notice that the debugger breaks, the command line is now available to run commands.
The stack backtrace of the thread where the break point was hit looks like:
0:010> kp
ChildEBP RetAddr
0136ee98 65f1be75 KERNEL32!CreateProcessA(char* lpApplicationName = 00000000 "", char* lpCommandLine = 0136f4b0 ""C:\IISDebugLabs\www\App\badCgi.exe" ", _SECURITY_ATTRIBUTES* lpProcessAttributes = 00000000 , _SECURITY_ATTRIBUTES* lpThreadAttributes = 00000000 , int bInheritHandles = 1, unsigned long dwCreationFlags = 0xc, void* lpEnvironment = 0136eed4 , char* lpCurrentDirectory = 0136fba8 "C:\IISDebugLabs\www", _STARTUPINFOA* lpStartupInfo = 0136f5b4 , _PROCESS_INFORMATION* lpProcessInformation = 0136f658 ) [D:\nt\private\windows\base\client\process.c @ 715]
0136f678 65f1b925 w3svc!HTTP_REQUEST::ProcessCGI(EXEC_DESCRIPTOR* pExec = 00e9eb2c , STR* pstrPath = 00000000 , STR* pstrWorkingDir = 0136fd10 , int* pfHandled = 0136feb0 , int* pfFinished = 0136fedc , STR* pstrCmdLine = 0136f5fc )+0x4c3 [D:\nt\private\inet\iis\svcs\w3\server\cgi.cxx @ 1179]
0136fd48 65f2a610 w3svc!HTTP_REQUEST::ProcessGateway(EXEC_DESCRIPTOR* pExec = 00e9eb2c , int* pfHandled = 0136feb0 , int* pfFinished = 0136fedc , int fTrusted = 0)+0x34e [D:\nt\private\inet\iis\svcs\w3\server\cgi.cxx @ 919]
0136fec4 65f020d7 w3svc!HTTP_REQUEST::DoWork(int* pfFinished = 0136fedc )+0x43f [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 839]
0136fee4 65f01b47 w3svc!CLIENT_CONN::DoWork(unsigned long BytesWritten = 0, unsigned long CompletionStatus = 0, int fIOCompletion = 0)+0x1aa [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]
0136ff08 65f01aa5 w3svc!CreateClient(_CLIENT_CONN_PARAMS* ClientParam = 0136ff20 )+0x7b [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310]
0136ff4c 6d701443 w3svc!W3OnConnectEx(void* patqContext = 012742d8 , unsigned long cbWritten = 0xe4, unsigned long err = 0x1131e8, _OVERLAPPED* lpo = 00e897b8 )+0x118 [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]
0136ff80 6d702203 ISATQ!AtqpProcessContext(ATQ_CONTEXT* pAtqContext = 00000000 , unsigned long cbWritten = 0xe4, _OVERLAPPED* lpo = 012742dc , int fRet = 1)+0x23a [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]
0136ffb4 77e887dd ISATQ!AtqPoolThread(unsigned long* param = abcdef01 )+0x1a8 [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127]
0136ffec 00000000 KERNEL32!BaseThreadStart(* lpStartAddress = 6d7091f7 , void* lpParameter = abcdef01 )+0x52 [D:\nt\private\windows\base\client\support.c @ 460]
Note that the second parameter to CreateProcess is the command line to trun that process which is in this case "C:\IISDebugLabs\www\App\badCgi.exe"
Let try to get the whole HTTP request that w3svc received.
For this, let dump the EXEC_DESCRIPTOR class.
0:010> dt EXEC_DESCRIPTOR 00e9eb2c
+0x000 _pstrURL : 0x00e9e370
+0x004 _pstrPhysicalPath : 0x00e9e5e0
+0x008 _pstrUnmappedPhysicalPath : 0x00e9e610
+0x00c _pstrGatewayImage : 0x00e9eac0
+0x010 _pstrPathInfo : 0x00e9e400
+0x014 _pstrURLParams : 0x00e9e3d0
+0x018 _pdwScriptMapFlags : 0x00e9eabc -> 0
+0x01c _pGatewayType : 0x00e9e060 300002 ( GATEWAY_CGI )
+0x020 _pRequest : 0x00e9e008
+0x024 _hChildEvent : (null)
+0x028 _fMustWaitForChildEvent : 0 +0x02c _pParentWamRequest : (null)
+0x030 _dwExecFlags : 0
+0x034 _pMetaData : 0x00e9ecd8
+0x038 _pPathInfoMetaData : (null)
+0x03c _pPathInfoURIBlob : (null)
+0x040 _pAppPathURIBlob : (null)
Now let dump the HTTP_REQUEST class.
0:010> dt HTTP_REQUEST 0x00e9e008
+0x000 __VFN_table : 0x65f08680
+0x008 _cFilesSent : 0
+0x00c _cFilesReceived : 0
......
......
+0x9a8 _bufClientRequest : BUFFER
......
......
+0xbac _pFileNameLock : (null)
+0xbb0 _fPossibleDefaultExecute : 0y0
=65f465b0 HTTP_REQUEST::sm_GetInfoFuncs : [26] 0x65f26a87 w3svc!HTTP_REQUEST::GetInfoA+0
+0xbb4 _strDLCString : STR
The "_bufClientRequest " points to the buffer where the HTTP request is stored.
0:010> da poi(0x00e9e008 + 0x9a8 )
006ae398 "GET /IISDebugLabs/App/badCgi.exe"
006ae3b8 "?s=so1_v1 HTTP/1.1..Accept: */*."
006ae3d8 ".Accept-Language: en-us..Accept-"
006ae3f8 "Encoding: gzip, deflate..User-Ag"
006ae418 "ent: Mozilla/4.0 (compatible; MS"
006ae438 "IE 6.0; Windows NT 5.0)..Host: l"
006ae458 "ocalhost..Connection: Keep-Alive"
006ae478 "...."
5) In the windbg command line, type in "g" and hit "Enter"
Notice a second windbg window is displayed, this is the one attached to the CGI executable "badCgi.exe".
6) In the Windbg instance attached to "badCgi.exe" type in "q" and hit "Enter".
7) Finaly, back to the main WinDbg window, press "Ctrl-Break" to break in to the debugger, type in "q" and hit "Enter"