1) Without symbols
a) Looking for the thread running the request.
The thread condidate in a slow response would most likely contain a call to "ZwDelayExecution()"
In our example, here is the thread making that call.
0 Id: 748.820 Suspend: 1 Teb: 7ffde000 Unfrozen
*** ERROR: Module load completed but symbols could not be loaded for badcgi.exe
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
0006fe3c 010018c6 00004e20 00000000 002324b0 ntdll!ZwDelayExecution+0xb
0006ff70 01001ddf 00000001 00232468 00232ce8 badcgi+0x18c6
*** ERROR: Symbol file could not be found. Defaulted to export symbols for KERNEL32.dll -
0006ffc0 77e9ca90 77f82b95 77f83920 7ffdf000 badcgi+0x1ddf 0006fff0
00000000 01001ce0 00000000 000000c8 KERNEL32!CreateProcessW+0x15a6
Basically, since only export symbols are used here, we cannot go far in analyzing this thread and finding out why this thread is slow in processing the request.
But it looks like a function in "badCgi.exe" is calling the delay execution function causing the slowness in the process since this process is only 1 thread
2) With symbols including public symbols of "badCgi.exe"
First, we need to add the private symbol path(company wide symbols) as well as public symbols of our exe a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pub a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pub
Here is how the above thread would look like with public symbols.
0:000> kvn50
# ChildEBP RetAddr Args to Child
00 0006fe1c 77e87617 00000000 0006fe34 00232570 ntdll!NtDelayExecution+0xb (FPO: [2,0,0]) [i386\usrstubs.asm @ 532]
01 0006fe3c 010018c6 00004e20 00000000 002324b0 KERNEL32!SleepEx+0x32 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\synch.c @ 1672]
02 0006ff70 01001ddf 00000001 00232468 00232ce8 badcgi!doSlow1+0x3f
03 0006ffc0 77e9ca90 77f82b95 77f83920 7ffdf000 badcgi!mainCRTStartup+0xff (CONV: cdecl) [crtexe.c @ 338]
04 0006fff0 00000000 01001ce0 00000000 000000c8 KERNEL32!BaseProcessStart+0x3d (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 504]
This thread shows that the main entry in the CGI executable "main" is calling the function "doSlow1()" which in turn is calling the win32 API "SleepEx".
The first parameter passed to "sleepEx" is the time interval in Milliseconds (Hex Format) for which thread execution is to be suspended.
To evaluate this parameter, do the following:
0:000> ? 00004e20
Evaluate expression: 20000 = 00004e20
This shows that the execution of this thread is suspended for 20,000 ms = 20 sec. This is why the response is slow.
To verify that the symbols are lined up properly for our extension, issue this command.
0:000> lmv mbadcgi
start end module name
01000000 01004000 badcgi (pdb symbols) c:\iisdebuglabs\symbols\pub\badcgi.pdb
Loaded symbol image file: badcgi.exe
Image path: C:\IISDebugLabs\www\app\badcgi.exe
Timestamp: Tue May 13 17:49:56 2003 (3EC16884) Checksum: 0000691D
3) with Private Symbols "badCgi.exe"
First, we need to add the private symbol path(company wide symbols) as well as private symbols of our exe a) Using the GUI:
- Click "File"
- Click "Symbol File Path"
- Type in SRV*C:\LabSymbols*\\symbols\symbols;C:\IISDebugLabs\symbols\Pri a
nd hit Enter
b) Using the ".sympath+" command:
.sympath+ SRV*C:\LabSymbols*\\symbols\symbols;C :\IISDebugLabs\symbols\Pri
The stack backtrace of the thread that should process the request looks like:
0:000> kvn50
# ChildEBP RetAddr Args to Child
00 0006fe1c 77e87617 00000000 0006fe34 00232570 ntdll!NtDelayExecution+0xb (FPO: [2,0,0]) [i386\usrstubs.asm @ 532]
01 0006fe3c 010018c6 00004e20 00000000 002324b0 KERNEL32!SleepEx+0x32 (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\synch.c @ 1672]
02 0006ff70 01001ddf 00000001 00232468 00232ce8 badcgi!doSlow1+0x3f (CONV: stdcall) [H:\nT\PRIVATE\inet\iis\svcs\w3\gateways\cgiDBG\badcgi.cpp @ 375]
03 0006ffc0 77e9ca90 77f82b95 77f83920 7ffdf000 badcgi!mainCRTStartup+0xff (CONV: cdecl) [crtexe.c @ 338]
04 0006fff0 00000000 01001ce0 00000000 000000c8 KERNEL32!BaseProcessStart+0x3d (FPO: [Non-Fpo]) (CONV: stdcall) [D:\nt\private\windows\base\client\support.c @ 504]
With private symbols, note the line of code in frame 02 that calls to "SleepEx" (badCgi.cpp @ 375). Lining up the source code shows the line as:
DWORD doSlow1(int Variation)
{
char *szVar = new char[10];
char *szVal = new char[10];
getQSVars (lpEcb, szVar, szVal);
int iTime; iTime = atoi( szVal );
iTime *=1000; //convert to ms since we get the time in sec
if (1 == Variation)
{
//Cause iTime delay
DWORD sl = SleepEx( iTime, FALSE ); // Line 355
DWORD dw = NoScenario(lpEcb);
}
}
Advanced Technique
Remember that our web application is running in the IIS out of process pooled application (dllhost.exe).
The CGI executable is loaded in its own memory space and it is basically spawned by "inetinfo.exe". So for all http requests involving directly the the CGI
without passing via another extension will be processed by both inetinfo and the CGI executable without passing via dllhost. (for more information, please
refer to the class docs Chapter:...).
To get a feeling on how the CGI processing is made, let do the following:
1) In the command prompt, switch to the folder where the debugger is installed and run the following command:
c:\ debuggers>windbg -pn inetinfo.exe
2) In WinDbg, let set a breakpoint in CreateProcess function of Kernel32
In the windbg command line, type in "bp KERNEL32!CreateProcessA"
3) In Windbg command line, type in "g" and hit "Enter"
4) Click the "AV CGI" button in this page.
Notice that the debugger breaks, the command line is now available to run commands.
The stack backtrace of the thread where the break point was hit looks like:
0:010> kp
ChildEBP RetAddr
0136ee98 65f1be75 KERNEL32!CreateProcessA(char* lpApplicationName = 00000000 "", char* lpCommandLine = 0136f4b0 ""C:\IISDebugLabs\www\App\badCgi.exe" ", _SECURITY_ATTRIBUTES* lpProcessAttributes = 00000000 , _SECURITY_ATTRIBUTES* lpThreadAttributes = 00000000 , int bInheritHandles = 1, unsigned long dwCreationFlags = 0xc, void* lpEnvironment = 0136eed4 , char* lpCurrentDirectory = 0136fba8 "C:\IISDebugLabs\www", _STARTUPINFOA* lpStartupInfo = 0136f5b4 , _PROCESS_INFORMATION* lpProcessInformation = 0136f658 ) [D:\nt\private\windows\base\client\process.c @ 715]
0136f678 65f1b925 w3svc!HTTP_REQUEST::ProcessCGI(EXEC_DESCRIPTOR* pExec = 00e9eb2c , STR* pstrPath = 00000000 , STR* pstrWorkingDir = 0136fd10 , int* pfHandled = 0136feb0 , int* pfFinished = 0136fedc , STR* pstrCmdLine = 0136f5fc )+0x4c3 [D:\nt\private\inet\iis\svcs\w3\server\cgi.cxx @ 1179]
0136fd48 65f2a610 w3svc!HTTP_REQUEST::ProcessGateway(EXEC_DESCRIPTOR* pExec = 00e9eb2c , int* pfHandled = 0136feb0 , int* pfFinished = 0136fedc , int fTrusted = 0)+0x34e [D:\nt\private\inet\iis\svcs\w3\server\cgi.cxx @ 919]
0136fec4 65f020d7 w3svc!HTTP_REQUEST::DoWork(int* pfFinished = 0136fedc )+0x43f [D:\nt\private\inet\iis\svcs\w3\server\httpreq.cxx @ 839]
0136fee4 65f01b47 w3svc!CLIENT_CONN::DoWork(unsigned long BytesWritten = 0, unsigned long CompletionStatus = 0, int fIOCompletion = 0)+0x1aa [D:\nt\private\inet\iis\svcs\w3\server\conn.cxx @ 781]
0136ff08 65f01aa5 w3svc!CreateClient(_CLIENT_CONN_PARAMS* ClientParam = 0136ff20 )+0x7b [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 310]
0136ff4c 6d701443 w3svc!W3OnConnectEx(void* patqContext = 012742d8 , unsigned long cbWritten = 0xe4, unsigned long err = 0x1131e8, _OVERLAPPED* lpo = 00e897b8 )+0x118 [D:\nt\private\inet\iis\svcs\w3\server\connect.cxx @ 215]
0136ff80 6d702203 ISATQ!AtqpProcessContext(ATQ_CONTEXT* pAtqContext = 00000000 , unsigned long cbWritten = 0xe4, _OVERLAPPED* lpo = 012742dc , int fRet = 1)+0x23a [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 883]
0136ffb4 77e887dd ISATQ!AtqPoolThread(unsigned long* param = abcdef01 )+0x1a8 [D:\nt\private\inet\iis\svcs\infocomm\atq\atqsupp.cxx @ 1127]
0136ffec 00000000 KERNEL32!BaseThreadStart(* lpStartAddress = 6d7091f7 , void* lpParameter = abcdef01 )+0x52 [D:\nt\private\windows\base\client\support.c @ 460]
Note that the second parameter to CreateProcess is the command line to trun that process which is in this case "C:\IISDebugLabs\www\App\badCgi.exe"
Let try to get the whole HTTP request that w3svc received.
For this, let dump the EXEC_DESCRIPTOR class.
0:010> dt EXEC_DESCRIPTOR 00e9eb2c
+0x000 _pstrURL : 0x00e9e370
+0x004 _pstrPhysicalPath : 0x00e9e5e0
+0x008 _pstrUnmappedPhysicalPath : 0x00e9e610
+0x00c _pstrGatewayImage : 0x00e9eac0
+0x010 _pstrPathInfo : 0x00e9e400
+0x014 _pstrURLParams : 0x00e9e3d0
+0x018 _pdwScriptMapFlags : 0x00e9eabc -> 0
+0x01c _pGatewayType : 0x00e9e060 300002 ( GATEWAY_CGI )
+0x020 _pRequest : 0x00e9e008
+0x024 _hChildEvent : (null)
+0x028 _fMustWaitForChildEvent : 0 +0x02c _pParentWamRequest : (null)
+0x030 _dwExecFlags : 0
+0x034 _pMetaData : 0x00e9ecd8
+0x038 _pPathInfoMetaData : (null)
+0x03c _pPathInfoURIBlob : (null)
+0x040 _pAppPathURIBlob : (null)
Now let dump the HTTP_REQUEST class.
0:010> dt HTTP_REQUEST 0x00e9e008
+0x000 __VFN_table : 0x65f08680
+0x008 _cFilesSent : 0
+0x00c _cFilesReceived : 0
......
......
+0x9a8 _bufClientRequest : BUFFER
......
......
+0xbac _pFileNameLock : (null)
+0xbb0 _fPossibleDefaultExecute : 0y0
=65f465b0 HTTP_REQUEST::sm_GetInfoFuncs : [26] 0x65f26a87 w3svc!HTTP_REQUEST::GetInfoA+0
+0xbb4 _strDLCString : STR
The "_bufClientRequest " points to the buffer where the HTTP request is stored.
0:010> da poi(0x00e9e008 + 0x9a8 )
006ae398 "GET /IISDebugLabs/App/badCgi.exe"
006ae3b8 "?s=av1_v1 HTTP/1.1..Accept: */*."
006ae3d8 ".Accept-Language: en-us..Accept-"
006ae3f8 "Encoding: gzip, deflate..User-Ag"
006ae418 "ent: Mozilla/4.0 (compatible; MS"
006ae438 "IE 6.0; Windows NT 5.0)..Host: l"
006ae458 "ocalhost..Connection: Keep-Alive"
006ae478 "...."
5) In the windbg command line, type in "g" and hit "Enter"
Notice a second windbg window is displayed, this is the one attached to the CGI executable "badCgi.exe".
6) In the Windbg instance attached to "badCgi.exe" type in "q" and hit "Enter".
7) Finaly, back to the main WinDbg window, press "Ctrl-Break" to break in to the debugger, type in "q" and hit "Enter"