IIS app pool crashing, C0000005 access violation errors
Joseph Lightgov
same server, same app pool. The past 4 weeks thousands of these
errors 'C0000005' are occurring. I know from IIS debug diag tool that
'C0000005' is an access violation error. Below is the top line from
my debug diag report.
In w3wp__PID__6656__Date__01_08_2011__Time_01_42_46AM__281__First
Chance Access Violation.dmp the assembly instruction at asp!
CActiveScriptEngine::GetApplication+27 in \\?\C:\Windows
\System32\inetsrv\asp.dll from Microsoft Corporation has caused an
access violation exception (0xC0000005) when trying to read from
memory location 0x00000000 on thread 29
Thread 29 - System ID 6736
Entry point 0x00000000
Create time 1/8/2011 12:46:26 AM
Time spent in user mode 0 Days 00:00:00.140
Time spent in kernel mode 0 Days 00:00:00.078
Function Source
asp!CActiveScriptEngine::GetApplication+27
vbscript!COleScript::GetDebugApplicationCoreNoRef+2b
vbscript!COleScript::FDebuggerEnabled+30
vbscript!COleScript::SetScriptSite+cd
asp!CActiveScriptEngine::Init+125
asp!CScriptManager::GetEngine+252
asp!AllocAndLoadEngines+28f
asp!ExecuteGlobal+17a
asp!Execute+b5
asp!CHitObj::ViperAsyncCallback+3fc
asp!CViperAsyncRequest::OnCall+6a
comsvcs!CSTAActivityWork::STAActivityWorkHelper+32
ole32!EnterForCallback+f4
ole32!SwitchForCallback+1a8
ole32!PerformCallback+a3
ole32!CObjectContext::InternalContextCallback+15b
ole32!CObjectContext::DoCallback+1c
comsvcs!CSTAActivityWork::DoWork+12f
comsvcs!CSTAThread::DoWork+18
comsvcs!CSTAThread::ProcessQueueWork+37
comsvcs!CSTAThread::WorkerLoop+135
msvcrt!_endthreadex+44
msvcrt!_endthreadex+ce
kernel32!BaseThreadInitThunk+e
ntdll!__RtlUserThreadStart+70
ntdll!_RtlUserThreadStart+1b
BELOW is the faulting module.
ASP report
Executing ASP requests 0 Request(s)
ASP templates cached 0 Template(s)
ASP template cache size 0.00 Bytes
Loaded ASP applications 1 Application(s)
ASP.DLL Version 7.5.7600.16620
ASP application report
ASP application metabase key
Physical Path
Virtual Root
Session Count 0 Session(s)
Request Count 0 Request(s)
Session Timeout 0 minutes(s)
Path to Global.asa
Server side script debugging enabled False
Client side script debugging enabled False
Out of process COM servers allowed False
Session state turned on False
Write buffering turned on False
Application restart enabled False
Parent paths enabled False
ASP Script error messages will be sent to browser False
ASP!CACTIVESCRIPTENGINE::GETAPPLICATION+27In
w3wp__PID__6656__Date__01_08_2011__Time_01_42_46AM__281__First Chance
Access Violation.dmp the assembly instruction at asp!
CActiveScriptEngine::GetApplication+27 in \\?\C:\Windows
\System32\inetsrv\asp.dll from Microsoft Corporation has caused an
access violation exception (0xC0000005) when trying to read from
memory location 0x00000000 on thread 29
recent events: server was being brute forced by hackers all of Dec
and probably earlier, they weren't able to gain access but did get a
virus on and blasted out spam. insatlled AVG and about the 17 or 22
latest patches. after that the app pool started crashing and the
server has crashed a couple times since then. I am in no mans land as
I am a developer and not a sys admin but I have to assume many roles.
So I'm reaching out for help.
Sometimes I will see hundreds of these 'C0000005' scriptengine errors
in the event log in a matter of seconds and other times just a few
times an hour. I googled this CACTIVESCRIPTENGINE::GETAPPLICATION and
GOT NOTHING. Now if you google it you'll get my other 2 posts, 1 on
iis.net forum and stackoverflow WHICH IS NOT FUNNY. I'm in a bad way
and need help, this is crashing my company's production server.
Thank you for your help
Brian Cryer
news:cb9653c3-4b9b-42a4...@m7g2000vbn.googlegroups.com...
> IIS 7.5, server 2008 r2, classic asp and asp .net 2.0, 3.5 website
> same server, same app pool. The past 4 weeks thousands of these
> errors 'C0000005' are occurring. I know from IIS debug diag tool that
> 'C0000005' is an access violation error. Below is the top line from
> my debug diag report.
In my experience (which doesn't cover my server being hacked - having read
the rest of your post), if the application pool is crashing its because of a
crash in an application its running. So, can you identify which
application/web-site was running at the time? I'm not sure whether your
"Function Source" indicates the call stack for the current application or
for IIS. In any event, if the application is written in house then I'd get
your developers involved, if its external then I'd seek help from the
vendor.
<snip>
> recent events: server was being brute forced by hackers all of Dec
> and probably earlier, they weren't able to gain access but did get a
> virus on and blasted out spam. insatlled AVG and about the 17 or 22
> latest patches. after that the app pool started crashing and the
> server has crashed a couple times since then. I am in no mans land as
> I am a developer and not a sys admin but I have to assume many roles.
> So I'm reaching out for help.
Have you let the server reboot after installing the patches? Its unlikley to
be it, but I have twice had problems on one of my test servers after
applying an update which updated part of the .net framework which then went
away after a reboot.
If your server did have a virus on it then you can never be completely
certain that you've got rid of everything. I know the answer is that its
probably not an option, but could you move your application to a different
server and see if that clears the problem? (and if it does then get your
server reformatted and reloaded?)
> Sometimes I will see hundreds of these 'C0000005' scriptengine errors
> in the event log in a matter of seconds and other times just a few
> times an hour. I googled this CACTIVESCRIPTENGINE::GETAPPLICATION and
> GOT NOTHING. Now if you google it you'll get my other 2 posts, 1 on
> iis.net forum and stackoverflow WHICH IS NOT FUNNY. I'm in a bad way
> and need help, this is crashing my company's production server.
Ugh. I do sympathise with your plight.
--
Brian Cryer
http://www.cryer.co.uk/brian
JosephM79
> "Joseph Lightgov" <lightgovreconci...@gmail.com> wrote in message
I just had to restart the webserver again about 45 minutes ago. I
attached Debug Diag to the worker process id's listed in IIS Manager
for the app pool that is crashing. It looks like several of them
crashed last night. The debug diag reports are the same as the ones I
caught already. I don't understand that faulting module and how I'm
supposed to find out what caused the crash. I didn't attach the whole
debug diag report, just the top line, and the faulting module. I
could post the whole report if you would like. It just makes a really
long post. I added 2 website directories to AVG file server 2011 but
from error log research I know this problem has been happening all
year on the test server, same setup as the production server so its
been an ongoing problem that the previous developers did not fix and
I'm stuck with.
debug diag log.
[1/13/2011 8:00:44 PM] First chance exception - 0xc0000005 caused by
thread with system id 6336
[1/13/2011 8:00:48 PM] Created dump file C:\Program Files
(x86)\DebugDiag\Logs\Crash rule for process id
5824\w3wp__PID__5824__Date__01_13_2011__Time_08_00_44PM__52__First
Chance Access Violation.dmp
[1/13/2011 8:00:48 PM] Action limit of 1 reached for Exception
0xC0000005.
[1/13/2011 8:00:48 PM] First chance exception - 0xc0000005 caused by
thread with system id 6336
[1/13/2011 8:00:48 PM] First chance exception - 0xc0000005 caused by
thread with system id 6336
[1/13/2011 8:00:48 PM] First chance exception - 0xc0000005 caused by
thread with system id 6336
[1/13/2011 8:00:48 PM] First chance exception - 0xc0000005 caused by
thread with system id 6336
[1/13/2011 8:00:48 PM] First chance exception - 0xc0000005 caused by
thread with system id 6336
[1/13/2011 8:00:48 PM] First chance exception - 0xc0000005 caused by
thread with system id 6336
[1/13/2011 8:00:48 PM] Thread exited. Exiting thread system id - 5684.
Exit code - 0x00000000
[1/13/2011 8:00:49 PM] Thread exited. Exiting thread system id - 4720.
Exit code - 0x00000000
[1/13/2011 8:00:49 PM] Thread exited. Exiting thread system id - 5328.
Exit code - 0x00000000
[1/13/2011 8:00:49 PM] Thread exited. Exiting thread system id - 6492.
Exit code - 0x00000000
[1/13/2011 8:00:49 PM] Thread exited. Exiting thread system id - 8908.
Exit code - 0x00000000
[1/13/2011 8:00:49 PM] Thread exited. Exiting thread system id - 8964.
Exit code - 0x00000000
[1/13/2011 8:00:49 PM] Thread exited. Exiting thread system id - 5516.
Exit code - 0x00000000
[1/13/2011 8:00:49 PM] Thread exited. Exiting thread system id - 5872.
Exit code - 0x00000000
[1/13/2011 8:00:49 PM] Thread exited. Exiting thread system id - 8712.
Exit code - 0x00000000
[1/13/2011 8:00:49 PM] Thread exited. Exiting thread system id - 4136.
Exit code - 0x00000000
[1/13/2011 8:00:49 PM] Thread exited. Exiting thread system id - 6436.
Exit code - 0x00000000
[1/13/2011 8:00:49 PM] Thread exited. Exiting thread system id - 5012.
Exit code - 0x00000000
[1/13/2011 8:00:49 PM] Thread exited. Exiting thread system id - 7428.
Exit code - 0x00000000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\cachuri.dll
unloaded from 0x6df50000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\cachfile.dll
unloaded from 0x6dbb0000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\cachtokn.dll
unloaded from 0x6db10000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\cachhttp.dll
unloaded from 0x6db00000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\compstat.dll
unloaded from 0x6dae0000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\defdoc.dll unloaded
from 0x6da70000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\dirlist.dll
unloaded from 0x6da60000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\protsup.dll
unloaded from 0x6da50000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\static.dll unloaded
from 0x6da40000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\authanon.dll
unloaded from 0x6da30000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\authbas.dll
unloaded from 0x6da20000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\modrqflt.dll
unloaded from 0x6da10000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\custerr.dll
unloaded from 0x6da00000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\loghttp.dll
unloaded from 0x6d9f0000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\iisreqs.dll
unloaded from 0x6d9e0000
[1/13/2011 8:00:50 PM] C:\Windows\system32\WSOCK32.dll unloaded from
0x6d9d0000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\isapi.dll unloaded
from 0x6d8a0000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\filter.dll unloaded
from 0x6d870000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\validcfg.dll
unloaded from 0x6d9c0000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\redirect.dll
unloaded from 0x6d860000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\cgi.dll unloaded
from 0x6d850000
[1/13/2011 8:00:50 PM] C:\Windows\System32\inetsrv\iisfcgi.dll
unloaded from 0x6d830000
[1/13/2011 8:00:50 PM] C:\Windows\system32\inetsrv\rewrite.dll
unloaded from 0x6d7e0000
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 8684.
Exit code - 0x00000000
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 6364.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 8872.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 8240.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 5812.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 8148.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 2576.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 3208.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 8608.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 8756.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 5388.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 5168.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 3320.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 4304.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 316.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 4912.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 7528.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 3912.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 6336.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 7328.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Thread exited. Exiting thread system id - 8744.
Exit code - 0xfffffffe
[1/13/2011 8:00:50 PM] Process exited. Exit code - 0xfffffffe
Thread 19 - System ID 6336
Entry point 0x00000000
Create time 1/13/2011 5:35:50 PM
Time spent in user mode 0 Days 00:00:03.437
Time spent in kernel mode 0 Days 00:00:01.187
Brian Cryer
news:a8b64db6-89d4-4e27...@j1g2000vbl.googlegroups.com...
Is there anything in the Windows event log? That might give you a clue -
although it might just say the same thing in a different way.
Do you have just one application in this application pool? (If not then I
suggest creating separate application pools for each application, as that
might help with tracking the problem down.)
The application/web-site which is crashing - assuming you know which
application is crashing - is it a third party application or one developed
in-house?
JosephM79
> "JosephM79" <lightgovreconci...@gmail.com> wrote in message
>
> - Show quoted text -
Since 1/16/2011 12:16:05 pm there have been no ScriptEngine errors and
I have no idea why. I added several directories to AVG Resident Shield
Exceptions over the course of several days even while the ScriptEngine
errors were continuing so I'm not sure if that helped. I also created
a new sql server login for the identity account in the app pool in
question, NETWORK SERVICE.
Also, the 16th was the last time the server crashed till now. I also
changed the session max count from 4 billion something way down to
1,000, disabled client-side debugging and server side debugging, I was
also getting those template persistent cache errors in the event log:
directory specified for caching compressed content C:\Windows\IIS
Temporary Compressed Files\"whichever directory" is invalid. Static
compression is being disabled.
Template Persistent Cache initialization failed for Application Pool
'DefaultAppPool' because of the following error: Could not create a
Disk Cache Sub-directory for the Application Pool.
I have since fixed. After hours of googleing/research I added NETWORK
SERVICE and IIS_IUSRS to the directories that it needed to write to,
created C:\Windows\IIS Temporary Compressed Files, and C:\Windows
\System32\inetsrv\ASP Compiled Templates - and after that it stopped
those errors in the event log. Unsure if related or not to the access
violation errors.
I can't remember if I made any other changes on the server or not. Its
been a crazy 2 weeks of research n work every day to figure out this
problem. I sure hope it don't come back again.