Exchange & IIS Services crash with 7031/7034 Errors
arrowtech....@gmail.com
I have a couple of servers that have started to display some odd
service crashes.
One server is Windows 2003 Small Business Server, the other is Windows
2003 Server with Exchange 2003 Standard.
Both servers are fully up to date with Microsoft patches as of last
"Patch Tuesday" and I have checked them both today to ensure there is
nothing new for them.
Both servers are running Symantec AntiVirus Corporate Edition 10 and
Symantec Mail Security for Exchange with the Premium AntiSpam plugin.
The symptoms on both servers are similar, but not exactly the same, I
am going to concentrate on the non-SBS server in this post to make it
easier for me to describe.
The initial symptom that led to investigation was the website/webmail
being down and mail sitting in the Outbox in Outlook.
Investigation of the System event log shows the following four errors:
---------------------------
Source: Service Control Manager
Type: Error
Event ID: 7031
Time: 11:51:01 PM
The IIS Admin Service service terminated unexpectedly. It has done this
58 time(s). The following corrective action will be taken in 1
milliseconds: Run the configured recovery program.
---------------------------
Source: Service Control Manager
Type: Error
Event ID: 7034
Time: 11:51:01 PM
The Microsoft Exchange Routing Engine service terminated unexpectedly.
It has done this 57 time(s).
---------------------------
Source: Service Control Manager
Type: Error
Event ID: 7034
Time: 11:51:01 PM
The Simple Mail Transfer Protocol (SMTP) service terminated
unexpectedly. It has done this 58 time(s).
---------------------------
I also sometimes, but not always, get these errors:
---------------------------
Source: W3SVC
Type: Error
Event ID: 1011
Time:
A process serving application pool 'ExchangeApplicationPool' suffered a
fatal communication error with the World Wide Web Publishing Service.
The process id was '7812'. The data field contains the error number.
---------------------------
Source: W3SVC
Type: Error
Event ID: 1013
Time:
A process serving application pool 'ExchangeApplicationPool' exceeded
time limits during shut down. The process id was '5120'.
---------------------------
Source: W3SVC
Type: Error
Event ID: 1010
Time:
Inetinfo terminated unexpectedly and the system was not configured to
restart IIS Admin. The World Wide Web Publishing Service has shut down.
---------------------------
It seems a lot like Microsoft KB Article 827214 however in that article
the link sends me here and then onto here where it links me to the
product update
The update was published in 2004 and it appears that it would be
contained in one or another service pack, so it doesn't seem like this
is the fix for me.
There are a few other people turning up here and there on the internet,
but not great reams yet.
I have a gut feeling that it is a new vulnerability, but it could just
as easily be something Symantec screwed up in an update, or just some
bug.
I have turned on logging for incoming SMTP connections and activated
the SMTP logging on both servers, and am now sitting with both servers
up on my screen in the middle of the night trying to find more
information.
I will keep it coming as I have it.
I am also going to cross post this to Experts Exchange and the
Petri.co.il - hopefully if it is a serious problem that will help
people find answers.
arrowtech....@gmail.com
arrowtech....@gmail.com wrote:
<SNIP>
Duh - the links don't work when you cut and paste to usenet......
Here are the links from above:
http://tinyurl.com/y96na7
http://tinyurl.com/y4zh6a
http://tinyurl.com/yytwz7
http://tinyurl.com/vs7k2
http://tinyurl.com/wh3pb
Sorry about that, the brain is getting fuzzy at this hour.
C
arrowtech....@gmail.com
arrowtech....@gmail.com wrote:
<SNIP>
Further Details
On my server that is SBS, I just got a crash at 00:36:28
The firewall and 2K3 server have their clocks synchronised, so it is
easier to pull together the logs on that one (I am having a separate
issue getting access to my firewall to adjust the time for a 3 minute
drift and daylight savings on the other site)
Here is how it went down:
Event log - 7031(IIS)/7034(NNTP)/7034(MS Exchange Routing
Engine)/7034(SMTP) all fall over at 00:36:28
Firewall - SMTP incoming traffic:
00:36:23 FROM 200.119.210.170 (Unresolvable)
00:36:24 FROM 160.83.65.200 (Unresolvable)
00:36:29 FROM 203.94.218.69 (dialup-mum-203.94.218.69.mtnl.net.in)
00:36:31 FROM 203.94.218.69 (dialup-mum-203.94.218.69.mtnl.net.in)
00:36:37 FROM 203.94.218.69 (dialup-mum-203.94.218.69.mtnl.net.in)
00:36:38 FROM 203.94.218.69 (dialup-mum-203.94.218.69.mtnl.net.in)
00:37:58 FROM 64.237.216.98 (adsl-64-237-216-98.prtc.net)
SMTP Logs (we are +11hr on GMT here)
2006-12-04 13:36:01 129.41.76.38 mail2038.rm02.net SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 BDAT -
+<26609938.116523933...@mx01.atlp2> 250 0 129 86408
20750 SMTP - - - -
2006-12-04 13:36:01 129.41.76.38 mail2038.rm02.net SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 QUIT - mail2038.rm02.net 240 23734 69 4
0 SMTP - - - -
2006-12-04 13:36:05 61.129.51.171 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - -
220+ESMTP+on+WinWebMail+[3.7.3.1]+ready.++http://www.winwebmail.com 0 0
67 0 188 SMTP - - - -
2006-12-04 13:36:05 61.129.51.171 OutboundConnectionCommand SMTPSVC1
SERVERNAMECHANGED - 25 EHLO - domainchanged.com.au 0 0 4 0 203 SMTP - -
- -
2006-12-04 13:36:05 61.129.51.171 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - - 250-SIZE 0 0 8 0 391 SMTP - - - -
2006-12-04 13:36:05 61.129.51.171 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - - 250+AUTH+LOGIN 0 0 14 0 703 SMTP - - - -
2006-12-04 13:36:05 61.129.51.171 OutboundConnectionCommand SMTPSVC1
SERVERNAMECHANGED - 25 MAIL - FROM:<> 0 0 4 0 703 SMTP - - - -
2006-12-04 13:36:05 61.129.51.171 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - - 250+OK 0 0 6 0 906 SMTP - - - -
2006-12-04 13:36:05 61.129.51.171 OutboundConnectionCommand SMTPSVC1
SERVERNAMECHANGED - 25 RCPT - TO:<x...@xxx.xxx> 0 0 4 0 906 SMTP - - - -
2006-12-04 13:36:05 61.129.51.171 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - - 250+OK,+recipient+accepted 0 0 26 0 1094
SMTP - - - -
2006-12-04 13:36:05 61.129.51.171 OutboundConnectionCommand SMTPSVC1
SERVERNAMECHANGED - 25 DATA - - 0 0 4 0 1094 SMTP - - - -
2006-12-04 13:36:06 61.129.51.171 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - -
354+Send+checkpointed+message,+ending+in+CRLF.CRLF 0 0 50 0 1297 SMTP -
- - -
2006-12-04 13:36:26 200.119.210.170 plwrag SMTPSVC1 SERVERNAMECHANGED
192.168.X.X 0 HELO - +plwrag 250 0 50 11 0 SMTP - - - -
2006-12-04 13:36:26 160.83.65.200 imr8.us.db.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 EHLO - +imr8.us.db.com 250 0 313 19 0
SMTP - - - -
2006-12-04 13:36:26 160.83.65.200 imr8.us.db.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 MAIL - +From:<xx...@xxxx.xxx> 250 0 45
52 0 SMTP - - - -
2006-12-04 13:36:26 160.83.65.200 imr8.us.db.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 RCPT - +To:<xx...@xxxxx.xxx> 250 0 39
36 0 SMTP - - - -
2006-12-04 13:36:27 200.119.210.170 plwrag SMTPSVC1 SERVERNAMECHANGED
192.168.X.X 0 MAIL - +FROM:+<x...@xxx.xxx> 250 0 43 31 0 SMTP - - - -
#Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Date: 2006-12-04 13:36:34
It feels to me like it is the 160.83.65.200 that is the source of the
problem, the other crashes have very similar charateristics - an
EHLO/MAIL/RCPT/Crash as if it starts to send the DATA and that is where
everything goes wrong.
Stay tuned for further details at 11.00 :-)
arrowtech....@gmail.com
arrowtech....@gmail.com wrote:
> arrowtech....@gmail.com wrote:
>
> <SNIP>
The plot thickens
An hour on, and who do we see in the log again?
Services died at 01:39:35
2006-12-04 14:39:16 142.245.29.136 vmx1.rbc.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 EHLO - +vmx1.rbc.com 250 0 314 17 0
SMTP - - - -
2006-12-04 14:39:16 142.245.29.136 vmx1.rbc.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 MAIL - +FROM:<XX...@XXXX.XXX> 250 0 45
42 0 SMTP - - - -
2006-12-04 14:39:16 142.245.29.136 vmx1.rbc.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 RCPT - +TO:<XX...@XXXX.XXX> 250 0 36 33
0 SMTP - - - -
2006-12-04 14:39:24 142.245.29.136 vmx1.rbc.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 DATA -
+<20061204135846673.z...@SYDNT105.pine.fg.rbc.com> 250
0 149 1299 7296 SMTP - - - -
2006-12-04 14:39:24 142.245.29.136 vmx1.rbc.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 MAIL - +FROM:<XX...@XXXX.XXX> 250 0 45
42 0 SMTP - - - -
2006-12-04 14:39:24 142.245.29.136 vmx1.rbc.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 RCPT - +TO:<XX...@XXXX.XXX> 250 0 39 36
0 SMTP - - - -
2006-12-04 14:39:25 142.245.29.136 vmx1.rbc.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 DATA -
+<20061204135846923.Z...@SYDNT105.pine.fg.rbc.com> 250
0 149 1305 421 SMTP - - - -
2006-12-04 14:39:29 142.245.33.100 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - - 220+mx1.istrbc.com+ESMTP 0 0 24 0 4094 SMTP
- - - -
2006-12-04 14:39:29 142.245.33.100 OutboundConnectionCommand SMTPSVC1
SERVERNAMECHANGED - 25 EHLO - domainchanged.com.au 0 0 4 0 4094 SMTP -
- - -
2006-12-04 14:39:29 142.245.33.100 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - - 250-mx1.istrbc.com 0 0 18 0 4422 SMTP - - -
-
2006-12-04 14:39:29 142.245.33.100 OutboundConnectionCommand SMTPSVC1
SERVERNAMECHANGED - 25 MAIL - FROM:<>+SIZE=2812 0 0 4 0 4422 SMTP - - -
-
2006-12-04 14:39:29 142.245.33.100 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - - 250+sender+<>+ok 0 0 16 0 4953 SMTP - - - -
2006-12-04 14:39:29 142.245.33.100 OutboundConnectionCommand SMTPSVC1
SERVERNAMECHANGED - 25 RCPT - TO:<XX...@XXXX.XXX> 0 0 4 0 4953 SMTP - -
- -
2006-12-04 14:39:30 142.245.33.100 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - - 250+recipient+<XX...@XXXX.XXX>+ok 0 0 39 0
5375 SMTP - - - -
2006-12-04 14:39:30 142.245.33.100 OutboundConnectionCommand SMTPSVC1
SERVERNAMECHANGED - 25 DATA - - 0 0 4 0 5375 SMTP - - - -
2006-12-04 14:39:30 142.245.33.100 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - - 354+go+ahead 0 0 12 0 6078 SMTP - - - -
2006-12-04 14:39:30 142.245.29.136 vmx1.rbc.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 QUIT - vmx1.rbc.com 240 15640 69 4 0
SMTP - - - -
2006-12-04 14:39:31 142.245.33.100 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - - 250+ok:++Message+215759852+accepted 0 0 35 0
6547 SMTP - - - -
2006-12-04 14:39:31 142.245.33.100 OutboundConnectionCommand SMTPSVC1
SERVERNAMECHANGED - 25 QUIT - - 0 0 4 0 6547 SMTP - - - -
2006-12-04 14:39:31 142.245.33.100 OutboundConnectionResponse SMTPSVC1
SERVERNAMECHANGED - 25 - - 221+mx1.istrbc.com 0 0 18 0 6875 SMTP - - -
-
2006-12-04 14:39:32 160.83.65.200 imr8.us.db.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 EHLO - +imr8.us.db.com 250 0 313 19 0
SMTP - - - -
2006-12-04 14:39:32 160.83.65.200 imr8.us.db.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 MAIL - +From:<XX...@XXXX.XXX> 250 0 45
52 0 SMTP - - - -
2006-12-04 14:39:32 160.83.65.200 imr8.us.db.com SMTPSVC1
SERVERNAMECHANGED 192.168.X.X 0 RCPT - +To:<XX...@XXXX.XXX> 250 0 39 36
0 SMTP - - - -
#Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Date: 2006-12-04 14:39:40
arrowtech....@gmail.com
arrowtech....@gmail.com wrote:
> > <SNIP>
>
And thickens further.
The server delivering the message that appears to be causing the above
machine to crash is imr8.us.db.com.
I have just discovered from the logs that the server causing the other
machine to crash is loninmrp6.uk.db.com.
I won't jump to any conclusions yet - could be spoofed IPs, or any of a
million other problems.
arrowtech....@gmail.com
arrowtech....@gmail.com wrote:
> arrowtech....@gmail.com wrote:
>
> > > <SNIP>
> >
Well guys, I turned off AV on one of the servers and waited the hour
for the message to retry, here is the email that made it so angry
--------------------------
Return Receipt
Your document:
FW: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
was received by:
at:
01/12/2006 15:37:23
--------------------------
The read receipt was generated by a notes user at DB in response to a
legitimate email. Who knows why that made Symantec angry, but I will be
calling them tomorrow to ask - it has made me a little angry too.
Oh well, off to bed now, 4 hours of solid rest till I am back at it.
A
Kirill Palagin
><snip >
> The read receipt was generated by a notes user at DB in response to a
> legitimate email. Who knows why that made Symantec angry, but I will be
> calling them tomorrow to ask - it has made me a little angry too.
>
> Oh well, off to bed now, 4 hours of solid rest till I am back at it.
>
> A
>
Thank you very much for sharing!!
Please share the outcome of the discussion with Symantec.
--
If my message is helpful, please help me by registering at
http://www.openoffice.org/servlets/Join and voting for the following issues:
http://www.openoffice.org/issues/show_bug.cgi?id=70753
http://www.openoffice.org/issues/show_bug.cgi?id=15220
http://www.openoffice.org/issues/show_bug.cgi?id=10931
http://www.openoffice.org/issues/show_bug.cgi?id=35579
http://www.openoffice.org/issues/show_bug.cgi?id=32785
http://www.openoffice.org/issues/show_bug.cgi?id=29807
http://www.openoffice.org/issues/show_bug.cgi?id=67838
http://www.openoffice.org/issues/show_bug.cgi?id=39527
http://www.openoffice.org/issues/show_bug.cgi?id=64785
Thank you very much!
Chris
bad ruleset included with a recent update. And surprising me was the fact
that they actually admitted this and told me how to work around it. They are
working on a new rule set that will be posted as an update hopefully today or
tomorrow, but in the meantime here is the workaround from the email they sent
me. Be sure to make a backup of the .xml file before proceeding. Apparently
there is no further action required - the update will fix the ruleset in the
.xml automatically:
Greetings,Chris
To modify bmiconfig.xml to work around the issue:
Open the services menu by going to Start -> Run and typing services.msc
Stop the Symantec Mail security for Microsoft exchange service, and the
Symantec Mail security spam statistics service, if they are started
Open <system drive>:\Program
iles\Symantec\SMSMSE\5.0\Server\SpamPrevention\bmiconfig.xml in a text editor
such as notepad
Go to the File menu, choose save as, and save the file as bmiconfig.old
Delete the following strings:
<ruleType>header_regex</ruleType>
<ruleType>body_regex</ruleType>
<ruleType>lang_header_regex</ruleType>
<ruleType>lang_body_regex</ruleType>
<ruleType>bodysig</ruleType>
· Once those entries are deleted, go to the File menu, and choose
save as, save the file as bmiconfig.xml
· Restart the Symantec mail security for Microsoft exchange service;
it is not necessary to restart the Spam statistics service.
Sincerely,Rudy
--------------------------------------------------------------------------------
Symantec Technical Support
I don't usually cross-post, but I'm going to throw this out on the few other
threads for this topic, so don't kill me. Good luck!
-Chris
arrowtech....@gmail.com
Thanks so much for posting this to all of the threads - it fixed our
problem, we could have been onto Symantec for a while getting this
escalated to get it sorted out.
Clayton
Chris wrote:
> Ok, guys. Just got off the phone with Symantec. As I suspected, there was a
> bad ruleset included with a recent update. And surprising me was the fact
> that they actually admitted this and told me how to work around it. They are
> working on a new rule set that will be posted as an update hopefully today or
> tomorrow, but in the meantime here is the workaround from the email they sent
> me. Be sure to make a backup of the .xml file before proceeding. Apparently
> there is no further action required - the update will fix the ruleset in the
> .xml automatically:
<SNIP>
Who knows?
the changes in the XML file and all seems well.
Thinking enough time had passed since this was reported to them, I did an
update. No such luck. Back to to bombing out. I edited the XML file again
and turned off the updates in SMS.
I'll watch this thread to see if anything new comes up, but it's not fixed
from the Symantec end yet!
arrowtech....@gmail.com
spam that is being allowed through as well - I assume not as many
fields are being scanned.
I would be interested if someone fresh rang Symantec with this problem
now whether the helpdesk would be onto it immediately or not.
With this on top of thel licensing disaster they are in the middle of
it,
DavidH
the hopes they get a fix soon:
since you can't see the colors: timestamps without anything after are when
the services crashed, I'm not sure how safe it is to just take out that one
line in the bmiconfig.xml file -(i'm going to cross post to the other
posting on the subject)
Red = time services failed and restarted
Blue = time xml file modified
09:52 (am today)
10:22
11:03
11:05
11:08
11:08
11:09
11:09
11:12
11:29
12:19
12:20
12:21
12:21
12:22
12:23
12:50 - removed 5 lines from xml file and restarted symantec mail security
service
01:46 - added "<ruleType>header_regex</ruleType>" line back to xml file and
restarted symantec mail security service
02:21 - added "<ruleType>body_regex</ruleType>" line back to xml file and
restarted symantec mail security service
02:49
03:00 - removed "<ruleType>body_regex</ruleType>" and added
"<ruleType>lang_header_regex</ruleType>" and restarted symantec mail
security service
03:37 - added "<ruleType>lang_body_regex</ruleType>" line back to xml file
and restarted symantec mail security service
04:08 - added "<ruleType>bodysig</ruleType>" line back to xml file and
restarted symantec mail security service
04:38 - no services stopping
Who knows?
not hearing an immediate response. I do hear then mail sits in their
outbox!
I don't have a support contract, or I would give them a call.....
Who knows?
<ruleType>body_regex</ruleType>
is the culprit near as you can tell?
arrowtech....@gmail.com
I resell Symantec software, and as far as I understand it you can't
have a subscription to SMSE or Premium AntiSpam without getting the
support with it as well?
I have the problem on our own server, as well as 3 clients, and we are
definitely getting more Spam through.
Clayton
Who knows?
remember SAV from the 7.0 days and you needed a support contract to go with
the subscription.
We resell Symantec as well and have at least 45-50 clients with SAV/SMS and
only a handful with PAS. Only have one with the problem so far knock on
wood!
DavidH
services haven't stopped- but I have no idea if any of these will add any
more spam protection to the server without that line - but with that info
hopefully symantec will issue the fix.
"Who knows?" <qub...@yahoo.com> wrote in message
news:Xns9891A7FCA989...@216.196.97.131...
catsarecool
they have a "bug" in their antispam definitions that's causing it. I
believe they finally fixed it. My servers liveupdated around 8:00
tonight, and, knock on wood, they've been up and clean for over 3 hours
now. The services used to crash about every 15 minutes or so. God I
hope Symantec doesn't do that to us again.
Kirill Palagin
> Same thing has happened to me. It's the Symantec software. Evidently
> they have a "bug" in their antispam definitions that's causing it. I
> believe they finally fixed it. My servers liveupdated around 8:00
> tonight, and, knock on wood, they've been up and clean for over 3 hours
> now. The services used to crash about every 15 minutes or so. God I
> hope Symantec doesn't do that to us again.
People at Symantec are busy fighting MS on legal front, quality takes
back seat.
jas...@mcclellans.ca
I appear to be having this exact issue still now, on Dec. 13. It
started mysteriously on the morning of Dec. 5, with the services you
describe crashing randomly. I've been working on it on and off ever
since trying everything I can think of, short of reinstalling the
server, to get it to stop.. to no avail. When I found this thread it
was like.. oh.. my.. god..
however making this change results in SMSME service not starting :( or
rather, starting then stopping)
I can't believe they haven't fixed this yet! And I just bought 65 seats
of the premium add-on.. plus renewed maintenance. :(
anyone getting any love with this problem?
dpovi...@gmail.com
it wasn't for this thread, we'd still be taking shots in the dark. Not
sure why Symantec could not have informed us of this in some way...
Anyway, thanks for this.
mga...@gmail.com
we have tried this solution and it is working withus, Thanks alot for
your help, appreciate
Maria
We were having the same problem and we made changes to the XML file - now
exchange 2003 is stable however we seem to be getting a lot of spam emails -
can some one advise on what is the resolution to stop spam mails also has
symantec any resolution to this problem.
Would really be grateful for a response if anyone has resolved this.
Maria