Exchange 2003 SP2 Greylisting bug - survey
Yizhar Hurwitz
As some of you already know, there is a bug with Exchange 2003 SP2 that
causes outbound mail to domains that implement greylisting to be jailed in a
black hole until the SMTP service is restarted.
A more detailed description of the problem can be found here:
http://www.microsoft.com/communities/newsgroups/list/en-us/default.aspx?mid=df7bcd16-0306-4787-a358-3888c4e1198f&dg=microsoft.public.exchange.admin
And/Or just by searching this newsgroup for the word "greylisting".
Please read the detailed story linked above before you comment to this post,
to make sure that we all know what we are talking about.
As many of us have also noticed, there is yet no official response from MS
about this bug, which is quite a shame...
This is because of the following reasons (among other):
1. At least for my case that I have openned with them, they were not able to
reproduce the problem, and also I wasn't able to reproduce it at my client
site (I guess the workarounds did the job).
2. MS PSS didn't get the full picture about the scale of this problem,
and didn't make the relations between one case to the other.
3. If the workarounds were solving the problem (as was in my case), it was
not escelated to the Exchange Development team for further investigation and
a proper fix.
So - what are the next steps, and how can we help MS fix this strange and
tricky bug?
Here are some answers:
1. If you are having (or had recently) this same problem, please contact
your local PSS , but reference my initial case which is:
EMEA case SRZ060302000872
This will help PSS relate the different cases to each other and to the same
bug.
I will quote a response I got from MS about this in reply to my own post here.
2. Participate in this survey by replying to this message, and provide as
much related details as you think.
Here are some important details:
* Do you currently have the problem, or had it in the past?
* Your Exchange version and SP (probably Exchange 2003 SP2).
* The OS version and SP (for example WIN2003 SP1).
* Was the OS upgraded from WIN2000 or is it a clean install?
* Anti virus and other related 3rd party software installed on the server.
* Have you openned a case with MS PSS?
If so, can you write the case number?
Is the case still open?
* Can you list some domain names that you had trouble sending email to?
for example:
technion.ac.il (I had problems sending to them).
* Do you have SMTP logs (if not - please enalbe them)?
Logging the SMTP Service:
http://www.msexchange.org/tutorials/Logging_the_SMTP_Service.html
* If you like, you can publish your contact information such as phone number.
* Any additional related details that you can provide, and a description of
the problem that you had/have.
Summary so far:
If you had/have this problem please do:
1. Contact MS PSS and reference
EMEA case SRZ060302000872
2. Reply to this post and provide details.
Bye,
--
Yizhar Hurwitz
http://yizhar.mvps.org
Yizhar Hurwitz
Here is a quote of what the Team Manager in PSS (second level) had asked me
to write, and I provide it here as is:
******** Begin quote **********
If you are having an issue with greylisting, please review the problem
description below. Should your problem match the description below and the
workaround works for you, Microsoft has advised me to ask you to raise a
support call with them. When you raise your support call, please ask that
they link the call to the EMEA case SRZ060302000872. This way, should a
number of customers be seeing the problem, it can be tracked within Microsoft.
SYMPTOMS
=========
When Exchange tries to send mails to certain domains that implement
‘greylisting’, the mails fail to get delivered, without any intimation to the
sender on the first attempt. Thereafter on either restarting the server or
SMTP service, the mails get delivered to the destination domains. At times,
NDRs for these delayed mails are also generated after rebooting.
PROBLEM
========
The issue is intermittent and occurs only when the destination domains have
greylisting implemented and mail is sent to the greylisted domain the very
first time.
WORKAROUND
============
1. Write a script to restart the SMTP service at least once a day.
2. Modify the registry on the sender Exchange server, to change
the Glitch Retry key.
3. Clarify that there is no 3rd party AV software in the
environment which could be causing the issue.
******** End quote **********
--
Yizhar Hurwitz
http://yizhar.mvps.org
Rappaport@discussions.microsoft.com Dominik Rappaport
I’m Dominik from Vienna, Austria and I read you postings about the Exchange
Server 2003 Greylisting bug. I experienced this bug on at least two machines
but was not able to reproduce it. Two days ago I installed a new Small
Business Server 2003 for a customer. I sent a test mail to my private e-Mail
account and discovered that the e-Mail did not arrive. My ISP uses
greylisting for spam protection.
As the bug was familiar to me I immediately made some investigations:
1. The message was sent only one time and the server responded with a 4xx
temporary error code
2. Contrary to what should happen, the message was never sent again
3. Even worse, the message did NOT appear in the Exchange Server Queue viewer!
4. I used Mfcmapi to look into the TempTables and there I found the missing
message
As the system was fresh installed without any virus scan program or third
party spam filter installed, I opened a call to Microsoft PSS. I made the
same experiences as you. The friendly woman on the other side on the phone
focused just on the configuration of my machine. She advised me to create the
GlitchRetrySeconds key in the registry as described in
http://technet.microsoft.com/en-us/library/8b43be56-48e6-400b-8014-54c95f87d1de.aspx.
After that I had to restart the SMTP service. Now everything behaved as
expected. The message (I used a different sender address for proper testing)
was correctly delivered after the intended delay.
Now I did the crosscheck and removed the GlitchRetrySeconds key and
restarted the SMTP service. Unfortunately the error did not reoccur. I never
believed that the bug was related to the GlitchyRetry interval but now I
could not reproduce the bug and had nothing to show for PSS.
I discussed this with the Microsoft support engineer and argued that the
fact that a message was in the TempTables but is not shown in the queue
viewer must be considered as a bug and if they could make further
investigations. She completely refused to do so and said that the message can
now be properly delivered and that was the goal of the support call that I
have consented.
I replied that the error could reoccur every time and that I’m very
dissatisfied that she absolutely denied the possibility of a bug. Then she
said “Please wait a minute!”. After she reported back she said there exists a
brand new hotfix KB 934709 that she could send to me. It is designed to
“check SMTP temp table regularly and try to resend corrupted messages”. For
me, that sounds like a bug fix. Even more, the hotfix replaces the aqueue.dll
file. One core DLL of the advanced queuing engine of the SMTP service. The
hotfix is so brand new that today no KB article exists but will be written in
the near future (22th of June, 2007).
I can’t say if this hotfix resolves the problem as it did not reappear.
My support call ID was EMEA SRQ070620601190.
I’m curious what the KB article related to the hotfix will say about it and
if it will mention Greylisting.
Kind regards,
Dominik
--
MCSA Messaging
Howden@discussions.microsoft.com Steve Howden
Howden@discussions.microsoft.com Steve Howden
Very old NDRs popping up after a server or SMTP svc restart.
It wasn't until I noticed a very long SMTP conversation in the SMTP svc logs
that i started to get suspicious.
Exchange was re-sending a mail on receipt of a 450 after only 1 second! In
one case it kept trying every second in the same conversation 10 times before
quitting.
I contacted the admin for the recipient mail server and he informed me of
their Greylisting policy which expects a retry after 120 secs. He also
pointed out that any mail server retrying every second for ten times might
make them think it is a DOS attack and blacklist our IP address :-(
I found the GlitchRetrySeconds registry edit and made it 120 secs. That
seemed to have fixed the problem......... until today when I discovered a
mail server with greylisting that expects a 300 sec delay for replies. I wish
they would all settle on a standard time. I now have GlitchRetry up to 300
secs and testing with various fresh emails seems to have solved it.
Smells buggy to me.
regards, Steve
jru...@hdp.com
I have been going back and forth with this as well. I finally got
this hotfix from MS. The engineer I am working with does not think
this is my issue as I am not seeing these messages in the Queue as
described in this article. We are still doing some tests, but I have
a gut feeling this will fix it.
http://support.microsoft.com/default.aspx?scid=kb;EN-US;934709
Raj
have explained. This is we can confirm 101% because I put up a test bed on a
greylist server and made exchange deliver a mail to it. After the first 4.7.1
that exchange recieves during the first attempt , Exchange does not reattempt
the delivery at all.
The Queue shows that the mail was retried Once.
CHanging the GLITCHRETYSECONDS to 120 and restarting smtp service did not
work , nor for 300 nor for 360. I did not try 60 becuase elsewhere it is
documented that the default is 60 seconds.
I wanted to confirm if the key QUEUING should be under smtpsvc or under
smtpsvc/parameters ?
Is there a link to download the hotfix ?
TIA
Raj
Rich Matheisen [MVP]
>I agree that the Exchange 2003 SP2 that we have behaves just the way you
>have explained. This is we can confirm 101% because I put up a test bed on a
>greylist server and made exchange deliver a mail to it. After the first 4.7.1
>that exchange recieves during the first attempt , Exchange does not reattempt
>the delivery at all.
>The Queue shows that the mail was retried Once.
>
>CHanging the GLITCHRETYSECONDS
Some registry names are case sensitive. Try "GlitchRetrySeconds".
>to 120 and restarting smtp service did not
>work , nor for 300 nor for 360. I did not try 60 becuase elsewhere it is
>documented that the default is 60 seconds.
>
>I wanted to confirm if the key QUEUING should be under smtpsvc
Yes.
>or under
>smtpsvc/parameters ?
No.
>Is there a link to download the hotfix ?
Call Microsoft.
--
Rich Matheisen
MCSE+I, Exchange MVP
MS Exchange FAQ at http://www.swinc.com/resource/exch_faq.htm
Don't send mail to this address mailto:h.p...@getronics.com
Or to these, either: mailto:h.p...@pinkroccade.com mailto:melvin.mcp...@getronics.com mailto:melvin.mcp...@pinkroccade.com
Yizhar Hurwitz
> I found the GlitchRetrySeconds registry edit and made it 120 secs. That
> seemed to have fixed the problem......... until today when I discovered a
> mail server with greylisting that expects a 300 sec delay for replies. I wish
> they would all settle on a standard time. I now have GlitchRetry up to 300
> secs and testing with various fresh emails seems to have
You have missed the actual problem with the bug.
As long as your server does retry the message, this is OK.
After 3 attempts using glitchretry, the message should go to the regular
queue, and the server should retry again every X minutes (defined in the
properties of the default SMTP virtual server).
So - even if the recipient server has a timeout of, let's say 10 minutes,
the message should go through eventually, after several retries.
However, when the problem happens, Exchange fails to retry even on the first
attempt, and fails to requeue the message.
Changing the glitchretry might help as some people posted, bug I guess that
the "Advanced" queue update might be the real fix (I haven't tested it, and
didn't get any details from MS about it other than what was posted here):
On a Windows Server 2003-based SMTP gateway server, some messages may remain
in the queue folder until the SMTP service is restarted:
http://support.microsoft.com/default.aspx?scid=kb;EN-US;934709
I think that using the workarounds that I described in my initial post is
also a good practice:
* Using a smart host for outbound mail traffic (defined in the SMTP
connector), if possible and applicable.
* daily restart of SMTP service (scheduled tasks).
And the workarounds/fixes described above, such as:
* Setting glitchretry to something like 60-120 seconds.
* Installing hotfix KB934709 .
--
Yizhar Hurwitz
http://yizhar.mvps.org
JGurtz
"Dominik Rappaport" wrote:
> said “Please wait a minute!”. After she reported back she said there exists a
> brand new hotfix KB 934709 that she could send to me. It is designed to
> “check SMTP temp table regularly and try to resend corrupted messages”. For
> me, that sounds like a bug fix. Even more, the hotfix replaces the aqueue.dll
> file. One core DLL of the advanced queuing engine of the SMTP service. The
> hotfix is so brand new that today no KB article exists but will be written in
> the near future (22th of June, 2007).
It is funny that the KB article only mentions Windows Server 2003 and NOT
Exchange at all, even though it seems to fix the problem for exchange also.
Does anyone know if this 4xx response to RCPT TO: is still a problem in
Exchange 2007? How about with the beta of Ex'07 SP1? What about Server 2008?
Yizhar Hurwitz
> It is funny that the KB article only mentions Windows Server 2003 and NOT
> Exchange at all, even though it seems to fix the problem for exchange also.
Yeah, if it wasn't sad, it could be funny...
> Does anyone know if this 4xx response to RCPT TO: is still a problem in
> Exchange 2007? How about with the beta of Ex'07 SP1? What about Server 2008?
Exchange 2007 has a totaly different SMTP architecture and is not affected
by this specific bug.
It is a different product which has new bugs to replace the older ones which
are no longer needed.
Jason Gurtz
>> Does anyone know if this 4xx response to RCPT TO: is still a problem in
>> Exchange 2007?
>>
> by this specific bug.
Wow, I wonder if it's actually getting to the point were it can be
directly accessible on the Internet? Do tell, does it play nice now!?
/me wonders how the MIMEfield situation is now.
Cheers,
~JasonG
--
Susan K
Do you know, or does anyone else know, if this MS hotfix patch no. 934709
has been tested on a Small Business Server 2003? We have been experiencing
the same problems, I obtained a copy of the patch, but I'm in a very small
environment and am very apprehensive to install the hotfix since I don't have
the ability to test it. I'll definitely try your other recommendations, but
any feed back about testing on SBS would be greatly appreciated.
Thanks
--
Susan
Rich Matheisen [MVP]
>Do you know, or does anyone else know, if this MS hotfix patch no. 934709
>has been tested on a Small Business Server 2003?
SBS, in this area, is no different to any other Windows/Exchange
server.
Alexander
Yizhar Hurwitz referred me to this thread since I posted a similar thing in
the technet exchange newsgroup.
A lot has been said about this problem already. I won´t repeat everything.
However, I do have to add one important difference:
This Problem definitely also appears on an Exchange 2000 (latest patchlevel)
server on a Windows 2000 (Latest patchlevel) server so it is not only an
Exchange 2003 SP2 problem. We recently had this problem on such a server at
one of our customers Exchange server. Messages as old as 1 month suddenly
were sent and NDRs were received. The boss of that company was very angry
with us, he alone was affected with 40 messages.
I do not have access to all NDRs but to a few but they are in German. (The
important things are language independant though):
--------------------------------------------------------------------------------
Ihre Nachricht hat einige oder alle Empfänger nicht erreicht.
Betreff: xxxxxxxxxxxxxxxx
Gesendet am: 11.12.2007 10:19
Folgende(r) Empfänger kann/können nicht erreicht werden:
xxx...@wwff.gv.at am 13.01.2008 15:34
Das E-Mail-System konnte diese Nachricht nicht übermitteln, ohne
einen besonderen Grund dafür anzugeben. Überprüfen Sie die Adresse, und
versuchen Sie es dann erneut. Wenn die Übermittlung nochmals fehlschlägt,
wenden Sie sich an den Systemadministrator.
<xxxxxxxx.xxxxxxx.at #4.0.0 smtp;421 temporary envelope failure
(#4.3.0)>
--------------------------------------------------------------------------------------------------
Ihre Nachricht hat einige oder alle Empfänger nicht erreicht.
Betreff: xxxxxxxxxxxxxxxxxxxxxxxxxxx
Gesendet am: 06.12.2007 10:05
Folgende(r) Empfänger kann/können nicht erreicht werden:
xxxx...@genion-clipping.si am 13.01.2008 15:33
Sie sind nicht berechtigt, Nachrichten an diesen Empfänger zu
senden. Wenden Sie sich an den Systemadministrator.
<xxxxx.xxxxxx.at #4.7.1 smtp;450 4.7.1
<xxxx...@genion-clipping.si>: Recipient address rejected: temporary blocked
due to greylisting; http://www.amis.net/greylisting/>
--------------------------------------------------------------------------------------------------------
Ihre Nachricht hat einige oder alle Empfänger nicht erreicht.
Betreff: Ihre »OBSERVER« STORNIERUNG zum Auftrag 'Media Focus' Nr.4993
Gesendet am: 20.12.2007 11:24
Folgende(r) Empfänger kann/können nicht erreicht werden:
xxxx...@focusmr.com am 13.01.2008 15:33
Diese Nachricht hat das E-Mail-System des Empfängers erreicht,
die Übermittlung der Nachricht wurde jedoch verweigert. Versuchen Sie
nochmals, diese Nachricht zu senden. Wenn die Übermittlung erneut
fehlschlägt, wenden Sie sich an den Systemadministrator.
<xxxxxx.xxxxxx.at #4.2.0 smtp;450 4.2.0 <xxx...@focusmr.com>:
Recipient address rejected: Greylisted, see
http://postgrey.schweikert.ch/help/focusmr.com.html>
----------------------------------------------------------------------------------------------
The hotfix provided by Microsoft won´t help us since it is for Exchange 2003
only. Also we have read that it won´t really fix the problem anyway. We will
implement a daily SMTP service restart.
Since the server will be replaced by an Exchange 2007 Server soon anyway we
won´t do much more or even open a PSS case with Microsoft but I am very
disappointed that an almost 2 year old known bug is not acknowledged by
Microsoft.
A guy on the Partner newsgroups even sent me on a detour exploring temp
tables and suggested deleting them but this is no real solution because it
will kill the messages inside and also this cannot be scripted because there
is no way to find out if Temptables are corrupted or not.
He also implied that Third party software is know to cause this problem. We
do use GFI mailessentials on that server. We did contact GFI about this but
according to posts in this thread this problem also happens on freshly
installed Exchange Servers so I doubt that Thirdparty Software is the real
cause for this.
No problem if Microsoft does not know the answer (no one is perfect) but
just blaming others is lame ... :-(
Bye,
Alex
Actionxp
That issue happened a lot and got many complains.
Gary James
with MS. We have not tried any of the suggested workarounds in hopes that MS
will address our issue directly.
Tracey@discussions.microsoft.com Christina Tracey
Windows Server 2000 SP4. We also have Symantec Mail Security installed and
doing our anti-virus and anti-spam for us. I can't believe there isn't a fix
for this yet! I have not opened a case with Microsoft yet but will certainly
reference your EMEA case #SRZ060302000872 when I do. I was relieved to find
we weren't the only ones experiencing this odd Exchange behavior.
Erich Fritz
Exchange 2003 SP2
Windows Server 2003 Standard SP2
Symantec Client Security 3.1 and Symantec Mail Security 6.xx
I have not opened a case, I will call in and reference your case number.
No logs yet. I will enable SMTP logging.
I have applied the "workarounds" that have been suggested. So far things
have been fine. For the past week however the SMTP service will not restart
properly. It hangs on the startup after the scheduled task runs. All
searching I've done on the issue to this point has not brought me any
results. No recent software or MSFT patches have been applied that might
cause the issue.
Thanks,
Erich Fritz
Christina Tracey
I found that I could not implement the daily restart of SMTPSVC because
whenever I tried to stop the SMTP service it would hang -- it couldn't be
stopped. I tried the GlitchRetrySeconds setting workaround (set to 120) and
that hasn't helped. I opened a case with Microsoft this morning and after a
lot of runaround and being on the phone for almost two hours, they finally
admitted my problem was indeed the "known bug" and that someone else (a TSL -
not sure what that is) would be getting back to me. I'll let you know if I
ever get a working fix.
Thanks,
Christina Tracey
Yizhar Hurwitz
Finally, Hopefully, beter late the never, etc...
MS acknoledge the problem, and now provides the bug fix.
Here it is:
E-mail senders do not receive an indication that some messages have been
held by Exchange Server 2003 until the SMTP service, the Microsoft Exchange
Information Store service, or the Exchange server is restarted
http://support.microsoft.com/kb/950757/en-us
Almost 3 years after the bug was introduced at October 2005, hmmmm.....
Please reply here and write if you installed the hotfix KB950757 and if it
solved the problem.
--
Yizhar Hurwitz
http://yizhar.mvps.org