For those who doesn't know - greylisting is used on some mail servers to tempfail first attempt of an email, asking the sending server to retry later.
In short (more details will follow) - Exchange 2003 SP2 failes to re-queue messages sent to some servers that implement greylisting. This does not happen all the time (some messages go through but sometimes it fails). When the problem happens, those emails are hidden in some kind of a black hole, and the sender does not get an NDR nor Delay notification, even after those timeouts expire. Such messsages can remain "lost" for days or even weeks, until the SMTP or Information Store service is restarted. After a restart of SMTP service, Exchange suddenly finds those lost emails (I guess they were in the Mailbox Store), and retries to send them or returns NDR to the sender.
I have just openned a PSS case about this and working with Microsoft. However this issue is not easy to re-produce, so I would like to get feedback from you as well.
My questions to you - Have you encountered similar problems, such as users complaining that they got NDR for a message they sent 2 weeks before, or that the recipient calls them and tell them "why did I got now and email you sent a week ago?" (And you find out that the recipient server uses greylisting).
Meanwhile I have found those workarounds and currently I don't have the problem, but it still needs further investigations:
Workaround 1: Send emails to greylisting domains via an ISP Smart Host (using SMTP connector).
Workaround 2: Use scheduled tasks to restart SMTP service every day.
Workaround 3: Change SMTP virtual server retry timeouts (this does not seem to help but I changed it anyway to values you can see below).
Here is a more detailed report that I have also sent to Microsoft PSS:
When sending outgoing email to mail server that implement greylisting, sometimes Exchange retries the message later (as expected), but sometimes Exchange simply does not retry delivery , and neither sends NDR nor Delay notification to the sender. Two weeks later when I restart the SMTP service for installing security updates (such as IMF updates), then Exchange tries again to send those "lost" messages.
The problem seems sporadic - sometime it works, sometimes it doesn't, with the same configuration.
The problem appears with several different destination domains. The common thing is that all of them use greylisting.
Description of the server A single server with the following software: Primary roles = DC + Exchange + File server. Windows 2003 Standard SP1 (upgarded from win2000 about 1 year ago). DC + DNS + FSMO ROLES (This is the only DC in the network). Exchange 2003 SP2 with IMF configured and enabled. Symatec Corporate 10.0.2 (file protection) + SMSMSE 5.0.1.208 (mail protection).
Connection to the internet: Cisco PIX 501 firewall ver 6.3(1) ADSL line to the ISP (PPPoE). ISP name = Bezeq International (www.bezeqint.net)
DNS settings: The same server is an internal DNS server, and uses ISP servers as "forwarders" .
SMTP connector settings: Send using DNS (the default).
SMTP virtual server settings (related to the issue): Logging = NCSA common log file format. Delivery retry interval (I have change the defaults): First retry = 1 minute Second retry = 2 minutes Third retry = 15 minutes Subsequent retry = 30 minutes Delay notification = 1 hours Expiration timeout = 2 days
---------------- This post is a suggestion for Microsoft, and Microsoft responds to the suggestions with the most votes. To vote for this suggestion, click the "I Agree" button in the message pane. If you do not see the button, follow this link to open the suggestion in the Microsoft Web-based Newsreader and then click "I Agree" in the message pane.
>For those who doesn't know - greylisting is used on some mail servers to >tempfail first attempt of an email, asking the sending server to retry later.
>In short (more details will follow) - Exchange 2003 SP2 failes to re-queue >messages sent to some servers that implement greylisting. >This does not happen all the time (some messages go through but sometimes it >fails). >When the problem happens, those emails are hidden in some kind of a black >hole, and the sender does not get an NDR nor Delay notification, even after >those timeouts expire. >Such messsages can remain "lost" for days or even weeks, until the SMTP or >Information Store service is restarted. >After a restart of SMTP service, Exchange suddenly finds those lost emails >(I guess they were in the Mailbox Store), and retries to send them or returns >NDR to the sender.
>I have just openned a PSS case about this and working with Microsoft. >However this issue is not easy to re-produce, so I would like to get >feedback from you as well.
>My questions to you - >Have you encountered similar problems, such as users complaining that they >got NDR for a message they sent 2 weeks before, or that the recipient calls >them and tell them "why did I got now and email you sent a week ago?" >(And you find out that the recipient server uses greylisting).
>Meanwhile I have found those workarounds and currently I don't have the >problem, but it still needs further investigations:
>Workaround 1: Send emails to greylisting domains via an ISP Smart Host >(using SMTP connector).
>Workaround 2: Use scheduled tasks to restart SMTP service every day.
>Workaround 3: Change SMTP virtual server retry timeouts (this does not seem >to help but I changed it anyway to values you can see below).
>Here is a more detailed report that I have also sent to Microsoft PSS:
>When sending outgoing email to mail server that implement greylisting, >sometimes Exchange retries the message later (as expected), >but sometimes Exchange simply does not retry delivery , >and neither sends NDR nor Delay notification to the sender. >Two weeks later when I restart the SMTP service for installing security >updates (such as IMF updates), >then Exchange tries again to send those "lost" messages.
>The problem seems sporadic - sometime it works, sometimes it doesn't, with >the same configuration.
>The problem appears with several different destination domains. The common >thing is that all of them use greylisting.
>Description of the server >A single server with the following software: >Primary roles = DC + Exchange + File server. >Windows 2003 Standard SP1 (upgarded from win2000 about 1 year ago). >DC + DNS + FSMO ROLES (This is the only DC in the network). >Exchange 2003 SP2 with IMF configured and enabled. >Symatec Corporate 10.0.2 (file protection) + SMSMSE 5.0.1.208 (mail >protection).
>Connection to the internet: >Cisco PIX 501 firewall ver 6.3(1) >ADSL line to the ISP (PPPoE). >ISP name = Bezeq International (www.bezeqint.net)
>DNS settings: >The same server is an internal DNS server, and uses ISP servers as >"forwarders" .
>SMTP connector settings: >Send using DNS (the default).
>SMTP virtual server settings (related to the issue): >Logging = NCSA common log file format. >Delivery retry interval (I have change the defaults): >First retry = 1 minute >Second retry = 2 minutes >Third retry = 15 minutes >Subsequent retry = 30 minutes >Delay notification = 1 hours >Expiration timeout = 2 days
I would think using a SMTP gateway for all the outbound mail is the best solution.
> I would think using a SMTP gateway for all the outbound mail is the > best solution.
I agree, but:
The sepcific ISP smart host servers are problematic and have limits on relay traffic which causes other problems.
I have found my workarounds, but there is still something to be fixed in Exchange - and I would like to help Microsoft find and fix it, to avoid future problems at other Exchange sites.
> >For those who doesn't know - greylisting is used on some mail servers to > >tempfail first attempt of an email, asking the sending server to retry later.
> >In short (more details will follow) - Exchange 2003 SP2 failes to re-queue > >messages sent to some servers that implement greylisting. > >This does not happen all the time (some messages go through but sometimes it > >fails). > >When the problem happens, those emails are hidden in some kind of a black > >hole, and the sender does not get an NDR nor Delay notification, even after > >those timeouts expire. > >Such messsages can remain "lost" for days or even weeks, until the SMTP or > >Information Store service is restarted. > >After a restart of SMTP service, Exchange suddenly finds those lost emails > >(I guess they were in the Mailbox Store), and retries to send them or returns > >NDR to the sender.
> >I have just openned a PSS case about this and working with Microsoft. > >However this issue is not easy to re-produce, so I would like to get > >feedback from you as well.
> >My questions to you - > >Have you encountered similar problems, such as users complaining that they > >got NDR for a message they sent 2 weeks before, or that the recipient calls > >them and tell them "why did I got now and email you sent a week ago?" > >(And you find out that the recipient server uses greylisting).
> >Meanwhile I have found those workarounds and currently I don't have the > >problem, but it still needs further investigations:
> >Workaround 1: Send emails to greylisting domains via an ISP Smart Host > >(using SMTP connector).
> >Workaround 2: Use scheduled tasks to restart SMTP service every day.
> >Workaround 3: Change SMTP virtual server retry timeouts (this does not seem > >to help but I changed it anyway to values you can see below).
> >Here is a more detailed report that I have also sent to Microsoft PSS:
> >When sending outgoing email to mail server that implement greylisting, > >sometimes Exchange retries the message later (as expected), > >but sometimes Exchange simply does not retry delivery , > >and neither sends NDR nor Delay notification to the sender. > >Two weeks later when I restart the SMTP service for installing security > >updates (such as IMF updates), > >then Exchange tries again to send those "lost" messages.
> >The problem seems sporadic - sometime it works, sometimes it doesn't, with > >the same configuration.
> >The problem appears with several different destination domains. The common > >thing is that all of them use greylisting.
> >Description of the server > >A single server with the following software: > >Primary roles = DC + Exchange + File server. > >Windows 2003 Standard SP1 (upgarded from win2000 about 1 year ago). > >DC + DNS + FSMO ROLES (This is the only DC in the network). > >Exchange 2003 SP2 with IMF configured and enabled. > >Symatec Corporate 10.0.2 (file protection) + SMSMSE 5.0.1.208 (mail > >protection).
> >Connection to the internet: > >Cisco PIX 501 firewall ver 6.3(1) > >ADSL line to the ISP (PPPoE). > >ISP name = Bezeq International (www.bezeqint.net)
> >DNS settings: > >The same server is an internal DNS server, and uses ISP servers as > >"forwarders" .
> >SMTP connector settings: > >Send using DNS (the default).
> >SMTP virtual server settings (related to the issue): > >Logging = NCSA common log file format. > >Delivery retry interval (I have change the defaults): > >First retry = 1 minute > >Second retry = 2 minutes > >Third retry = 15 minutes > >Subsequent retry = 30 minutes > >Delay notification = 1 hours > >Expiration timeout = 2 days
> I would think using a SMTP gateway for all the outbound mail is the > best solution.
There is clearly a bug with the handling of messages put on hold by the recipient server.
As a reference, the greylister does this : Final-Recipient: rfc822;...@... Action: failed Status: 4.0.0 Diagnostic-Code: smtp;450 <...@...>: Recipient address rejected: Greylisted for 300 seconds (see...
Visibly, exchange 2k3 does not handle this SMTP answer properly and then lose the message until an SMTP restart.
The original post was in April, is there any plan to fix this issue anytime soon ?
> > I would think using a SMTP gateway for all the outbound mail is the > > best solution.
> I agree, but:
> The sepcific ISP smart host servers are problematic and have limits on relay > traffic which causes other problems.
> I have found my workarounds, but there is still something to be fixed in > Exchange - > and I would like to help Microsoft find and fix it, to avoid future problems > at other Exchange sites.
> > >For those who doesn't know - greylisting is used on some mail servers to > > >tempfail first attempt of an email, asking the sending server to retry later.
> > >In short (more details will follow) - Exchange 2003 SP2 failes to re-queue > > >messages sent to some servers that implement greylisting. > > >This does not happen all the time (some messages go through but sometimes it > > >fails). > > >When the problem happens, those emails are hidden in some kind of a black > > >hole, and the sender does not get an NDR nor Delay notification, even after > > >those timeouts expire. > > >Such messsages can remain "lost" for days or even weeks, until the SMTP or > > >Information Store service is restarted. > > >After a restart of SMTP service, Exchange suddenly finds those lost emails > > >(I guess they were in the Mailbox Store), and retries to send them or returns > > >NDR to the sender.
> > >I have just openned a PSS case about this and working with Microsoft. > > >However this issue is not easy to re-produce, so I would like to get > > >feedback from you as well.
> > >My questions to you - > > >Have you encountered similar problems, such as users complaining that they > > >got NDR for a message they sent 2 weeks before, or that the recipient calls > > >them and tell them "why did I got now and email you sent a week ago?" > > >(And you find out that the recipient server uses greylisting).
> > >Meanwhile I have found those workarounds and currently I don't have the > > >problem, but it still needs further investigations:
> > >Workaround 1: Send emails to greylisting domains via an ISP Smart Host > > >(using SMTP connector).
> > >Workaround 2: Use scheduled tasks to restart SMTP service every day.
> > >Workaround 3: Change SMTP virtual server retry timeouts (this does not seem > > >to help but I changed it anyway to values you can see below).
> > >Here is a more detailed report that I have also sent to Microsoft PSS:
> > >When sending outgoing email to mail server that implement greylisting, > > >sometimes Exchange retries the message later (as expected), > > >but sometimes Exchange simply does not retry delivery , > > >and neither sends NDR nor Delay notification to the sender. > > >Two weeks later when I restart the SMTP service for installing security > > >updates (such as IMF updates), > > >then Exchange tries again to send those "lost" messages.
> > >The problem seems sporadic - sometime it works, sometimes it doesn't, with > > >the same configuration.
> > >The problem appears with several different destination domains. The common > > >thing is that all of them use greylisting.
> > >Description of the server > > >A single server with the following software: > > >Primary roles = DC + Exchange + File server. > > >Windows 2003 Standard SP1 (upgarded from win2000 about 1 year ago). > > >DC + DNS + FSMO ROLES (This is the only DC in the network). > > >Exchange 2003 SP2 with IMF configured and enabled. > > >Symatec Corporate 10.0.2 (file protection) + SMSMSE 5.0.1.208 (mail > > >protection).
> > >Connection to the internet: > > >Cisco PIX 501 firewall ver 6.3(1) > > >ADSL line to the ISP (PPPoE). > > >ISP name = Bezeq International (www.bezeqint.net)
> > >DNS settings: > > >The same server is an internal DNS server, and uses ISP servers as > > >"forwarders" .
> > >SMTP connector settings: > > >Send using DNS (the default).
> > >SMTP virtual server settings (related to the issue): > > >Logging = NCSA common log file format. > > >Delivery retry interval (I have change the defaults): > > >First retry = 1 minute > > >Second retry = 2 minutes > > >Third retry = 15 minutes > > >Subsequent retry = 30 minutes > > >Delay notification = 1 hours > > >Expiration timeout = 2 days
> > I would think using a SMTP gateway for all the outbound mail is the > > best solution.
>> > I would think using a SMTP gateway for all the outbound mail is the >> > best solution.
>> I agree, but:
>> The sepcific ISP smart host servers are problematic and have limits on relay >> traffic which causes other problems.
>> I have found my workarounds, but there is still something to be fixed in >> Exchange - >> and I would like to help Microsoft find and fix it, to avoid future problems >> at other Exchange sites.
>> > >For those who doesn't know - greylisting is used on some mail servers to >> > >tempfail first attempt of an email, asking the sending server to retry later.
>> > >In short (more details will follow) - Exchange 2003 SP2 failes to re-queue >> > >messages sent to some servers that implement greylisting. >> > >This does not happen all the time (some messages go through but sometimes it >> > >fails). >> > >When the problem happens, those emails are hidden in some kind of a black >> > >hole, and the sender does not get an NDR nor Delay notification, even after >> > >those timeouts expire. >> > >Such messsages can remain "lost" for days or even weeks, until the SMTP or >> > >Information Store service is restarted. >> > >After a restart of SMTP service, Exchange suddenly finds those lost emails >> > >(I guess they were in the Mailbox Store), and retries to send them or returns >> > >NDR to the sender.
>> > >I have just openned a PSS case about this and working with Microsoft. >> > >However this issue is not easy to re-produce, so I would like to get >> > >feedback from you as well.
>> > >My questions to you - >> > >Have you encountered similar problems, such as users complaining that they >> > >got NDR for a message they sent 2 weeks before, or that the recipient calls >> > >them and tell them "why did I got now and email you sent a week ago?" >> > >(And you find out that the recipient server uses greylisting).
>> > >Meanwhile I have found those workarounds and currently I don't have the >> > >problem, but it still needs further investigations:
>> > >Workaround 1: Send emails to greylisting domains via an ISP Smart Host >> > >(using SMTP connector).
>> > >Workaround 2: Use scheduled tasks to restart SMTP service every day.
>> > >Workaround 3: Change SMTP virtual server retry timeouts (this does not seem >> > >to help but I changed it anyway to values you can see below).
>> > >Here is a more detailed report that I have also sent to Microsoft PSS:
>> > >When sending outgoing email to mail server that implement greylisting, >> > >sometimes Exchange retries the message later (as expected), >> > >but sometimes Exchange simply does not retry delivery , >> > >and neither sends NDR nor Delay notification to the sender. >> > >Two weeks later when I restart the SMTP service for installing security >> > >updates (such as IMF updates), >> > >then Exchange tries again to send those "lost" messages.
>> > >The problem seems sporadic - sometime it works, sometimes it doesn't, with >> > >the same configuration.
>> > >The problem appears with several different destination domains. The common >> > >thing is that all of them use greylisting.
>> > >Description of the server >> > >A single server with the following software: >> > >Primary roles = DC + Exchange + File server. >> > >Windows 2003 Standard SP1 (upgarded from win2000 about 1 year ago). >> > >DC + DNS + FSMO ROLES (This is the only DC in the network). >> > >Exchange 2003 SP2 with IMF configured and enabled. >> > >Symatec Corporate 10.0.2 (file protection) + SMSMSE 5.0.1.208 (mail >> > >protection).
>> > >Connection to the internet: >> > >Cisco PIX 501 firewall ver 6.3(1) >> > >ADSL line to the ISP (PPPoE). >> > >ISP name = Bezeq International (www.bezeqint.net)
>> > >DNS settings: >> > >The same server is an internal DNS server, and uses ISP servers as >> > >"forwarders" .
>> > >SMTP connector settings: >> > >Send using DNS (the default).
>> > >SMTP virtual server settings (related to the issue): >> > >Logging = NCSA common log file format. >> > >Delivery retry interval (I have change the defaults): >> > >First retry = 1 minute >> > >Second retry = 2 minutes >> > >Third retry = 15 minutes >> > >Subsequent retry = 30 minutes >> > >Delay notification = 1 hours >> > >Expiration timeout = 2 days
>> > I would think using a SMTP gateway for all the outbound mail is the >> > best solution.
I hope this forum is the right way to get Microsoft to place some attention on this odd behavior. It's Feb 2007, and searching for "GREYLIST" in microsoft KB comes up with no hits. Will this problem at least be acknowledged by the QA team ?
I have a recent Exchange 2003 SP2 install that is not a DC, and exhibits the problem described very well by Hurwitz. It took a full day of analysing the behavior in SMTP logs and user reports, and then web searching just to got to this thread.
In particular greylist servers that send the SMTP 451 temporary error, sometimes trigger this black holing of emails until server restart. In the past month I had at least 33 messages 'dissappear'.
Here is an example of a message that dissappeared from the SMTPSVC log. Private data obscured. It is curious that Exchange, in this example, retried then gave up. The resend never appears again in the logs until after the server was restarted. Silent non-delivery is a SERIOUS problem.
Just some notes to update those that are reading about the problem:
* The workarounds described in my initial post here, worked for me. I don't know which one of them did the trick.
1. Use "Microsoft Update"/automatic updates to install latest updates for Exchange (and Windows).
2. Use a batch file to restart the SMTP service every day.
3. Take a look at the other workarounds that I wrote in my previos post here.
* I have openned a case at PSS, about a year ago, but the case was closed, simply because I wasn't able to reproduce the problem. The Microsoft PSS team were a bit small-minded, and focused on my specific case only. They didn't try to reserach and contact other sysadmins globaly that had the same problem, and they didn't escelate it to the development team.
* Since posting here, I get once in a several weeks an email from another sysadmin around the world, facing the same problem. If someone at Microsoft will take the intiative (or an MVP) - I will send him a list of the contact sysadmins experiencing this problem that contacted me about it.
Anyone else have news about this old undocumented bug in Exchange 2003 SP2?
"Kazys" wrote: > I hope this forum is the right way to get Microsoft to place some attention > on this odd behavior. It's Feb 2007, and searching for "GREYLIST" in > microsoft KB comes up with no hits. Will this problem at least be > acknowledged by the QA team ?
> I have a recent Exchange 2003 SP2 install that is not a DC, and exhibits the > problem described very well by Hurwitz. It took a full day of analysing the > behavior in SMTP logs and user reports, and then web searching just to got to > this thread.
> In particular greylist servers that send the SMTP 451 temporary error, > sometimes trigger this black holing of emails until server restart. In the > past month I had at least 33 messages 'dissappear'.
> Here is an example of a message that dissappeared from the SMTPSVC log. > Private data obscured. It is curious that Exchange, in this example, retried > then gave up. The resend never appears again in the logs until after the > server was restarted. Silent non-delivery is a SERIOUS problem.
