Mail.Global.FrontBridge.com
Arnim Sommer
unfortunately, @work we are hosted at microsoftonline.com.
Since monday, we are getting mail delays with one of our partners, and
I could track it down to address verification getting timeouts.
Has anyone else seen that problem, or is it just one isolated problem?
Thank you for your input.
Wietse Venema
See the "Pipelining of Sender Verify" thread, started yesterday.
This thread has SMTP session transcripts that allow anyone
to verify the nature of the problem.
The transcripts show that Microsoft mis-implements PIPELINING and
hangs up on receipt of QUIT, before it has replied to all the
commands that were sent before QUIT.
Older Frontbridge software did not have that bug (they were based
on Postfix). The newer version uses Microsoft's own mail software
which mis-implements RFC 2920.
Wietse
Arnim Sommer
Silly me...
Terry Gilsenan
From: owner-pos...@postfix.org [owner-pos...@postfix.org] On Behalf Of Arnim Sommer [rant...@googlemail.com]
Sent: Friday, 26 November 2010 11:11 PM
To: postfi...@postfix.org
Subject: Mail.Global.FrontBridge.com
>Hello,
>unfortunately, @work we are hosted at microsoftonline.com.
>
>Since monday, we are getting mail delays with one of our partners, and
>I could track it down to address verification getting timeouts.
>
>Has anyone else seen that problem, or is it just one isolated problem?
>
>Thank you for your input.
>
Hello,
In my experience....:
Microsoft's b0rged hosted exchange and frontbridge system is very broken.
Firstly from an MX (inward smtp) point of view, they smell very spammy, rDNS in one domain, helo in another, and envelope sender from yet another.
Added to this sender verification does not work, and if you are using greylisting, the chances are very good that the triplet will never be seen again, which means a delivery failure for their customers.
I have had to poke a lot of holes into our system to cope with frontbridge customers. This has meant that all the spammers that relay through microsofts customers, or even via Microsoft hosted exchange directly, get delivery to our system. The amount of spam in our mailboxes has trippled since i removed sender verification for frontbridge.com and bigfish.com (yep that is the Helo domain - and probably demonstrated Microsofts view of themselves.)
Microsoft pay no heed to standards, they are a marketing machine. Open Source, and RFC's are unlikely to bring in the buck$ for them, so instead they invent their own.
How to fix...:
Put pressure on Microsofts customers. Microsoft is not interested in what you or I say, but they suddenly get interested when their customers vote with their checkbooks.
When Microsofts customers complain that they cant send email to your users, be conforted in the knowledge that there are a lot of places they can not send email now, based on Microsofts brokenness. Microsoft doesnt care.
My 2 bits.
T
Michael J Wise
ofFullDisclosure, I work at what was formerly known as BigFish, then Frontbridge, but is now Forefront Online for Office, in the capacity of Knowledge Engineer (Spam Analysis), and among other things help out with abuse and deliverability issues.
And also... this discussion really doesn't seem apropos to the Postfix mailing list, so this will probably be my only post on the subject.
And no, I don't have any input into anything to do with Exchange, etc. I'm here simply because I use Postfix on my home server, at least in part because that is what we used at Previous Employ.
That having been said....
On Nov 26, 2010, at 1:36 PM, Terry Gilsenan wrote:
> Hello,
> In my experience....:
Ok, granted.
> Microsoft's b0rged hosted exchange and frontbridge system is very broken.
So ... the mail doesn't work at all, or it doesn't work the way you expect it to?
If our customers have problems getting their mail delivered, we work with whomever to get the issue resolved.
We doesn't seem to have too many issues getting its customers' mail delivered.
And yes, I'd be in a position to know.
> Firstly from an MX (inward smtp) point of view, they smell very spammy, rDNS in one domain, helo in another, and envelope sender from yet another.
You mean that from the point of view of you being the MX and receiving traffic from them?
Envelope sender will always be unpredictable for a filtering service with many, many customer domains behind it, that's a given.
And for rDNS and HELO ... there are many reasons why the rDNS and the HELO might disagree, especially if load balancers are involved, which they are.
> Added to this sender verification does not work, ...
From who's point of view?
Are we still talking MX?
This ... seems to be your primary beef.
> and if you are using greylisting, the chances are very good that the triplet will never be seen again, which means a delivery failure for their customers.
Unless emergency situations so dictate, if a triplet comes at you from one IP address and you greylist it, it's going to be coming back at you from that same IP address when it retries. If situations DO so dictate, the message might be moved to another server, but this is by no means common.
> I have had to poke a lot of holes into our system to cope with frontbridge customers.
What kinds of holes?
> This has meant that all the spammers ...
Others SEEM to be of the opinion that our outbounds are rather clean, considering.
This is the first one I checked:
<https://www.senderscore.org/lookup.php?lookup=65.55.88.11&ipLookup.x=25&ipLookup.y=9>
> that relay through microsofts customers, ...
If they try, they don't relay for long.
> or even via Microsoft hosted exchange directly, get delivery to our system.
Do you offer a feedback loop? :)
> The amount of spam in our mailboxes has trippled since i removed sender verification for frontbridge.com ...
All from Frontbridge?
To what domain?
And when you say, "Sender Verification", I'm not sure what you mean.
Are you trying to make sure that the sender address exists?
That's not always going to work, since some of our customers (despite begging and pleading) do not share with us their user data... so that can make us knowing if they have defined a given username difficult at times. We have a very heterogenous customer-base.
But abuse is not allowed, and is dealt with rather rapidly.
> and bigfish.com (yep that is the Helo domain - and probably demonstrated Microsofts view of themselves.)
Bigfish.com was the original domain name of the service about ... I think it was a decade or so ago.
Microsoft acquired it about 5 years ago.
So the domain name has nothing to do with Microsoft's opinion of itself.
> Microsoft pay no heed to standards, ...
Microsoft pays heed to standards, or a lot of the Internet just wouldn't work.
> they are a marketing machine.
That, plus a lot of other things as well. We've been known to write the odd piece of software.
> Open Source, and RFC's are unlikely to bring in the buck$ for them, so instead they invent their own.
Open Source and RFCs as such don't bring in bucks at all, by definition.
Aloha,
Michael.
--
"Please have your Internet License http://kapu.net/~mjwise/
and Usenet Registration handy..."
Stephen Thorne
> <snipped incorherant email>
Michael,
Here is a very quick description of the issue.
Frontbridge advertises that it supports pipelined requests. But it
disconnects after the current request is served if any request in the
pipeline is a QUIT.
This causes a serious problem because sender address verification, in
effect, asks all at once, "ehlo can i send from this address to this
address, nevermind, bye!"
Frontbridge responds with "hello, sender ok, bye!" When the valid thing
to do is to say, "hello, sender ok, recipient ok, reset ok, bye!"
Our only option at this point is to turn off pipelining when talking to
frontbridge, because frontbridge's pipelines are broken. This is a
terribly negative thing. It would be much better if this broken
behaviour were fixed. Do you think you could use your contacts to raise
this issue with the right people?
--
Regards,
Stephen Thorne
Development Engineer
Netbox Blue
Michael J Wise
I'll do my best, but no guarantees.
I will do my best.
Michael J Wise
By the way, important aside...
Timeframes.
I do not know how fast (or even if) this can be addressed, it's so not my department, but I will do my best to present the issue to the right people and assert its importance and annoyance factor to the rest of the community, but ... it's not gonna be fixed Monday morning. It may take quite some time. Think "Months", possibly many. We don't just let anyone with an editor open up a key source file, make some tweaks, and promote it to production.
I'm just saying, ok?
Just trying to pre-emptively manage expectations, because the code in question is not managed by us, it's ... way over there, we're just clients as such.
But I will do my best.
Victor Duchovni
> That having been said....
>
> On Nov 26, 2010, at 1:36 PM, Terry Gilsenan wrote:
Yes, Terry was posturing. No need to take it too seriously. Putting the
posturing aside, there is an underlying interoperability issue. The new
software behind mail.global.frontbridge.com does not correctly implement
RFC 2920. Only the first command in a group is processed when the group
ends in QUIT.
With a bit of luck (and likely some time) this will be fixed. I am
reposting what it takes to reproduce the problem. That's the substance
of this thread, the rest is idle chatter...
RFC 2920 Failure:
Client IP and envelope sender/recipient masked as they are not germane.
S> 220 VA3EHSMHS012.bigfish.com Microsoft ESMTP MAIL Service ready at Sat, 27 Nov 2010 06:47:57 +0000
C> EHLO amnesiac.example.com
S> 250-VA3EHSMHS012.bigfish.com Hello [192.0.2.1]
S> 250-SIZE 157286400
S> 250-PIPELINING
S> 250-ENHANCEDSTATUSCODES
S> 250-STARTTLS
S> 250-AUTH
S> 250-8BITMIME
S> 250-BINARYMIME
S> 250 CHUNKING
C> MAIL FROM:<sen...@example.com>
C> RCPT TO:<reci...@example.com>
C> RSET
C> QUIT
S> 250 2.1.0 Sender OK
S> 221 2.0.0 Service closing transmission channel
<client sees premature EOF>
Delaying the QUIT results in an RFC-compliant interaction:
S> 220 VA3EHSMHS012.bigfish.com Microsoft ESMTP MAIL Service ready at Sat, 27 Nov 2010 06:47:57 +0000
C> EHLO amnesiac.example.com
S> 250-VA3EHSMHS012.bigfish.com Hello [192.0.2.1]
S> 250-SIZE 157286400
S> 250-PIPELINING
S> 250-ENHANCEDSTATUSCODES
S> 250-STARTTLS
S> 250-AUTH
S> 250-8BITMIME
S> 250-BINARYMIME
S> 250 CHUNKING
C> MAIL FROM:<sen...@example.com>
C> RCPT TO:<reci...@example.com>
C> RSET
S> 250 2.1.0 Sender OK
S> 250 2.1.5 Recipient OK
S> 250 2.0.0 Resetting
C> QUIT
S> 221 2.0.0 Service closing transmission channel
Because Postfix does not typically wait for a "QUIT" response, it would
cost Postfix little to delay "QUIT", but this would tie up resources on
the receiving server, which is now tied up for another RTT.
So the pipelined QUIT is Postfix being a good citizen, and it would be
best if the problem were fixed at the server which advertises RFC 2920
compliance, but does not fully implement the specification.
--
Viktor.
Victor Duchovni
> I do not know how fast (or even if) this can be addressed, it's so
> not my department, but I will do my best to present the issue to the
> right people and assert its importance and annoyance factor to the rest
> of the community, but ... it's not gonna be fixed Monday morning. It
> may take quite some time.
Yes, we'll wait. It took a while before PIPELINING of
[message-content]<CRLF>
.<CRLF>
QUIT<CRLF>
was fixed, but it did get fixed eventually.
http://archives.neohapsis.com/archives/postfix/2003-02/0469.html
http://support.microsoft.com/default.aspx?scid=kb;EN-US;813050
Postfix is perhaps more modular than Exchange, and with just two or so
developers looking at the code, reports of a similar bug in the Postfix
smtpd(8) probably would result in a fix by Monday. :-)
--
Viktor.
Mihira Fernando
> Postfix is perhaps more modular than Exchange, and with just two or so
> developers looking at the code, reports of a similar bug in the Postfix
> smtpd(8) probably would result in a fix by Monday. :-)
Robert Schetterer
> _______________________________________
> From: owner-pos...@postfix.org [owner-pos...@postfix.org] On Behalf Of Arnim Sommer [rant...@googlemail.com]
> Sent: Friday, 26 November 2010 11:11 PM
> To: postfi...@postfix.org
> Subject: Mail.Global.FrontBridge.com
>
>> Hello,
>> unfortunately, @work we are hosted at microsoftonline.com.
>>
>> Since monday, we are getting mail delays with one of our partners, and
>> I could track it down to address verification getting timeouts.
>>
>> Has anyone else seen that problem, or is it just one isolated problem?
>>
>> Thank you for your input.
>>
>
> Hello,
> In my experience....:
>
youre right
>
> Firstly from an MX (inward smtp) point of view, they smell very spammy, rDNS in one domain, helo in another, and envelope sender from yet another.
>
> Added to this sender verification does not work, and if you are using greylisting, the chances are very good that the triplet will never be seen again, which means a delivery failure for their customers.
>
> I have had to poke a lot of holes into our system to cope with frontbridge customers. This has meant that all the spammers that relay through microsofts customers, or even via Microsoft hosted exchange directly, get delivery to our system. The amount of spam in our mailboxes has trippled since i removed sender verification for frontbridge.com and bigfish.com (yep that is the Helo domain - and probably demonstrated Microsofts view of themselves.)
>
> Microsoft pay no heed to standards, they are a marketing machine. Open Source, and RFC's are unlikely to bring in the buck$ for them, so instead they invent their own.
>
> How to fix...:
>
> Put pressure on Microsofts customers. Microsoft is not interested in what you or I say, but they suddenly get interested when their customers vote with their checkbooks.
>
> When Microsofts customers complain that they cant send email to your users, be conforted in the knowledge that there are a lot of places they can not send email now, based on Microsofts brokenness. Microsoft doesnt care.
>
> My 2 bits.
>
> T
the microsoft customer service as well others from big companies are
nightmares, if they react , it takes weeks at minimum, the only way may
get faster is by pressing from customer itself, mostly you have to fix
problems at your side
i think investing time to disuss that is loose
--
Best Regards
MfG Robert Schetterer
Germany/Munich/Bavaria
Morten P.D. Stevens
> Microsoft's b0rged hosted exchange and frontbridge system is very broken.
Yes, that's right.
The other question is ...
Why has Microsoft changed from Postfix to Forefront?
Postfix is faster, safer and more compatible with RFC.
> Added to this sender verification does not work, and if you are using greylisting, the chances are very good that the triplet will never be seen again, which means a delivery failure for their customers.
I think greylisting is the worst thing you can do.
I suggest to use postscreen and amavisd-new as pre-queue filter for rejecting spam mails directly in the smtp transaction.
Since postfix 2.8 is not yet available, we do this currently with sendmails greet_pause feature and amavisd-new as pre pre-queue filter.
Since then greylisting is unnecessary for us.
Best regards,
Morten
Wietse Venema
> On Fri, Nov 26, 2010 at 11:03:58PM -0800, Michael J Wise wrote:
>
> > I do not know how fast (or even if) this can be addressed, it's so
> > not my department, but I will do my best to present the issue to the
> > right people and assert its importance and annoyance factor to the rest
> > of the community, but ... it's not gonna be fixed Monday morning. It
> > may take quite some time.
>
> Yes, we'll wait. It took a while before PIPELINING of
>
> [message-content]<CRLF>
> .<CRLF>
> QUIT<CRLF>
>
> was fixed, but it did get fixed eventually.
>
> http://archives.neohapsis.com/archives/postfix/2003-02/0469.html
>
> http://support.microsoft.com/default.aspx?scid=kb;EN-US;813050
>
> developers looking at the code, reports of a similar bug in the Postfix
> smtpd(8) probably would result in a fix by Monday. :-)
And it would take years before people can use it, because many
distributions only do security updates. Open source lag times
can be as bad as that of any closed-source product.
I think it would be helpful if someone could produce a short and
precise writeup of the problem, how to reproduce, what part of RFC
2920 is violated, and pointers to Microsoft bugfixes that solve
other problems in this area. Then I can put it on a postfix.org
webpage and help people who want to report this bug to Microsoft.
It would also be helpful if this note made clear that this is not
a Postfix address verification problem. That would encourage MS to
just fix the mail-rcpt-rset-quit case and not the underlying problem.
On the Postfix side, the workaround for Microsoft's Frontbridge
would be a regular expression map that matches /^220.+Microsoft/
and that turns on workarounds such as disable_esmtp, delay_dotcrlf,
delay_quit and more as new bugs come to light. Then, in three years
time it will be standard in the binary distributions.
I don't have time to implement the above immediately. I was going
to work on non-Postfix, but someone else did not deliver, so I used
the time to implement the (DNSXL) address pattern matcher instead.
Wietse
DTNX/NGMX Postmaster
>> Microsoft pay no heed to standards, ...
>
> Microsoft pays heed to standards, or a lot of the Internet just wouldn't work.
This does seem a bit funny, given the subject under discussion. I understand that it is a big gorilla, and hard to turn on a dime, if at all. I also understand that it is a complex infrastructure consisting of a mix of frontend and backend servers, load balancers etcetera.
But put your money where your mouth is, I'd say ;-) Working 'postmaster' accounts for the 'frontbridge.com' and 'bigfish.com' domains, read by clueful people would be a good start, instead of the bounce I got when we tried to report the PIPELINING issue on November 8th;
<postm...@frontbridge.com>: host 10.9.14.23[10.9.14.23] said: 550
<postm...@frontbridge.com>: Recipient address rejected: User unknown in
virtual mailbox table (in reply to RCPT TO command)
Reporting-MTA: dns; mail15-tx2-R.bigfish.com
X-Postfix-Queue-ID: 295AE2E03D4
X-Postfix-Sender: rfc822; sup...@ur.nl
Arrival-Date: Mon, 8 Nov 2010 21:18:23 +0000 (UTC)
Or the runaround we got when we tried alternate channels to report a problem, last year, with missing DNS records for some of the nodes in the cluster. The person it eventually got to wanted to know why we wanted to contact 'postm...@bigfish.com', when he got the (fully documented) problem right there in the same e-mail. Another November 8th, actually, perhaps that's just a bad date over there ;-)
We currently have a '*.bigfish.com' exception in place for our EHLO resolve check, and for the PIPELINING problem we've got this in our 'main.cf';
==
# Added to deal with MS Frontbridge madness (2010/11/08 21:50)
smtp_discard_ehlo_keywords = pipelining
==
I bet most of the people here are not interested in Microsoft bashing, they just want to keep the spice flowing. And given Microsoft's history, this being ANOTHER problem in a long line of issues the average mail admin has had to deal with with regard to Microsoft products and services, without being able to get a proper response from a human being ... perhaps you can understand where some of that frustration originates :-)
Anyway, thanks for the response, and the effort you're putting in. Hope you'll be able to make a fix happen.
Cya,
Jona