Postfix does not re-encode to base64/qouted-printable.
Wietse Venema
> Hi,
>
> I recently talked to a big site who run a commercial MTA (that does not
> get RFC-822 right, BTW) and are considering to switch their MTA to the
> end of the year. They have categorically excluded sendmail, and seem to
> favor Solaris. They
>
> However, one of their topics is "can the new MTA re-encode 8-bit mail
> for 7-bit destinations." I know that *no* alternative that's in their
> choice does that. Not Postfix, not Exim, not qmail. They say "if the
> destination SMTP receiver does not advertise 8BITMIME, what do we do?
> Reencode or bounce are our options."
In a politically correct world, yes. However, in the real world it
is unlikely to find an MTA on the Internet that is not 8bit capable.
Sendmail has been 8-bit clean for many years now, and the machines
that run older versions should be retired by now.
> Well, of course I can tell them Postfix scales better, is faster, has
> less I/O, is more secure than Exim and Sendmail (on par with qmail
> AFAICS), tell them where to find information and offer personal
> vis-a-vis information on Postfix and qmail for free, tell them Postfix
> flexible, easy and smooth in operation, but some of my arguments may
> easily reek much like a "ex tribus malis minimum eligendum est"
> defaitist argumentation which is not precisely constructive.
I guess all three MTAs would do a decent job.
> I recall that once, there has been a patch by Pavel Yakovlev, to
> re-encode stuff to base64 should the recipient not advertise 8bit
> cleanliness, however, that patch is against 19990506 and when I tried it
> against a current version last year or early this year, it did not
> apply.
With a little work it would apply. However, this was one of those
partial solutions that skip over the hard part.
> I see Postfix and its maintainer are reluctant to parse MIME, and for
> good reason, as has been elaborately discussed more than two years ago
> on this list, OTOH, MIME awareness for content filtering has recently
> been mentioned.
Actually these are two different problems. Converting 8bit to 7bit
requires only a best effort. If someone sends an improperly
formatted MIME message then it is the sender's problem.
Detecting viruses on the other hand is more like building an
intrusion detection system. Just like an IDS, a content scanner
has to deal with ambiguity in MIME standards, with idiosyncracies
of MIME implementations (how does a Microsoft mail client handle
a message that violates MIME standards in some peculiar way?), and
with deliberate attempts to exploit those idiosyncracies in order
to slip past defenses. The issue reminds me of how an IDS deals
with overlapping IP fragments and with overlapping TCP retransmissions
- you don't really know how the end system is going to treat some
ambiguous data unless to completely rewrite the content to an
unambiguous form.
In other words, if someone sends an improperly formatted MIME
message and the content filter does not know that this message will
explode in some mail client, then it is not the sender's problem.
It is the recipient's problem.
> There has also been been mentioning of changes for client-side SMTP
> connection caching, would not the relevant changes also be the ideal
> place to introduce the code for a "bounce_8bit_for_7bit_destination"
> parameter?
With connection caching one separates connection management and
initial handshake from the actual message delivery. This means that
the connection and the EHLO handshake results are stored together.
> What are the next changes that the Snapshots will undergo?
I am killing myself to get the new snapshot out the door. Some MTA
authors seem to make their life a lot easier in this respect.
> How do I tell that site they can use Postfix although it does neither
> re-encode to 7bit nor bounce?
If none of the acceptable MTAs re-encodes 8-bit mail, what is the problem?
Wietse
-
To unsubscribe, send mail to majo...@postfix.org with content
(not subject): unsubscribe postfix-users
Ralf Hildebrandt
> Detecting viruses on the other hand is more like building an
> intrusion detection system. Just like an IDS, a content scanner
> has to deal with ambiguity in MIME standards, with idiosyncracies
> of MIME implementations (how does a Microsoft mail client handle
> a message that violates MIME standards in some peculiar way?), and
> with deliberate attempts to exploit those idiosyncracies in order
> to slip past defenses. The issue reminds me of how an IDS deals
Right now, Thomas Roessler is investigating this -- he wrote some code
that generates arbitrary messages violating the MIME standrads in all
ways possible. Then these messages are sent through various MTA's
using various MUA's and virus scanner products.
PolyMIME it's called.
> I am killing myself to get the new snapshot out the door.
^^^^^^^^^^^^^^^^^^^^
If that helps...
--
Ralf Hildebrandt http://www.arschkrebs.de
The only "intuitive" interface is the nipple. After that, it's all
learned.
Nick Simicich
>How do I tell that site they can use Postfix although it does neither
>re-encode to 7bit nor bounce?
I think that in the years I have run postfix, I may have had one piece of
mail bounce because it was 8 bit and going to a 7 bit destination so the
other end bounced it....in general, what happens if you just send
it? Admittedly. most of the mail that goes through my system is crudely
folded to 7 bit by demime.
7 bit....I think that there was once a transport that was not 8 bit
safe. We are still paying for the people who originally defined mail not
just saying, "Gosh, 7 bit transport....that is a problem you have, you
should solve it, mail will be 8 bit." But it didn't seem important because
everyone uses US-ASCII :-) and it is too late now, this is the way it is.
I know that you are required to assume that anyone that either HELOs or
does not respond with 8BITMIME to EHLO is not 8 bit safe, and furthermore,
if you have an 8 bit message, sending it to anyone who does not EHLO
including 8BITMIME without re-encoding it using base64 or quoted-printable
violates http://www.faqs.org/rfcs/rfc1652.html. rfc1652 allows you to
re-encode messages and stay within standards, but it does not require such
encoding, you can bounce.
This is clearly more of a problem for folks who commonly use 8 bit
character sets. Sendmail supports just sending 8 bit mail regardless of
what the receiver says with the smtp8 mailer, but that is probably rfc
ignorant.
I could see one easy fix - there is the whole interface for virus scanning
- a stage could be added to that whole procedure which simply re-codes
every 8 bit message as 7 bit quoted printable, but that would require that
you descend into mime structures (at least into the first level, the entire
contained message/rfc822 can be encoded in a single method), parse mime
headers, and re-encode every section that is 8 bit into 7 bit. Then the
issue of having to bounce messages in accordance with 1652 would never be a
postfix problem, because this external program would simply re-encode
everything.
Back when I used to use sendmail, it would re-encode mail from 8 bit to
quoted printable and back to 8 bit, but only when the mail consisted of a
single section where the encoding flags and the content type and all that
was in the main section headers, and I don't think that sendmail would take
a multipart/mixed where one of the internal sections was a plain text 8 bit
section and reach into the body and re-code it. I think that all Sendmail
ever did was re-encode single section e-mail without separators. Does
anyone know anything different? If they do, I'd appreciate knowing
this. If not, and sendmail really does not reach down into multipart/mixed
and do re-coding, then you can report to the people who have asked for this
function that no mailer, including sendmail, really does a complete job of
what they asked for.
And, of course, you could write an smtp mailer that did this and was
independent of postfix, which either hooked in with pipe or lmtp.
--
War is an ugly thing, but it is not the ugliest of things. The decayed and
degraded state of moral and patriotic feeling which thinks that nothing is
worth war is much worse. A man who has nothing for which he is willing to
fight, nothing he cares about more than his own personal safety, is a
miserable creature who has no chance of being free, unless made so by the
exertions of better men than himself. -- John Stuart Mill
Nick Simicich - n...@scifi.squawk.com
Victor Duchovni
If the client is generating MIME it should be able to
generate a quoted printable encoding of any 8bit text entities.
Which MIME aware clients are not easily configurable to do
quoted-printable encapsulation???
If the client is not generating MIME, but just a single
text message with 8 bit content, then quoted-printable encoding
is much easier to add, because the body can be converted without
any MIME parsing, but this requires a two-pass algorithm as the
MIME headers that indicate quoted-printable encoding need to be
added after the first 8bit character is found.
To reiterate what Wietse said, most receiving systems do
8bit bodies even when the capability is not advertised, and the
number of systems that don't is shrinking. It can be argued that
converting to quoted-printable at the gateway does more harm than
good. The customer who is asking for quoted-printable conversion
at the gateway should be encourated to reevaluate their assumptions.
--
Viktor.
Matthias Andree
> I could see one easy fix - there is the whole interface for virus
> scanning - a stage could be added to that whole procedure which simply
> re-codes every 8 bit message as 7 bit quoted printable, but that would
> require that you descend into mime structures (at least into the first
> level, the entire contained message/rfc822 can be encoded in a single
> method), parse mime headers, and re-encode every section that is 8 bit
> into 7 bit. Then the issue of having to bounce messages in accordance
> with 1652 would never be a postfix problem, because this external
> program would simply re-encode everything.
True, and it's probably the most useful plan, yet, it'd be more useful
if the filter knew if it's needed after all, you don't want to re-encode
if the target is 8bit clean.
For parsing, well, there is MIME::Parser for Perl5. I'd nowadays prefer
Python, but I've not yet mastered it, so I'd go for Perl for now, but
maybe there is a piece of software out there that is good quality and
re-encodes stuff
> And, of course, you could write an smtp mailer that did this and was
> independent of postfix, which either hooked in with pipe or lmtp.
True, but then, the content filtering interface is already there.
--
Matthias Andree
"Those who give up essential liberties for temporary safety deserve
neither liberty nor safety." - Benjamin Franklin
Matthias Andree
> If the client is generating MIME it should be able to
> generate a quoted printable encoding of any 8bit text entities.
> Which MIME aware clients are not easily configurable to do
> quoted-printable encapsulation???
Well, the point is, you do not want to bounce 8bit mail blindly, because
the recipient's MX might actually handle it, so you'd usually bounce
selectively to not tamper without need and to not waste your CPU
ressources. You can't do that (bounce selectively) with Postfix right
now. The admin of the MTA does not want to risk 8bit characters eaten by
just-send-8.
Postfix does not currently offer information about the next hop's 8bit
awareness, so you cannot selectively bounce, and if you run a central
Postfix as a relay for a set of somewhat autonomous departments, you
will encounter misconfigured MUAs and malformed MIME mail, that's for
sure.
If you look around, Qmail and Postfix advertise 8BITMIME, Exim does not
by default (you can configure that, though) advertise this to have the
other site convert their mail, but it's 8bit-clean still.
> If the client is not generating MIME, but just a single
> text message with 8 bit content, then quoted-printable encoding
> is much easier to add, because the body can be converted without
> any MIME parsing, but this requires a two-pass algorithm as the
> MIME headers that indicate quoted-printable encoding need to be
> added after the first 8bit character is found.
Well, that's the minor part of the problem, a mail header usually fits
into memory, if it doesn't, like in "exceeds a preconfigured limit, say
64 kB", just bounce it. If someone is to gate huge PGP keys from
ZConnect, well, tough luck, he can resend the key in the mail body.
--
Matthias Andree
"Those who give up essential liberties for temporary safety deserve
neither liberty nor safety." - Benjamin Franklin
Nick Simicich
>Nick Simicich <n...@scifi.squawk.com> writes:
>
> > I could see one easy fix - there is the whole interface for virus
> > scanning - a stage could be added to that whole procedure which simply
> > re-codes every 8 bit message as 7 bit quoted printable, but that would
> > require that you descend into mime structures (at least into the first
> > level, the entire contained message/rfc822 can be encoded in a single
> > method), parse mime headers, and re-encode every section that is 8 bit
> > into 7 bit. Then the issue of having to bounce messages in accordance
> > with 1652 would never be a postfix problem, because this external
> > program would simply re-encode everything.
>
>True, and it's probably the most useful plan, yet, it'd be more useful
>if the filter knew if it's needed after all, you don't want to re-encode
>if the target is 8bit clean.
>
>For parsing, well, there is MIME::Parser for Perl5. I'd nowadays prefer
>Python, but I've not yet mastered it, so I'd go for Perl for now, but
>maybe there is a piece of software out there that is good quality and
>re-encodes stuff
Personally, and this is because I already have working code, I'd just
start with demime and use the header parsing and mime parsing pieces of
that, but I'd change it to do the encoding. The reason I'd do that is
because, contrary to what is on some web page somewhere, I don't use
mime-tools for that, I only render html with CPAN packages, I've had bad
luck exporting programs that are dependent on CPAN packages - and
reinventing the wheel makes you free of such dependencies, besides, this is
a small wheel.
Writing a short loop that re-encoded a section to quoted printable would be
reasonably simple.
If you want to be completely sendmail compatible, you also have to
re-encode the quoted-printable and base64 to 8bit if sending to an 8 bit
clean destination. But there is no RFC that requires that.
> > And, of course, you could write an smtp mailer that did this and was
> > independent of postfix, which either hooked in with pipe or lmtp.
>
>True, but then, the content filtering interface is already there.
Right, but that interface can't *not* do 7bit if 7bit is unneeded.
--
War is an ugly thing, but it is not the ugliest of things. The decayed and
degraded state of moral and patriotic feeling which thinks that nothing is
worth war is much worse. A man who has nothing for which he is willing to
fight, nothing he cares about more than his own personal safety, is a
miserable creature who has no chance of being free, unless made so by the
exertions of better men than himself. -- John Stuart Mill
Nick Simicich - n...@scifi.squawk.com
-
Nick Simicich
>"Victor Duchovni" <Victor....@morganstanley.com> writes:
>
> > If the client is not generating MIME, but just a single
> > text message with 8 bit content, then quoted-printable encoding
> > is much easier to add, because the body can be converted without
> > any MIME parsing, but this requires a two-pass algorithm as the
> > MIME headers that indicate quoted-printable encoding need to be
> > added after the first 8bit character is found.
>
>Well, that's the minor part of the problem, a mail header usually fits
>into memory, if it doesn't, like in "exceeds a preconfigured limit, say
>64 kB", just bounce it. If someone is to gate huge PGP keys from
>ZConnect, well, tough luck, he can resend the key in the mail body.
No, you have to change the headers and the headers need to be transmitted
before the body. So before you finish transmitting the headers, you have
to know what the encoding is and whether you are going to change it because
even if it is a single section mail, you have to change or set
content-transfer-encoding. So, at some point when the mail had previously
been processed, perhaps upon ingress, the mail would have to be marked to
indicate whether it had any 8 bit characters in it somewhere in Postfix's
internal format. the mailers, so far as I can tell, have access to the
postfix internal format, so it could be stashed somewhere in there. The
state of the mail, 7bit or 8bit, should already be established, as should
the advertised ability of the connection to deal with 8bit - when the mail
is 8 bit, and a connection is advertised as 8bitmime, it can put the right
flag in the mail from<> tag, else it can recode the mail if it is 8bit and
needs to be re-encoded.
I don't think that there is a need to hold the whole header in memory, it
can be dealt with line by line. You are either going to re-encode if the
content is flagged as 8bit and the connection can't deal with it, or you
are not. If you are going to re-encode, there are some changes you are
gong to make to the header, and those can be changed line by line.
--
War is an ugly thing, but it is not the ugliest of things. The decayed and
degraded state of moral and patriotic feeling which thinks that nothing is
worth war is much worse. A man who has nothing for which he is willing to
fight, nothing he cares about more than his own personal safety, is a
miserable creature who has no chance of being free, unless made so by the
exertions of better men than himself. -- John Stuart Mill
Nick Simicich - n...@scifi.squawk.com
-