A strange thing with long SMTP message lines
Malcolm Dunnett
I’ve run into an interesting thing with PMDF 6.4 receiving SMTP messages with long text lines.
The "standard" maximum text line length in SMTP is 1000 bytes, 998 when one accounts for the <CR><LF> (as per section 4.5.3.1.6 of RFC 5321). Servers are free to accept longer lines.
My testing indicates that PMDF 6.4 will accept SMTP text lines up to 1023 characters (not counting the <CR><LF>).
If the text line is > 1024 bytes then PMDF will truncate it and create a "PMDF-SMTP-Warning" header
However if the text line is exactly 1024 characters then PMDF will truncate it to 1023, but not create a "PMDF-SMTP-Warning" header. This would appear to be a bug. It would seem that PMDF should create the warning header for text lines of 1024 characters so that there would be a record that truncation to 1023 characters had occurred.
Of course any client (in this case Apple Mail 2.753) that generates lines > 1000 characters is broken, but then living with broken clients is a cross we all have to bear. It might be better if PMDF rejected messages with excessively long lines by returning a 500 error ( as per section 4.5.3.1.9 of RFC 5351 ) or at least had an option to do so. This may be preferable to delivering a message that may be corrupted due to one or more lines being truncated.
Thoughts?
Ken Connelly
arbitrarily long lines. I'm still running 6.2-1 here (long, sad story)
and I believe that with that version, lines greater than 1000 bytes are
truncated at 1000 bytes and a header stating that long lines have been
truncated is added.
- ken
--
- Ken
=================================================================
Ken Connelly Associate Director, Security and Systems
ITS Network Services University of Northern Iowa
email: Ken.Co...@uni.edu p: (319) 273-5850 f: (319) 273-7373
Malcolm Dunnett
>PMDF seems to be one of a very few mailers that doesn't gleefully
accept
>arbitrarily long lines. I'm still running 6.2-1 here (long, sad story)
>and I believe that with that version, lines greater than 1000 bytes are
>truncated at 1000 bytes and a header stating that long lines have been
>truncated is added.
I agree that PMDF accepting arbitrarily long lines (or at least a much
longer size such as 32767 bytes) would be the ideal solution, but I
don't know if that might entail significant re-engineering. The two
suggestions I made seem like rather simple fixes that would at least be
better than the current situation.
Ken Connelly
>
> I agree that PMDF accepting arbitrarily long lines (or at least a much
> longer size such as 32767 bytes) would be the ideal solution, but I
> don't know if that might entail significant re-engineering. The two
> suggestions I made seem like rather simple fixes that would at least be
> better than the current situation.
>
>
following the standards by truncating long lines. What *would* be a
good thing is fixing the plethora of broken clients that emit mail
messages which violate the standards.
Malcolm Dunnett
Ken Connelly wrote:
>I disagree. While more consistency would be an improvement, PMDF is
>following the standards by truncating long lines.
PMDF is following the "minimum" standard by truncating long lines,
however
the standard also says:
"To the maximum extent possible, implementation techniques which impose
no limits on the length of these objects should be used."
So it's not a violation of the standard to accept longer lines, it's
only a violation to reject lines shorter than 1000 characters. In fact
PMDF "should" (in the meaning of the RFC) accept longer lines ( I
would argue that the "maximum extent possible" would be to accept
lines up to 32767 characters, this being the limit on RMS records
and the maximum one can describe in a VMS string descriptor - however
I will grant there may be other technical issues which would impose
a shorter limit)
> What *would* be a
>good thing is fixing the plethora of broken clients that emit mail
>messages which violate the standards.
Yes, it would - but I know of no way to influence that, whereas
discussions in this group *may* influence Process Software to
modify PMDF to accept longer lines. As an admin I just want the
mail to go through with minimum hassle, so if PMDF can be modified
to be more tolerant of these broken clients that is a good thing.
If not I still don't like the current model of truncating longer
lines, I would prefer PMDF to reject the message, to make it
clear to the broken client that their message did not get through
intact.
ned+in...@mauve.mrochek.com
> Ken Connelly wrote:
> >I disagree. While more consistency would be an improvement, PMDF is
> >following the standards by truncating long lines.
> PMDF is following the "minimum" standard by truncating long lines, however
> the standard also says:
> "To the maximum extent possible, implementation techniques which impose
> no limits on the length of these objects should be used."
The standard you are quoting here is RFC 821 (section 4.5.3 to be specific).
Problem is, it, along with RFC 822, are now obsolete, having been replaced by
RFCs 5321 and 5322. (Yes, I'm aware that these are current at draft and 821/822
are full standards, but progression to full standard seems inevitable at this
point so any attempt to cling to the older specifications for guidance has a
lifespan best measured in months at this point.) These new specifications are
much more strict about this stuff. RFC 5322 in particular has this to say
(section 2.1.1):
There are two limits that this specification places on the number of
characters in a line. Each line of characters MUST be no more than
998 characters, and SHOULD be no more than 78 characters, excluding
the CRLF.
The 998 character limit is due to limitations in many implementations
that send, receive, or store IMF messages which simply cannot handle
more than 998 characters on a line. Receiving implementations would
do well to handle an arbitrarily large number of characters in a line
for robustness sake. However, there are so many implementations that
(in compliance with the transport requirements of [RFC5321]) do not
accept messages containing more than 1000 characters including the CR
and LF per line, it is important for implementations not to create
such messages.
This really could not be more clear, especially the part about "not creating
such messages".
> So it's not a violation of the standard to accept longer lines, it's
> only a violation to reject lines shorter than 1000 characters.
That's quite simply false. Messages containing lines longer that 1000
characters are incompliant with current standards and may be handled in any
manner an implementation chooses, including forcible line wrapping or even
summary rejection.
> In fact PMDF "should" (in the meaning of the RFC) accept longer lines ( I
> would argue that the "maximum extent possible" would be to accept
> lines up to 32767 characters, this being the limit on RMS records
> and the maximum one can describe in a VMS string descriptor - however
> I will grant there may be other technical issues which would impose
> a shorter limit)
The argument that PMDF "should" do this doesn't come close to washing given
what the standards currently say (and have said for quite a while - this
language or something close to it also appeared in RFCs 2821 and 2822).
And while tweaking PMDF so that all lines could be, say, 2048 characters or
some other fixed value, changing things so that arbitrarily long lines could be
accomodated would be EXTREMELY difficult, and if done would probably compromise
performance quite significantly. And any fixed limit won't help with clients
that insist on spewing out megabytes of HTML on a single line.
What would help, and is the appproach we adopted in iMS some time back, is to
forcibly wrap such lines. We also added an option to reject messages with
overly long lines because believe it or not, quite a few people demanded it.
So now sites can choose between truncation, wrapping, or rejection.
Oh, and we also set the wrap/truncate/reject point to 998 instead of 1023/1024
as it is in PMDF. And the reason we did this is - wait for it - sending out
messages with lines even slightly past the limit was causing interop problems
with quite a few other mail systems. So much for the notion that everybody out
there accepts arbitrarily long lines.
Wrapping, if done, seems to work best if the heuristics assume HTML-ish input,
since a significant number of these broken messages are text/html or something
similar.
I can provide the code to Process to do line wraps if they want it, however,
the SMTP server code has diverged so from the PMDF original I cannot provide a
complete retrofit in this case.
> > What *would* be a
> > good thing is fixing the plethora of broken clients that emit mail
> > messages which violate the standards.
> Yes, it would - but I know of no way to influence that,
Well, summary rejection of such messages would definitely "send" a "message",
especially if lots of sites did it. But I doubt if enought sites have the
stomach for that.
> whereas
> discussions in this group *may* influence Process Software to
> modify PMDF to accept longer lines. As an admin I just want the
> mail to go through with minimum hassle, so if PMDF can be modified
> to be more tolerant of these broken clients that is a good thing.
I'm sorry, but I have to say I see this attitude as quite dangerous and in fact
at least partly responsible for much of the mess that email is now in.
Nondelivery notification handling is a particularly good example of this.
Blowback spam is a fairly significant problem for a lot of sites, so much so
that legitimate notifications can be swamped with garbage. Applying the
"minimum hassle" metric in this case leads to an obvious conclusion: Throw them
all away, or better still, simply reject all messages with a empty MAIL FROM
right there in the SMTP dialogue. (I know of sites that have done this so this
is not an academic example.) Trouble is, every once in a while a nondelivery
notification does convey critical information, and you've just screwed the user
who was depending on that. All in the interests of "minimum hassle".
Now, AFAIK the situation with long lines in messages hasn't resulted in this
sort of damage. But there are a variety of ways in which it could easily reach
that point. (The obvious one is a massive influx of spam that looks completely
legitimate other than containing very long lines. Think what the likely
consequence of that scenario will be on the "just send it along unchanged"
crowd.) At least if the message has been truncated or rejected someone will be
made aware of what happened, and can figure out why. But wrapping long lines is
really the best option IMO, even though it's arguably the biggest hassle
(because if the wrapping breaks the message it's hard to avoid owning the
breakage).
> If not I still don't like the current model of truncating longer
> lines, I would prefer PMDF to reject the message, to make it
> clear to the broken client that their message did not get through
> intact.
Adding an option to reject such messages would be a good idea IMO, but
adding one to forcibly wrap would be even better.
Ned
P.S. Yet another option is forcible encoding of messages. The only problem with
this is it's an implementation nightmare.
Malcolm Dunnett
Thanks for your thoughts. I respect that you know a whole
lot more about the issue than I do, but I do have to
disagree with some of what you said.
>The standard you are quoting here is RFC 821 (section 4.5.3 to be
specific).
>Problem is, it, along with RFC 822, are now obsolete, having been
replaced by
>RFCs 5321 and 5322.
Actually I was quoting section 4.5.3.1 of RFC 5321. While the example
they give is an address and not a text line it seems to me from the
context of the section numbering that they are talking about all
the objects within the section, including "4.5.3.1.6 Text Line"
> more than 998 characters on a line. Receiving implementations would
> do well to handle an arbitrarily large number of characters in a
line
> for robustness sake.
That's what I'm asking for, the receiving implementation to handle
an arbitrarily large number of characters (for robustness sake.).
Realistically I'm not asking for arbitrarily large, just "as large
as is reasonable to implement given the existing code base"
> However, there are so many implementations that
> (in compliance with the transport requirements of [RFC5321]) do not
> accept messages containing more than 1000 characters including the
CR
> and LF per line, it is important for implementations not to create
> such messages.
>This really could not be more clear, especially the part about "not
creating
>such messages".
Precisely. I never suggested anyone should create such messages (the
actual
creation would be a UA issue, not an MTA issue). However I was
suggesting
that if presented with one the MTA should (or "would do well to" if
you prefer) accommodate it (as per the quote above).
>> So it's not a violation of the standard to accept longer lines, it's
>> only a violation to reject lines shorter than 1000 characters.
>That's quite simply false. Messages containing lines longer that 1000
>characters are incompliant with current standards and may be handled in
any
>manner an implementation chooses, including forcible line wrapping or
even
>summary rejection.
That doesn't make what I said false, it merely indicates (as I said)
that
nothing in the standard *prevents* PMDF from receiving longer lines.
I will grant you that it's not *required* to.
> In fact PMDF "should" (in the meaning of the RFC) accept longer lines
( I
> would argue that the "maximum extent possible" would be to accept
> lines up to 32767 characters, this being the limit on RMS records
> and the maximum one can describe in a VMS string descriptor - however
> I will grant there may be other technical issues which would impose
> a shorter limit)
>The argument that PMDF "should" do this doesn't come close to washing
given
>what the standards currently say (and have said for quite a while -
this
>language or something close to it also appeared in RFCs 2821 and 2822).
As I was quoting from the current standard I can't see that your
argument
is valid.
>And while tweaking PMDF so that all lines could be, say, 2048
characters or
>some other fixed value, changing things so that arbitrarily long lines
could be
>accomodated would be EXTREMELY difficult, and if done would probably
compromise
>performance quite significantly. And any fixed limit won't help with
clients
>that insist on spewing out megabytes of HTML on a single line.
I will certainly bow to your superior knowledge of what is feasible
in PMDF. AS I said earlier, there may well be implementation reasons
that preclude accepting longer lines. My 32767 example was merely
noting an example of an upper bound that the current implementation
may impose.
>What would help, and is the appproach we adopted in iMS some time back,
is to
>forcibly wrap such lines.
That sounds like an elegant solution (assuming it wraps them
intelligently
so as to not corrupt the syntax of the message)
>We also added an option to reject messages with
>overly long lines because believe it or not, quite a few people
demanded it.
Why wouldn't I believe it, if you read my earlier message you'll see
I asked for that as a second choice. If the non-conforming message
can't be delivered intact by PMDF I think it would be better to reject
it (along with a suitable message to the effect that the format
is non-conforming).
As an aside, if I create a message with the PMDF API and submit
message body lines of more than 1024 characters they are
truncated, but no "lines truncated header" is created, nor is
an error status returned by PMDF_write_text. Another example
of where PMDF can silently corrupt the message.
>Wrapping, if done, seems to work best if the heuristics assume HTML-ish
input,
>since a significant number of these broken messages are text/html or
something
>similar.
The cases I ran into were HTML code, so I would concur with that
statement.
>I can provide the code to Process to do line wraps if they want it,
however,
>the SMTP server code has diverged so from the PMDF original I cannot
provide a
>complete retrofit in this case.
That of course is up to Process, however I would certainly applaud
them adopting that change.
>Well, summary rejection of such messages would definitely "send" a
"message",
>especially if lots of sites did it. But I doubt if enought sites have
the
>stomach for that.
Probably true - and something I can understand. End users tend
to assume that if we can't accept a message from their Macintosh,
or can't send one to Hotmail, that it must be our fault and
demand that we "fix it". However I would still prefer to
justify to a client why I rejected his message rather than why I
delivered a version that was "corrupted" due to line truncation.
>> modify PMDF to accept longer lines. As an admin I just want the
>> mail to go through with minimum hassle, so if PMDF can be modified
>> to be more tolerant of these broken clients that is a good thing.
>
>I'm sorry, but I have to say I see this attitude as quite dangerous and
in fact
>at least partly responsible for much of the mess that email is now in.
Well that may be true - if nobody accepted lines longer than 1000
characters then the broken UAs would probably get fixed. But in
the real world that isn't going to happen, is it?
>Now, AFAIK the situation with long lines in messages hasn't resulted in
this
>sort of damage. But there are a variety of ways in which it could
easily reach
>that point. (The obvious one is a massive influx of spam that looks
completely
>legitimate other than containing very long lines. Think what the likely
>consequence of that scenario will be on the "just send it along
unchanged"
>crowd.)
Why is this worse than any other kind of spam? Are you thinking about
buffer overflow attacks in the MTA or UA?
> At least if the message has been truncated or rejected someone will be
>made aware of what happened, and can figure out why. But wrapping long
lines is
>really the best option IMO, even though it's arguably the biggest
hassle
>(because if the wrapping breaks the message it's hard to avoid owning
the
>breakage).
Well wrapping is certainly less likely to break the message than
truncation is, so overall it's far more likely to be an
acceptable solution.
dav...@alpha1.mdx.ac.uk
>Ned Freed wrote:
>
>Thanks for your thoughts. I respect that you know a whole
>lot more about the issue than I do, but I do have to
>disagree with some of what you said.
>
>specific).
>>Problem is, it, along with RFC 822, are now obsolete, having been
>replaced by
>>RFCs 5321 and 5322.
>
>they give is an address and not a text line it seems to me from the
>context of the section numbering that they are talking about all
>the objects within the section, including "4.5.3.1.6 Text Line"
>
>> do well to handle an arbitrarily large number of characters in a
>line
>> for robustness sake.
>
>an arbitrarily large number of characters (for robustness sake.).
>Realistically I'm not asking for arbitrarily large, just "as large
>as is reasonable to implement given the existing code base"
>
increase the buffer size for the PMDF SMTP server so that it could receive
messages with larger line lengths but then to apply a default linelength
channel keyword of 1024 which would result in messages with long
linelengths being MIME encoded - thereby removing any need to alter other
parts of PMDF.
David Webb
Security team leader
CCSS
Middlesex University
>> However, there are so many implementations that
>> (in compliance with the transport requirements of [RFC5321]) do not
>> accept messages containing more than 1000 characters including the
>CR
>> and LF per line, it is important for implementations not to create
>> such messages.
>
>>This really could not be more clear, especially the part about "not
>creating
>>such messages".
>
>Precisely. I never suggested anyone should create such messages (the
>actual
>creation would be a UA issue, not an MTA issue). However I was
>suggesting
>that if presented with one the MTA should (or "would do well to" if
>you prefer) accommodate it (as per the quote above).
>
>>> So it's not a violation of the standard to accept longer lines, it's
>>> only a violation to reject lines shorter than 1000 characters.
>
>>That's quite simply false. Messages containing lines longer that 1000
>>characters are incompliant with current standards and may be handled in
>any
>>manner an implementation chooses, including forcible line wrapping or
>even
>>summary rejection.
>
>That doesn't make what I said false, it merely indicates (as I said)
>that
>nothing in the standard *prevents* PMDF from receiving longer lines.
>I will grant you that it's not *required* to.
>
>> In fact PMDF "should" (in the meaning of the RFC) accept longer lines
>( I
>> would argue that the "maximum extent possible" would be to accept
>> lines up to 32767 characters, this being the limit on RMS records
>> and the maximum one can describe in a VMS string descriptor - however
>> I will grant there may be other technical issues which would impose
>> a shorter limit)
>
>>The argument that PMDF "should" do this doesn't come close to washing
>given
>>what the standards currently say (and have said for quite a while -
>this
>>language or something close to it also appeared in RFCs 2821 and 2822).
>
>As I was quoting from the current standard I can't see that your
>argument
>is valid.
>
>>And while tweaking PMDF so that all lines could be, say, 2048
>characters or
>>some other fixed value, changing things so that arbitrarily long lines
>could be
>>accomodated would be EXTREMELY difficult, and if done would probably
>compromise
>>performance quite significantly. And any fixed limit won't help with
>clients
>>that insist on spewing out megabytes of HTML on a single line.
>
> I will certainly bow to your superior knowledge of what is feasible
>in PMDF. AS I said earlier, there may well be implementation reasons
>that preclude accepting longer lines. My 32767 example was merely
>noting an example of an upper bound that the current implementation
>may impose.=20
>
>>What would help, and is the appproach we adopted in iMS some time back,
>is to
>>forcibly wrap such lines.=20
>
>That sounds like an elegant solution (assuming it wraps them
>intelligently
>so as to not corrupt the syntax of the message)
>
>>We also added an option to reject messages with
>>overly long lines because believe it or not, quite a few people
>demanded it.
>
>Why wouldn't I believe it, if you read my earlier message you'll see
>I asked for that as a second choice. If the non-conforming message
>can't be delivered intact by PMDF I think it would be better to reject
>it (along with a suitable message to the effect that the format
>is non-conforming).
>
>As an aside, if I create a message with the PMDF API and submit
>message body lines of more than 1024 characters they are
>truncated, but no "lines truncated header" is created, nor is
>an error status returned by PMDF_write_text. Another example
>of where PMDF can silently corrupt the message.
>
>>Wrapping, if done, seems to work best if the heuristics assume HTML-ish
>input,
>>since a significant number of these broken messages are text/html or
>something
>>similar.
>
>The cases I ran into were HTML code, so I would concur with that
>statement.
>
>>I can provide the code to Process to do line wraps if they want it,
>however,
>>the SMTP server code has diverged so from the PMDF original I cannot
>provide a
>>complete retrofit in this case.
>
>That of course is up to Process, however I would certainly applaud
>them adopting that change.
>
>>Well, summary rejection of such messages would definitely "send" a
>"message",
>>especially if lots of sites did it. But I doubt if enought sites have
>the
>>stomach for that.
>
>Probably true - and something I can understand. End users tend
>to assume that if we can't accept a message from their Macintosh,
>or can't send one to Hotmail, that it must be our fault and
>demand that we "fix it". However I would still prefer to
>justify to a client why I rejected his message rather than why I
>delivered a version that was "corrupted" due to line truncation.
>
>>> modify PMDF to accept longer lines. As an admin I just want the
>>> mail to go through with minimum hassle, so if PMDF can be modified
>>> to be more tolerant of these broken clients that is a good thing.
>>
>>I'm sorry, but I have to say I see this attitude as quite dangerous and
>in fact
>>at least partly responsible for much of the mess that email is now in.
>
>Well that may be true - if nobody accepted lines longer than 1000
>characters then the broken UAs would probably get fixed. But in
>the real world that isn't going to happen, is it?
>
>>Now, AFAIK the situation with long lines in messages hasn't resulted in
>this
>>sort of damage. But there are a variety of ways in which it could
>easily reach
>>that point. (The obvious one is a massive influx of spam that looks
>completely
>>legitimate other than containing very long lines. Think what the likely
>>consequence of that scenario will be on the "just send it along
>unchanged"
>>crowd.)
>
>Why is this worse than any other kind of spam? Are you thinking about
>buffer overflow attacks in the MTA or UA?=20
>
>> At least if the message has been truncated or rejected someone will be
>>made aware of what happened, and can figure out why. But wrapping long
>lines is
>>really the best option IMO, even though it's arguably the biggest
>hassle
>>(because if the wrapping breaks the message it's hard to avoid owning
>the
>>breakage).
>
Tom Wade
their products, and ensure they adhere to the common standard.
We don't live in an ideal world.
Vendors whose products occupy a dominant position have a vested interest
in ensuring that they do diverge from the standard, and provide extra
functionality. Users then perceive other products as being inferior
because what works with "most of the stuff out" there doesn't work with
such-and-such product, and that provides migratory pressure on the user
base. This is why Outlook has a 'richtext' encoding option (TNEF) that
makes its attachments incompatible with everyone elses - it applies a
steady pressure to get people to standardize on Outlook (it even seems
to be the default setting). This can only be the case when the errant
mailer already has a huge share of the market.
Microsoft didn't invent this. IBM were probably the first to adopt the
"we're-big-enough-to-be-the-standard", and so did Unix in the early days
of the Internet (which is why Multinet to this day as an FTP server that
can pretend to be a Unix system to satisfy many of the clients out there
that think everything is either Windows or Unix).
The hard commercial fact is if you are trying to sell a mailer that must
compete with Exchange, then if something works fine with Exchange and
fails with your mailer, it is no use pointing out that their's is out of
step, and your's is the correct standard. The paying punters don't see
it that way, and will vote with their wallets.
Because of this, mailers such as PMDF have a lot of features
('msexchange' keyword, MIME header correction capabilities etc) designed
to cope with the way the dominant player strays from the standard when
it suits them. The issue is whether the effort is worth the gain, and
this is primarily a commercial decision, not a standards decision, and
certainly not one based on notions of 'fair' or 'correct'.
I suspect it was for commercial reasons that Sun decided to make iMS
able to handle lines that were longer than the standard. There is also
the tried and trusted "be conversative in what you send, and liberal in
what you accept" principle, which means you adhere to the letter of the
standard with what you emit, and try to allow as much as possible for
the many broken mailers when accepting traffic in.
I accept Ned's assertion that trying to increase this limit in PMDF will
be a big upheaval and could seriously impact performance, as nobody
knows more about the innards of PMDF. I would suggest to Process that
Ned's suggestion of automatically wrapping long lines along white space
boundaries should be considered as an urgent enhancement. Yes, this is
not a 100% correct way of handling a message, but is a much superior
option to simply chopping the line, and if it does mess up a bodypart,
then (A) it was already broken to start with and (B) it wouldn't fare
much better going to an existing competing mailer.
I find it difficult to explain to paying customers that something that
works fine with competitor products (based on Exchange or say iMS) and
not on ours (based on PMDF) is not our fault - they simply see it was
one less reason to go with PMDF, which is a great pity, as the last
thing we need is to facilitate migrations away from that great product.
---------------------------------------------------------
Tom Wade | EMail: tee dot wade at eurokom dot ie
EuroKom | Tel: +353 (1) 296-9696
A2, Nutgrove Office Park | Fax: +353 (1) 296-9697
Rathfarnham | Disclaimer: This is not a disclaimer
Dublin 14 | Tip: "Friends don't let friends do Unix !"
Ireland
Francesco Gennai
up to date the PMDF as much as possible.
(For example, I think that one of the next big effort for Internet application
developers will be IPv6 support).
Thanks, Tom.
Francesco
ned+in...@mauve.mrochek.com
> >Problem is, it, along with RFC 822, are now obsolete, having been replaced by
> >RFCs 5321 and 5322.
> Actually I was quoting section 4.5.3.1 of RFC 5321. While the example
> they give is an address and not a text line it seems to me from the
> context of the section numbering that they are talking about all
> the objects within the section, including "4.5.3.1.6 Text Line"
Hmm. I thought that text had vanished. It is quite unfortunate it has been
left in. (I have similar issues with some of the language surrounding CRLF
handling that are causing far more interop problems than they are solving, but
that's a story for another day.)
Regardless, the fact remains that emitting a message with lines longer than 998
violates a MUST elsewhere. A mild recommendation that implementations should
not limit things unnecessarily (note the lack of compliance language) does not
suffice to override a MUST. That means any implementation that accepts such
lines also has to figure out what it's going to do with them once it has them.
Sending them out unchanged is NOT an acceptable option.
> > more than 998 characters on a line. Receiving implementations would
> > do well to handle an arbitrarily large number of characters in a
> line
> > for robustness sake.
> That's what I'm asking for, the receiving implementation to handle
> an arbitrarily large number of characters (for robustness sake.).
> Realistically I'm not asking for arbitrarily large, just "as large
> as is reasonable to implement given the existing code base"
Actually, there is no reason not to handle arbitraily long lines if what you do
is modify them either by folding, truncating or rejecting them right away.
Sending them through unchanged is what's hard. And the problem with any fixed
buffer size is something out there is almost guaranteed to exceed it. (OK,
maybe if you set it to a couple of megabytes the number of messages that will
exceed it will be negligable, but that's the sort of fixed limit you have to
have if you expect this to actually salvage the real-world malformed messages
people send. But that big of a buffer size would be extremely inefficient, even
in this time of ever-increasing memory.)
> > However, there are so many implementations that
> > (in compliance with the transport requirements of [RFC5321]) do not
> > accept messages containing more than 1000 characters including the CR
> > and LF per line, it is important for implementations not to create
> > such messages.
> >This really could not be more clear, especially the part about "not
> creating
> >such messages".
> Precisely. I never suggested anyone should create such messages (the
> actual
> creation would be a UA issue, not an MTA issue). However I was
> suggesting
> that if presented with one the MTA should (or "would do well to" if
> you prefer) accommodate it (as per the quote above).
I disagree. "Handling" these lines without alteration, in addition to violating
that MUST, would mean either changing the fixed buffer to a completely
unreasonable value or rewriting a significant fraction of the MTA to work in a
very different way. While such a rewrite is possible, it would unavoidably lead
to a performance loss and given the extent of the changes regressions are
certain to occur.
All this just to cater to clearly broken behavior. The costs just aren't in
line with the benefits, especially since there's a viable alternative that
solves most of the problem without having to "handle" such material.
> >> So it's not a violation of the standard to accept longer lines, it's
> >> only a violation to reject lines shorter than 1000 characters.
> >That's quite simply false. Messages containing lines longer that 1000
> >characters are incompliant with current standards and may be handled in
> any
> >manner an implementation chooses, including forcible line wrapping or
> even
> >summary rejection.
> That doesn't make what I said false, it merely indicates (as I said) that
> nothing in the standard *prevents* PMDF from receiving longer lines.
Sorry, I misread your statement and thought you were saying it was
required.
> I will grant you that it's not *required* to.
On that we agree.
> > In fact PMDF "should" (in the meaning of the RFC) accept longer lines
> ( I
> > would argue that the "maximum extent possible" would be to accept
> > lines up to 32767 characters, this being the limit on RMS records
> > and the maximum one can describe in a VMS string descriptor - however
> > I will grant there may be other technical issues which would impose
> > a shorter limit)
> > The argument that PMDF "should" do this doesn't come close to washing given
> > what the standards currently say (and have said for quite a while - this
> > language or something close to it also appeared in RFCs 2821 and 2822).
> As I was quoting from the current standard I can't see that your
> argument is valid.
Again, a mild recommendations that limits not be imposed unnecessarily does not
come close to overriding MUSTs elsewhere.
> > What would help, and is the appproach we adopted in iMS some time back, s to
> > forcibly wrap such lines.
> That sounds like an elegant solution (assuming it wraps them intelligently
> so as to not corrupt the syntax of the message)
That's effectively impossible, if for no other reaons than there are plenty of
text formats that employ extremely long lines and which are totally intolerant
of ANY line breaks being inserted. (CSV is a good exmaple of such a format
that's in common use.)
The best that can be done here is to try and not put the breaks in bad spots.
But the reality is a few things are still going to break. And they'll also
break if you encode the material, for that matter - even at this late date not
everyone does MIME/uuencode/whatever decoding.
> Why wouldn't I believe it, if you read my earlier message you'll see
> I asked for that as a second choice. If the non-conforming message
> can't be delivered intact by PMDF I think it would be better to reject
> it (along with a suitable message to the effect that the format
> is non-conforming).
> As an aside, if I create a message with the PMDF API and submit
> message body lines of more than 1024 characters they are
> truncated, but no "lines truncated header" is created, nor is
> an error status returned by PMDF_write_text. Another example
> of where PMDF can silently corrupt the message.
Yep, and in this case I think the API's behavior is reasonable and should not
be changed. The PMDF API is not bound by any standard to behave in any
particular way.
> >Wrapping, if done, seems to work best if the heuristics assume HTML-ish
> input,
> >since a significant number of these broken messages are text/html or
> something
> >similar.
> The cases I ran into were HTML code, so I would concur with that statement.
It is almost always possible to insert breaks into HTML in such a way that
things don't break. Of course finding the right points is another matter. A
fairly simple algorithm seems to mostly work, though.
> ...
> >> modify PMDF to accept longer lines. As an admin I just want the
> >> mail to go through with minimum hassle, so if PMDF can be modified
> >> to be more tolerant of these broken clients that is a good thing.
> >
> >I'm sorry, but I have to say I see this attitude as quite dangerous and
> in fact
> >at least partly responsible for much of the mess that email is now in.
> Well that may be true - if nobody accepted lines longer than 1000
> characters then the broken UAs would probably get fixed. But in
> the real world that isn't going to happen, is it?
> >Now, AFAIK the situation with long lines in messages hasn't resulted in
> this
> >sort of damage. But there are a variety of ways in which it could
> easily reach
> >that point. (The obvious one is a massive influx of spam that looks
> completely
> >legitimate other than containing very long lines. Think what the likely
> >consequence of that scenario will be on the "just send it along
> unchanged"
> >crowd.)
> Why is this worse than any other kind of spam?
It's not the spam but the behavior such spam would cause among spam-blockers.
If such spam messages were to become common spam blockers will start using line
length as a criteria for deciding that messages are spam. And if that happens
you can kiss your ability to send messages with overly long lines goodbye no
matter how big the population of misbehaving MUAs happens to be. Like it or
not, the final arbiter os what's acceptable in email is now determined by the
antispam tools.
And this is not idle theorizing on my part - there are several past exanmples
of this sort of thing. The one I remeber best is when a particularly virulent
spambot spread a few years ago whose signature characteristic was that it's
SMTP client said "HELO host" instead of "HELO host.something". Before that time
very few sites enforced the restriction that HELO arguments have to be fully
qualified domain names and a huge number of sites happily used shortform named
in HELO, but after the spambot showed up things changed rapidly so that if you
broke this rule you stood a good chance of having your mail bounce, or worse,
silently lost. And yes, there was lots of complaining, but when you're being
buried in spam somehow the license of others to break the rules doesn't seem
like that big a deal.
Ned
Carl Karcher
>...
> Actually, there is no reason not to handle arbitraily long lines if what you do
> is modify them either by folding, truncating or rejecting them right away.
> Sending them through unchanged is what's hard. And the problem with any fixed
> buffer size is something out there is almost guaranteed to exceed it. (OK,
> maybe if you set it to a couple of megabytes the number of messages that will
> exceed it will be negligable, but that's the sort of fixed limit you have to
> have if you expect this to actually salvage the real-world malformed messages
> people send. But that big of a buffer size would be extremely inefficient, even
> in this time of ever-increasing memory.)
Even if you increase PMDF's limit you'll have other limits (like RMS) to
deal with as evidenced by recent bounce:
%PMDF-E-MESFILERR, Error in accessing intermediate message file
-RMS-W-RTB, 4152564 byte record too large for user's buffer
The client was:
X-Mailer: Apple Mail (2.929.2)
--
-- Carl Karcher, Waisman Computing Services, Waisman Center, UW-Madison
-- kar...@waisman.wisc.edu