[Django] #22561: EmailMessage should respect RFC2822 on max line length
Django
-------------------------------+--------------------
Reporter: notsqrt | Owner: nobody
Type: Uncategorized | Status: new
Component: Core (Mail) | Version: 1.6
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------
Follow-up of thread [[https://groups.google.com/forum/#!topic/django-
users/BdjFVVdX7QU|Email encoding (DKIM, long lines, etc..)]] on django-
users
== RFC ==
The [[https://tools.ietf.org/html/rfc2822#section-2.1.1|RFC2822]] states
that:
"Each line of characters MUST be no more than 998 characters, and SHOULD
be no more than 78 characters, excluding the CRLF."
This statement has not been modified in 2008 in the updated version :
[[https://tools.ietf.org/html/rfc5322#section-2.1.1|RFC5322]]
== History ==
For utf-8 encoded emails, Python uses:
* shortest of "quoted-printable" and "base64" for the email subject
* "base64" for the body
{{{#!python
# stdlib, identical in python2.7 and python3.3 : email/charset.py
CHARSETS = {
'utf-8': (SHORTEST, BASE64, 'utf-8'),
}
}}}
The historical reason seems to be that support for 8bit characters in
emails was not largely adopted, hence the need to encode them into ASCII.
Back in 2007, in ticket [[ticket:3472]] (changeset [changeset:5143]), it
was decided to always use "quoted-printable", because using base64 seems
to negatively affect spam scores.
{{{#!python
# Don't BASE64-encode UTF-8 messages so that we avoid unwanted attention
from
# some spam filters.
Charset.add_charset('utf-8', Charset.SHORTEST, Charset.QP, 'utf-8')
}}}
In 2011, in ticket [ticket:11212] (changeset [changeset:16178], django
1.4), it was decided to remove "quoted-printable", and let python
automatically switch between 7-bit or 8-bit encodings, based on the fact
that 8-bit emails were widely supported, and MTAs were in charge of the
downgrading to 7-bit if necessary.
{{{#!python
Charset.add_charset('utf-8', Charset.SHORTEST, None, 'utf-8')
}}}
The (unintended?) side-effect of using base64 or "quoted-printable" was in
fact a guarantee to have short lines in emails (for instance, rfc for
quoted-printable [[http://tools.ietf.org/html/rfc2045#page-20|rfc2045]]
states that max-length is 76 characters).
=== Summary of invoqued reasons for these choices ===
* base64 is too big (bandwidth)
* base64 is not supported by all clients
* base64 has a negative effect on spam scores (cf
[[http://wiki.apache.org/spamassassin/Rules/MIME_BASE64_TEXT|SpamAssassin's
rule]] on **unnecessarily** using base64 encoding to disguise text, but
this rule also states that "This does not apply to text in the UTF-8 or
big5 character sets.")
* quoted-printable is no longer necessary, since MTAs and email clients
have adopted 8bit support
== Current state ==
=== Django ===
There was an additional ticket [[ticket:12422]], but not relevant to this
ticket.
The current code in django/core/mail/message.py looks like:
{{{#!python
# Don't BASE64-encode UTF-8 messages so that we avoid unwanted attention
from
# some spam filters.
utf8_charset = Charset.Charset('utf-8')
utf8_charset.body_encoding = None # Python defaults to BASE64
}}}
=== Clients ===
Email clients like Gmail seem to wrap lines at 80 characters for
text/plain, and switch to "Content-Transfer-Encoding: quoted-printable"
for text/html and text/plain if there are non-ascii characters.
== Importance ==
Mail Transfer Agent like Postfix often split lines that do not respect the
RFC by inserting "\r\n " at the 998-th position of the line.
DKIM signatures of emails are based on the unmodified body, but the
signature validation by receivers is based on the modified body, resulting
in a check failure.
Apart from my own django projects, I have seen long lines in html emails
sent by Sentry, for instance.
== Choices ==
For reference, [[http://search.cpan.org/~rjbs/MIME-
Lite-3.030/lib/MIME/Lite.pm#Construction|Perl library MIME-Lite]]
recommends:
{{{
Use encoding: | If your message contains:
------------------------------------------------------------
7bit | Only 7-bit text, all lines <1000 characters
8bit | 8-bit text, all lines <1000 characters
quoted-printable | 8-bit text or long lines (more reliable than
"8bit")
base64 | Largely non-textual data: a GIF, a tar file, etc.
}}}
One way or another, we have to guarantee that email lines are <1000
characters.
base64 and quoted-printable do that for us.
No using them means that we have to find a reliable way to split long
lines into shorter ones, but the risk is to break html code in the case of
text/html emails.
I am not aware of other encodings that can be used for this, nor of
reliable ways to split long lines.
On django-users, Russ Magee warned about possible downstream consequences.
== Other references ==
[[http://www.w3.org/Protocols/rfc1341/5_Content-Transfer-Encoding.html]]
[[http://trac.edgewall.org/ticket/1754|relevant discussion on trac's
trac]]
[[http://wiki.apache.org/spamassassin/Rules/MIME_QP_LONG_LINE|SpamAssassin's
rule]] on quoted-printable messages not respecting the 76-max line length
rule.
--
Ticket URL: <https://code.djangoproject.com/ticket/22561>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: notsqrt | Owner: nobody
Type: Uncategorized | Status: new
Component: Core (Mail) | Version: 1.6
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by petr.hroudny@…):
* cc: petr.hroudny@… (added)
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:1>
Django
Reporter: notsqrt | Owner: nobody
Type: Uncategorized | Status: new
Component: Core (Mail) | Version: 1.6
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by phr):
Quoted-printable should only be used to downconvert emails to 7bit-only,
not to workaround their RFC incompliance regarding line lengths.
Please note that QP works decently just for languages based on ASCII with
only a few accentuated characters, but performs miserably for all others.
Thus reintroducing any form of 7bit downconversion is not the proper
solution to this problem.
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:2>
Django
Reporter: notsqrt | Owner: nobody
Type: Bug | Status: new
Component: Core (Mail) | Version: 1.6
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timo):
* type: Uncategorized => Bug
* stage: Unreviewed => Accepted
Comment:
Russ seemed to accept the problem on the mailing list.
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:3>
Django
Reporter: notsqrt | Owner: nobody
Type: Bug | Status: new
Component: Core (Mail) | Version: 1.6
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by ralphje):
Apart from breaking DKIM, this behaviour also affects appearance of emails
containing long lines (spaces appear to be added, at least in Gmail) and
breaks inline HTML or CSS.
Note that long lines are commonplace when CSS rules are automatically
being inlined.
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:4>
Django
Reporter: notsqrt | Owner: nobody
Type: Bug | Status: new
Component: Core (Mail) | Version: 1.6
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
> No using them means that we have to find a reliable way to split long
lines into shorter ones, but the risk is to break html code in the case of
text/html emails.
I don't get your point here, how would we break html code by splitting
lines? Content inside `<pre>`?
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:5>
Django
Reporter: notsqrt | Owner: levkowetz
Type: Bug | Status: assigned
Component: Core (Mail) | Version: 1.6
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by levkowetz):
* owner: nobody => levkowetz
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:6>
Django
Reporter: notsqrt | Owner: levkowetz
Type: Bug | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by ris):
Replying to [comment:5 claudep]:
> I don't get your point here, how would we break html code by splitting
lines? Content inside `<pre>`?
Putting a newline in the middle of a tag (href with a loooooong url?)
would do it.
This issue is causing me some pain too.
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:7>
Django
Reporter: notsqrt | Owner: levkowetz
Type: Bug | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by ris):
* cc: bugs@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:8>
Django
Reporter: notsqrt | Owner: levkowetz
Type: Bug | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by dichlofos):
Please note that `SpamAsassin` adds points only when base64 is used for
encodings that '''do not require it''':
http://wiki.apache.org/spamassassin/Rules/MIME_BASE64_TEXT
A message with long lines '''do require''' some kind of encoding (not to
violate the RFC), so if SA adds scores for such letters, it is a bug in SA
(and contradiction with its documentation).
To be sure, you can check that message body has long lines and enable
base64 encoding flag only in that case. This will definitely save
bandwidth in case of well-formatted message bodies.
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:9>
Django
Reporter: notsqrt | Owner: levkowetz
Type: Bug | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by nijel):
* cc: michal@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:10>
Django
Reporter: notsqrt | Owner: levkowetz
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by claudep):
* has_patch: 0 => 1
* version: 1.6 => master
Comment:
I added [https://github.com/django/django/pull/6469 this PR] to fallback
to QP encoding when the body has lines longer than 998. Would this be a
good solution?
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:11>
Django
Reporter: notsqrt | Owner: levkowetz
Type: Bug | Status: assigned
Component: Core (Mail) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by notsqrt):
Seems a good solution to me !
Replying to [comment:11 claudep]:
> I added [https://github.com/django/django/pull/6469 this PR] to fallback
to QP encoding when the body has lines longer than 998. Would this be a
good solution?
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:12>
Django
Reporter: notsqrt | Owner: levkowetz
Type: Bug | Status: assigned
Component: Core (Mail) | Version: master
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:13>
Django
Reporter: notsqrt | Owner: levkowetz
Component: Core (Mail) | Version: master
Keywords: | Triage Stage: Ready for checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz <claude@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"836d475afefecd643d5e7f44027d7209df3ac690" 836d475a]:
{{{
#!CommitTicketReference repository=""
revision="836d475afefecd643d5e7f44027d7209df3ac690"
Fixed #22561 -- Prevented too long lines in email messages
Thanks NotSqrt for the excellent report and Tim Graham for the review.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:14>
Django
Reporter: notsqrt | Owner: levkowetz
Component: Core (Mail) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by ppokrovsky):
* status: closed => new
* resolution: fixed =>
Comment:
It appears the fix does not work properly on Cyrillic strings.
Tried it out with following snippet:
https://gist.github.com/ppokrovsky/d06d0d9e3c8f55bf15984ecd22954683
with body set to {{{ test_body_lat }}}, the {{{has_long_lines}}} flag in
{{{ django.core.mail.message.SafeMIMEText() }}} is set to {{{ True }}} and
therefore correctly applies 'quoted-printable' encoding, while when body
is set to {{{ test_body_ru }}, it leaves {{{has_long_lines}}} as {{{ False
}}} therefore leaving default charset, which results in broken email body.
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:15>
Django
Reporter: notsqrt | Owner: levkowetz
Component: Core (Mail) | Version: master
Keywords: | Triage Stage: Ready for checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz):
* status: new => closed
* resolution: => fixed
Comment:
Instead of reopening the fixed ticket, could you please create a new one
(where you can mention this ticket)?
I already have a pull request ready to fix your issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:16>
Django
Reporter: notsqrt | Owner: levkowetz
Type: Bug | Status: closed
Component: Core (Mail) | Version: master
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Pavel Pokrovskiy):
Replying to [comment:16 Claude Paroz]:
> Instead of reopening the fixed ticket, could you please create a new one
(where you can mention this ticket)?
> I already have a pull request ready to fix your issue.
#27696
Appreciated
--
Ticket URL: <https://code.djangoproject.com/ticket/22561#comment:17>