[Django] #36119: Attaching email file to email fails if the attachment is using 8bit Content-Transfer-Encoding

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
---------------------------+---------------------------------------
Reporter: Trenton H | Type: Bug
Status: new | Component: Core (Mail)
Version: 5.1 | Severity: Normal
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------+---------------------------------------
If the attached email file is attached to an email, such as the code
snipped below, the sending will fail due to the presence of non-ASCII
content in the attachment.

{{{#!python
email = EmailMessage(
subject="subject",
body="body",
to="som...@somewhere.com",
)
email.attach_file(original_file)
n_messages = email.send()
}}}

{{{
Traceback (most recent call last):
File "/usr/src/paperless/src/documents/signals/handlers.py", line 989,
in email_action
n_messages = email.send()
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-
packages/django/core/mail/message.py", line 301, in send
return self.get_connection(fail_silently).send_messages([self])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-
packages/django/core/mail/backends/smtp.py", line 136, in send_messages
sent = self._send(message)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-
packages/django/core/mail/backends/smtp.py", line 156, in _send
from_email, recipients, message.as_bytes(linesep="\r\n")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-
packages/django/core/mail/message.py", line 148, in as_bytes
g.flatten(self, unixfrom=unixfrom, linesep=linesep)
File "/usr/local/lib/python3.12/email/generator.py", line 117, in
flatten
self._write(msg)
File "/usr/local/lib/python3.12/email/generator.py", line 182, in _write
self._dispatch(msg)
File "/usr/local/lib/python3.12/email/generator.py", line 219, in
_dispatch
meth(msg)
File "/usr/local/lib/python3.12/email/generator.py", line 286, in
_handle_multipart
g.flatten(part, unixfrom=False, linesep=self._NL)
File "/usr/local/lib/python3.12/email/generator.py", line 117, in
flatten
self._write(msg)
File "/usr/local/lib/python3.12/email/generator.py", line 182, in _write
self._dispatch(msg)
File "/usr/local/lib/python3.12/email/generator.py", line 219, in
_dispatch
meth(msg)
File "/usr/local/lib/python3.12/email/generator.py", line 372, in
_handle_message
g.flatten(msg.get_payload(0), unixfrom=False, linesep=self._NL)
File "/usr/local/lib/python3.12/email/generator.py", line 117, in
flatten
self._write(msg)
File "/usr/local/lib/python3.12/email/generator.py", line 182, in _write
self._dispatch(msg)
File "/usr/local/lib/python3.12/email/generator.py", line 219, in
_dispatch
meth(msg)
File "/usr/local/lib/python3.12/email/generator.py", line 446, in
_handle_text
super(BytesGenerator,self)._handle_text(msg)
File "/usr/local/lib/python3.12/email/generator.py", line 263, in
_handle_text
self._write_lines(payload)
File "/usr/local/lib/python3.12/email/generator.py", line 156, in
_write_lines
self.write(line)
File "/usr/local/lib/python3.12/email/generator.py", line 420, in write
self._fp.write(s.encode('ascii', 'surrogateescape'))

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'ascii' codec can't encode characters in position
10-15: ordinal not in range(128)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36119>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding

-----------------------------+--------------------------------------
Reporter: Trenton H | Owner: (none)

Type: Bug | Status: new
Component: Core (Mail) | Version: 5.1

Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------+--------------------------------------
Changes (by Trenton H):

* Attachment "problem.eml" added.

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+--------------------------------------
Reporter: Trenton H | Owner: (none)
Type: Bug | Status: new
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Tim Graham):

Trenton, It would be helpful if you could provide some more information
about this because I'm not sure we have any triagers who are experts in
this area. Where is Django at fault and how can we fix it?
--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:1>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+--------------------------------------
Reporter: Trenton H | Owner: (none)
Type: Bug | Status: new
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Trenton H):

Assuming SMTP is configured for a project:

{{{#!python
from django.core.mail import EmailMessage

email = EmailMessage(
subject="subject",
body="body",
to="som...@somewhere.com",
)

email.attach_file("problem.eml")
# or
email.attach_file("problem.eml", "message/rfc822")
n_messages = email.send()
}}}

I would expect this to work without issue, but as shown above, Django
appear to either assume ASCII or uses the standard library in a way that
assumes ASCII. As a user, I would not expect to need extra steps or
processing for this code to work without a crash and just attach the file
instead.

I don't know about the internals to say how it would be fixed. Perhaps
checking the headers of attached messages? Defaulting to utf-8 somewhere?
--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:2>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+--------------------------------------
Reporter: Trenton H | Owner: (none)
Type: Bug | Status: new
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Sarah Boyce):

* cc: Mike Edmunds (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:3>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+------------------------------------

Reporter: Trenton H | Owner: (none)
Type: Bug | Status: new
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:

Keywords: compat32 | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+------------------------------------

Changes (by Mike Edmunds):

* keywords: => compat32
* stage: Unreviewed => Accepted

Comment:

This came up [https://forum.djangoproject.com/t/using-emailmessage-with-
an-attached-email-file-crashes-due-to-non-ascii/37981 in the forum].
Here's a minimal case to reproduce:

{{{#!python
from django.core.mail import EmailMessage

# Content of message to attach, using 8bit CTE with raw utf-8:
att = """\
Subject: attachment
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

¡8-bit content!
""".encode()

email = EmailMessage(to=["te...@example.com"])
email.attach("attachment.eml", att, "message/rfc822")

email.message().as_bytes()
# ...python3.12/email/generator.py", line 409, in write
# self._fp.write(s.encode('ascii', 'surrogateescape'))
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# UnicodeEncodeError: 'ascii' codec can't encode character '\xa1' in
position 0: ordinal not in range(128)
}}}

The problem is that
[https://github.com/django/django/blob/5244ecbd2259365ecd6bbf96747285a673b2ee69/django/core/mail/message.py#L399-L402
EmailMessage._create_mime_attachment()] uses
`message_from_string(force_str(content))` to convert the attachment
content to a Python Message object. But message_from_string() doesn't
properly handle Unicode characters. (See
[https://github.com/python/cpython/issues/83565#issuecomment-1093851962
python/cpython#83565]. This dates back to Python 2 when strings couldn't
include Unicode.)

A working alternative is message_from_bytes():

{{{#!python
from email import message_from_string, message_from_bytes

message_from_string(att.decode()).as_bytes()
# UnicodeEncodeError: 'ascii' codec can't encode character '\xa1' in
position 0: ordinal not in range(128)

message_from_bytes(att).as_bytes()
# b'Subject: attachment\nContent-Type: text/plain; charset=utf-8\nContent-
Transfer-Encoding: 8bit\n\n\xc2\xa18-bit content!\n'
}}}

So the simplest fix is probably to change Django to use
`message_from_bytes(force_bytes(content))`.

(This would also be fixed by upgrading Django to use Python's modern email
APIs, #35581.)
--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:4>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+------------------------------------
Reporter: Trenton H | Owner: (none)
Type: Bug | Status: new
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:
Keywords: compat32 | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+------------------------------------

Comment (by Gregory Mariani):

I can have a try but #35581 looks like close to be released ? Any advice
to start the work ?
--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:5>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+------------------------------------
Reporter: Trenton H | Owner: (none)
Type: Bug | Status: new
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:
Keywords: compat32 | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+------------------------------------

Comment (by Mike Edmunds):

#35581 is stalled waiting for some Python core library fixes, so won't get
merged before Django 6.0 at the earliest. I think it's worth fixing this
bug before then.

Maybe start by adding a new, failing test case in
django/tests/mail/tests.py. (That's a long file, and not particularly well
organized, but most of the attachment-related tests are kind of grouped
together in the middle, so maybe somewhere near them.) You could probably
even use the minimal example from my earlier comment. (Or make it even
more minimal: neither `to` nor `Subject` are relevant to the bug.)

Once that test fails, I'm pretty sure you can fix it in
django/core/mail/message.py by changing
`message_from_string(force_str(content))` to
`message_from_bytes(force_bytes(content))` (and fixing up the imports).
And then running all the tests to see if that breaks anything else. (I
wouldn't expect it to, but I'm often wrong, so it's good we have tests.)
--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:6>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding

-----------------------------+-------------------------------------------
Reporter: Trenton H | Owner: Gregory Mariani
Type: Bug | Status: assigned

Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:
Keywords: compat32 | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------+-------------------------------------------
Changes (by Gregory Mariani):

* owner: (none) => Gregory Mariani
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:7>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+-------------------------------------------
Reporter: Trenton H | Owner: Gregory Mariani
Type: Bug | Status: assigned
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:
Keywords: compat32 | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+-------------------------------------------

Comment (by Gregory Mariani):

Patch should be done, CI is runnning and I can't stay awake to see the end
--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:8>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+-------------------------------------------
Reporter: Trenton H | Owner: Gregory Mariani
Type: Bug | Status: assigned
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:
Keywords: compat32 | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+-------------------------------------------

Changes (by Gregory Mariani):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:9>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding

-----------------------------+---------------------------------------------

Reporter: Trenton H | Owner: Gregory Mariani
Type: Bug | Status: assigned
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:

Keywords: compat32 | Triage Stage: Ready for checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------+---------------------------------------------
Changes (by Mike Edmunds):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:10>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+-------------------------------------------

Reporter: Trenton H | Owner: Gregory Mariani
Type: Bug | Status: assigned
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:

Keywords: compat32 | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+-------------------------------------------

Changes (by Mike Edmunds):

* stage: Ready for checkin => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:11>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+-------------------------------------------
Reporter: Trenton H | Owner: Gregory Mariani
Type: Bug | Status: assigned
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:
Keywords: compat32 | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-----------------------------+-------------------------------------------

Changes (by Sarah Boyce):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:12>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+-------------------------------------------
Reporter: Trenton H | Owner: Gregory Mariani
Type: Bug | Status: assigned
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:
Keywords: compat32 | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-----------------------------+-------------------------------------------

Changes (by Gregory Mariani):

* needs_better_patch: 1 => 0

Comment:

@sarah it's ok I think
--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:13>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding

-----------------------------+---------------------------------------------

Reporter: Trenton H | Owner: Gregory Mariani
Type: Bug | Status: assigned
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution:

Keywords: compat32 | Triage Stage: Ready for checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------+---------------------------------------------
Changes (by Sarah Boyce):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:14>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+---------------------------------------------
Reporter: Trenton H | Owner: Gregory Mariani

Type: Bug | Status: closed

Component: Core (Mail) | Version: 5.1

Severity: Normal | Resolution: fixed

Keywords: compat32 | Triage Stage: Ready for checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+---------------------------------------------

Changes (by Sarah Boyce <42296566+sarahboyce@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"89e28e13ecbf9fbcf235e16d453c08bbf2271244" 89e28e13]:
{{{#!CommitTicketReference repository=""
revision="89e28e13ecbf9fbcf235e16d453c08bbf2271244"
Fixed #36119 -- Fixed UnicodeEncodeError when attaching a file with 8bit
Content-Transfer-Encoding.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:15>

[Django]

#36119: Attaching email file to email fails if the attachment is using 8bit
Content-Transfer-Encoding
-----------------------------+---------------------------------------------
Reporter: Trenton H | Owner: Gregory Mariani
Type: Bug | Status: closed
Component: Core (Mail) | Version: 5.1
Severity: Normal | Resolution: fixed
Keywords: compat32 | Triage Stage: Ready for checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+---------------------------------------------

Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"2146bd1261905f74df615ebfa1ac56dd735baee8" 2146bd1]:
{{{#!CommitTicketReference repository=""
revision="2146bd1261905f74df615ebfa1ac56dd735baee8"
[5.2.x] Fixed #36119 -- Fixed UnicodeEncodeError when attaching a file
with 8bit Content-Transfer-Encoding.

Backport of 89e28e13ecbf9fbcf235e16d453c08bbf2271244 from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36119#comment:16>