[Django] #35361: SafeMIMEText.set_payload() crash on Python 3.11.9

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9
-------------------------------------------+------------------------
Reporter: Sarah Boyce | Owner: nobody
Type: Bug | Status: new
Component: Core (Mail) | Version: 4.2
Severity: Release blocker | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------------+------------------------
A recent bug fix to Python [https://github.com/python/cpython/issues/76511
gh-76511] has been back ported to Python 3.11 and 3.12.
See the release notes of
[https://docs.python.org/release/3.11.9/whatsnew/changelog.html#python-3-11-9
Python 3.11.9] and the soon to be released
[https://docs.python.org/3/whatsnew/changelog.html Python 3.12.3].

As part of the work to prepare Django for
[https://code.djangoproject.com/ticket/34900 Python 3.13], a fix for this
has already been merged to main in
b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8.

Believe this needs to be back ported to Django 4.2 and Django 5.0 to
support the latest (or soon to be latest) Python 3.11 and Python 3.12
versions.

Confirmed when testing Django 4.2 against Python 3.11.9, we know have the
following test failure:

{{{
======================================================================
ERROR [0.034s]: test_dont_base64_encode
(tests.mail.tests.MailTests.test_dont_base64_encode)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tests/django/tests/mail/tests.py", line 902, in
test_dont_base64_encode
s = msg.message().as_string()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tests/django/django/core/mail/message.py", line 133, in as_string
g.flatten(self, unixfrom=unixfrom, linesep=linesep)
File "/usr/local/lib/python3.11/email/generator.py", line 115, in
flatten
self._write(msg)
File "/usr/local/lib/python3.11/email/generator.py", line 180, in _write
self._dispatch(msg)
File "/usr/local/lib/python3.11/email/generator.py", line 217, in
_dispatch
meth(msg)
File "/usr/local/lib/python3.11/email/generator.py", line 246, in
_handle_text
msg.set_payload(msg._payload, charset)
File "/tests/django/django/core/mail/message.py", line 168, in
set_payload
has_long_lines = any(
^^^^
File "/tests/django/django/core/mail/message.py", line 169, in <genexpr>
len(line.encode()) > RFC5322_EMAIL_LINE_LENGTH_LIMIT
^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode characters in position
28-33: surrogates not allowed

}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35361>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9

---------------------------------+------------------------------------

Reporter: Sarah Boyce | Owner: nobody
Type: Bug | Status: new
Component: Core (Mail) | Version: 4.2

Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

---------------------------------+------------------------------------
Changes (by Natalia Bidart):

* stage: Unreviewed => Accepted

Comment:

Related forum discussion: https://forum.djangoproject.com/t/send-mail-
throws-unicodeencodeerror-under-python-3-11-9/29871/3
--
Ticket URL: <https://code.djangoproject.com/ticket/35361#comment:1>

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9

-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: Natalia
| Bidart
Type: Bug | Status: assigned

Component: Core (Mail) | Version: 4.2
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Natalia Bidart):

* has_patch: 0 => 1
* owner: nobody => Natalia Bidart
* status: new => assigned

Comment:

PR for main adding a test (to be backported along with the fix from
b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8):
https://github.com/django/django/pull/18057/
--
Ticket URL: <https://code.djangoproject.com/ticket/35361#comment:2>

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: Natalia
| Bidart
Type: Bug | Status: assigned
Component: Core (Mail) | Version: 4.2
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"338ec052b4824d3f621629b3bf0f344c7a43c3dc" 338ec052]:
{{{#!CommitTicketReference repository=""
revision="338ec052b4824d3f621629b3bf0f344c7a43c3dc"
Refs #35361 -- Added test for Email line length checks when dealing with
surrogate pairs.

Refs #33173, #34118 and #34900.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35361#comment:3>

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: Natalia
| Bidart

Type: Bug | Status: closed

Component: Core (Mail) | Version: 4.2

Severity: Release blocker | Resolution: fixed

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce <42296566+sarahboyce@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"42435fc55cbf7c04c1389ee46cc50e2565b40e37" 42435fc5]:
{{{#!CommitTicketReference repository=""
revision="42435fc55cbf7c04c1389ee46cc50e2565b40e37"
Fixed #35361 -- Added release notes for 4.2.12 and 5.0.5 for backport of
b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35361#comment:4>

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: Natalia
| Bidart
Type: Bug | Status: closed
Component: Core (Mail) | Version: 4.2
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"23d605d36435f0db8ae4c3e0ccab24287d299199" 23d605d3]:
{{{#!CommitTicketReference repository=""
revision="23d605d36435f0db8ae4c3e0ccab24287d299199"
[5.0.x] Fixed #35361 -- Added release notes for 4.2.12 and 5.0.5 for
backport of b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8.

Backport of 42435fc55cbf7c04c1389ee46cc50e2565b40e37 from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35361#comment:6>

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: Natalia
| Bidart
Type: Bug | Status: closed
Component: Core (Mail) | Version: 4.2
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"2e6ae1e96a405fc2ac318325956ce276cfb9830c" 2e6ae1e9]:
{{{#!CommitTicketReference repository=""
revision="2e6ae1e96a405fc2ac318325956ce276cfb9830c"
[5.0.x] Refs #34900, Refs #35361 -- Fixed SafeMIMEText.set_payload() crash
on Python 3.13.

Payloads with surrogates are passed to the set_payload() since
https://github.com/python/cpython/commit/f97f25ef5dfcdfec0d9a359fd970abd139cf3428

Backport of b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8 from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35361#comment:5>

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: Natalia
| Bidart
Type: Bug | Status: closed
Component: Core (Mail) | Version: 4.2
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"0d3ddcaf2c74638a32781f361d467af572ced95f" 0d3ddcaf]:
{{{#!CommitTicketReference repository=""
revision="0d3ddcaf2c74638a32781f361d467af572ced95f"
[4.2.x] Refs #34900, Refs #35361 -- Fixed SafeMIMEText.set_payload() crash

on Python 3.13.

Payloads with surrogates are passed to the set_payload() since
https://github.com/python/cpython/commit/f97f25ef5dfcdfec0d9a359fd970abd139cf3428

Backport of b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8 from main.
}}}
--

Ticket URL: <https://code.djangoproject.com/ticket/35361#comment:7>

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: Natalia
| Bidart
Type: Bug | Status: closed
Component: Core (Mail) | Version: 4.2
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"27c32cc9916502132a57cd2a60b1402869e00537" 27c32cc]:
{{{#!CommitTicketReference repository=""
revision="27c32cc9916502132a57cd2a60b1402869e00537"
[4.2.x] Fixed #35361 -- Added release notes for 4.2.12 for backport of
b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8.

Backport of 42435fc55cbf7c04c1389ee46cc50e2565b40e37 from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35361#comment:8>

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: Natalia
| Bidart
Type: Bug | Status: closed
Component: Core (Mail) | Version: 4.2
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"cd823778e66307b82469858cfd8d1aa75613b49a" cd823778]:
{{{#!CommitTicketReference repository=""
revision="cd823778e66307b82469858cfd8d1aa75613b49a"
Refs #35361 -- Clarified release notes for 4.2.12 and 5.0.5.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35361#comment:9>

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: Natalia
| Bidart
Type: Bug | Status: closed
Component: Core (Mail) | Version: 4.2
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"a1baaddb67a59f32db9e10f44ac76e9835ca4e84" a1baaddb]:
{{{#!CommitTicketReference repository=""
revision="a1baaddb67a59f32db9e10f44ac76e9835ca4e84"
[5.0.x] Refs #35361 -- Clarified release notes for 4.2.12 and 5.0.5.

Backport of cd823778e66307b82469858cfd8d1aa75613b49a from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35361#comment:10>

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: Natalia
| Bidart
Type: Bug | Status: closed
Component: Core (Mail) | Version: 4.2
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"1d85b416aa1382e7274e718c79b83dd050fb24ab" 1d85b416]:
{{{#!CommitTicketReference repository=""
revision="1d85b416aa1382e7274e718c79b83dd050fb24ab"
[4.2.x] Refs #35361 -- Clarified release notes for 4.2.12.

Backport of cd823778e66307b82469858cfd8d1aa75613b49a from main.
}}}
--

Ticket URL: <https://code.djangoproject.com/ticket/35361#comment:11>

[Django]

#35361: SafeMIMEText.set_payload() crash on Python 3.11.9
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: Natalia
| Bidart
Type: Bug | Status: closed
Component: Core (Mail) | Version: 4.2
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Anders Kaseorg):

A disadvantage of the current May 6 schedule for the release with these
Python 3.11.9 and 3.12.3 compatibility fixes is that Ubuntu 24.04 LTS will
be releasing with Python 3.12.3 on [https://discourse.ubuntu.com/t/noble-
numbat-release-schedule/35649 April 25].
--
Ticket URL: <https://code.djangoproject.com/ticket/35361#comment:12>