Re: [Django] #35365: Add RFC 3834 Auto-Submitted header to emails by default

[Django]

#35365: Add RFC 3834 Auto-Submitted header to emails by default
---------------------------------+------------------------------------
Reporter: Tobias Bengfort | Owner: cgracin
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------
Comment (by Tobias Bengfort):

The documentation already contains this line:

Not all features of the EmailMessage class are available through the
send_mail() and related wrapper functions. If you wish to use advanced
features, such as BCC’ed recipients, file attachments, or multi-part
email, you’ll need to create EmailMessage instances directly.
https://docs.djangoproject.com/en/5.0/topics/email/#the-emailmessage-
class

For consistency I think we should also add the `Auto-Submitted` header in
the wrapper functions.

The special thing here would be that dropping down to `EmailMessage`
allows you to ''remove'' a header rather than add one. I am not sure how
best to explain that in the docs. I see that most parameters are only
documented once for `send_mail()` and not repeated for the other wrapper
functions. So maybe it would be sufficient to add a note only to
`send_mail()`. The note could be something like this:

`send_mail()` uses the `Auto-Submitted` mail header to indicate that
the mail was created by software rather than a human.
--
Ticket URL: <https://code.djangoproject.com/ticket/35365#comment:9>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35365: Add RFC 3834 Auto-Submitted header to emails by default
---------------------------------+------------------------------------
Reporter: Tobias Bengfort | Owner: cgracin
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by cgracin):

Thank you guys for the comments, I'll work on implementing these requests
and submit an updated PR.
--
Ticket URL: <https://code.djangoproject.com/ticket/35365#comment:10>

[Django]

#35365: Add RFC 3834 Auto-Submitted header to emails by default
---------------------------------+------------------------------------
Reporter: Tobias Bengfort | Owner: cgracin
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by David Smith):

Does the header need to be removed entirely? Could we advise folk to set
the header to "no" to disable it?
https://www.iana.org/assignments/auto-submitted-keywords/auto-submitted-
keywords.xhtml

The current patch has this to add the header:

{{{
if "Auto-Submitted" not in self.extra_headers:
# Default to adding the Auto-Submitted: auto-generated header
self.extra_headers["Auto-Submitted"] = "auto-generated"
}}}

So could we documentat this by something like...

> By default `EmailMessage` sets the `Auto-Submitted` header to `auto-
generated` to indicate that the mail was created by software rather than a
human. The value of the `Auto-Submitted` header can be customised by the
`headers` option. To disable the header the value of `Auto-Submitted` can
be set to "no".
--
Ticket URL: <https://code.djangoproject.com/ticket/35365#comment:11>

[Django]

#35365: Add RFC 3834 Auto-Submitted header to emails by default
---------------------------------+------------------------------------
Reporter: Tobias Bengfort | Owner: cgracin
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by Florian Apolloner):

I don't feel good about this change. If we are to add this header by
default we need a way to disable it by default as well imo (especially
since one cannot patch 3rd party apps easily). This probably ties into
making smtp backends configurable and for instance add options to set
headers there…
--
Ticket URL: <https://code.djangoproject.com/ticket/35365#comment:12>

[Django]

#35365: Add RFC 3834 Auto-Submitted header to emails by default
---------------------------------+------------------------------------
Reporter: Tobias Bengfort | Owner: cgracin
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Changes (by Mike Edmunds):

* cc: Mike Edmunds (added)

Comment:

[[https://github.com/anymail/django-anymail django-anymail] maintainer
here]

This will break sending for many users working with an Email Service
Provider.

Most ESPs restrict the headers that can be used. Some will silently strip
headers they don't allow, but others will reject the message entirely.
(The restrictions are sometimes different for the ESP's SMTP endpoint vs.
their HTTP API. A few ESPs don't allow ''any'' extra headers.)

I would suggest, instead, documenting how users can set this header
themselves when sending a message. For users who want it on all messages,
one option is a "wrapper" email backend that adds the header:

{{{#!python
# Add this to your project, and in settings.py set:
# EMAIL_BACKEND="path.to.this.file.AutoSubmittedSMTPBackend"
class
AutoSubmittedSMTPBackend(django.core.mail.backends.smtp.EmailBackend):
def send_messages(messages):
for message in messages:
message.extra_headers.setdefault("Auto-Submitted", "auto-
generated")
return super().send_messages(messages)
}}}

Long term, allowing configurable defaults for each email backend could
simplify this. Looks like #35514 is starting down that path.

FWIW, django-anymail email backends
[https://anymail.dev/en/stable/sending/anymail_additions/#send-defaults,
already support defaults]. If you're using an Anymail backend (with an ESP
that allows this header), you can use:

{{{#!python
# settings.py
ANYMAIL = {
...,
"SEND_DEFAULTS": {
"extra_headers": {
"Auto-Submitted": "auto-generated",
},
},
}
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35365#comment:13>

[Django]

#35365: Add RFC 3834 Auto-Submitted header to emails by default
---------------------------------+------------------------------------
Reporter: Tobias Bengfort | Owner: cgracin
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by Mike Edmunds):

Took a closer look at the proposed patch, and as currently implemented
this will only affect email backends that call `EmailMessage.message()`.
Many non-SMTP email backends don't, so my earlier concern wouldn't apply
to them. But this will still break anyone using an ESP's SMTP endpoint if
their ESP doesn't allow the header. (And if the docs recommend setting
`"Auto-Submitted": "no"` in `headers` to disable the feature, that will
break both SMTP ''and non-SMTP'' backends for those ESPs.)

I appreciate the thought and effort that went into this change (and that
it attracted a new contributor!), but I also don't feel good about it.
After re-reading RFC 3834 section 5 a few times, it's not at all clear to
me that `Auto-Submitted: auto-generated` is an appropriate generalization
for the wide variety of apps that are implemented with Django, and the
emails they might send.

For example, if you're using something like django-helpdesk to reply to a
user's support ticket, you are creating what the RFC calls a "manually
generated message" (it's a direct reply to another message, and not
automatic). The RFC says auto-generated "MUST NOT be used on manually
generated messages." In practical terms, you ''might want'' to see
vacation autoresponses or challenge messages necessary for your support
response to be delivered. At best, this seems like it should be a django-
helpdesk setting, not something Django applies by default to all apps
built with the framework.

I suppose django-helpdesk might be a case where "Django is used to
implement an email client" of sorts. My concern is, I'm not sure a
general-purpose framework like Django can really be sure that sort of use
is a "rare exception."

[Also, to be clear, django-helpdesk ''needs to'' (and does) set `Auto-
Submitted: auto-replied` on any ''automatic'' responses it sends, to
prevent mail loops. And preventing mail loops is the primary point of RFC
3834. But "auto-replied" is always going to be app specific logic, and
isn't covered by this proposed change.]
--
Ticket URL: <https://code.djangoproject.com/ticket/35365#comment:14>

[Django]

#35365: Add RFC 3834 Auto-Submitted header to emails by default
---------------------------------+------------------------------------
Reporter: Tobias Bengfort | Owner: cgracin

Type: New feature | Status: closed

Component: Core (Mail) | Version: dev

Severity: Normal | Resolution: wontfix

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Changes (by Florian Apolloner):

* resolution: => wontfix
* status: assigned => closed

Comment:

I think Mikes analysis is great. It makes sense for applications to set
those headers as needed and we can audit our usage of send_mail in Django
itself (ie admin error emails), but it doesn't seem to make sense for a
framework. Even if it were possible to set this header manually to "no" it
would be backwards incompatible enough that it would be hard to get in
without a transitional setting to opt into the behavior.
--
Ticket URL: <https://code.djangoproject.com/ticket/35365#comment:15>

[Django]

#35365: Add RFC 3834 Auto-Submitted header to emails by default
---------------------------------+------------------------------------
Reporter: Tobias Bengfort | Owner: cgracin

Type: New feature | Status: new

Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------
Changes (by Florian Apolloner):

* resolution: wontfix =>
* status: closed => new

Comment:

Sorry didn't mean to already close the ticket (wontfix is indeed what I am
leaning towards, but that is just my opinion).
--
Ticket URL: <https://code.djangoproject.com/ticket/35365#comment:16>

[Django]

#35365: Add RFC 3834 Auto-Submitted header to emails by default
---------------------------------+------------------------------------
Reporter: Tobias Bengfort | Owner: cgracin
Type: New feature | Status: new
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by Tobias Bengfort):

I am fine with closing this as wontfix as long as

1. There is an option available to add the header if desired. Mike already
provided a backend, but also mentioned that it does not work in all cases.
2. The documentation explains how to do this. This could use more work.
--
Ticket URL: <https://code.djangoproject.com/ticket/35365#comment:17>

[Django]

#35365: Add RFC 3834 Auto-Submitted header to emails by default
---------------------------------+------------------------------------
Reporter: Tobias Bengfort | Owner: cgracin
Type: New feature | Status: new
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by Mike Edmunds):

> Mike already provided a backend, but also mentioned that it does not
work in all cases.

My `AutoSubmittedSMTPBackend` example above is untested code, but should
work in every case where this proposed change would have worked. (It won't
work if your ESP doesn't allow this header, but neither would this PR. My
other example above was specific to using the third-party django-anymail
library, so wouldn't belong in the Django docs.)

I'd be very much in favor of Florian's "making smtp backends configurable
and for instance add options to set headers there" idea. That should be a
separate ticket. This Auto-Submitted header use case would make a great
example for that feature's documentation.
--
Ticket URL: <https://code.djangoproject.com/ticket/35365#comment:18>

[Django]

#35365: Add RFC 3834 Auto-Submitted header to emails by default
---------------------------------+------------------------------------
Reporter: Tobias Bengfort | Owner: cgracin

Type: New feature | Status: closed

Component: Core (Mail) | Version: dev

Severity: Normal | Resolution: wontfix

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Changes (by Sarah Boyce):

* resolution: => wontfix
* status: new => closed

Comment:

Replying to [comment:18 Mike Edmunds]:

> I'd be very much in favor of Florian's "making smtp backends
configurable and for instance add options to set headers there" idea. That
should be a separate ticket. This Auto-Submitted header use case would
make a great example for that feature's documentation.

Agreed 👍 if anyone would like to raise this ticket, I will accept it
I think we are in agreement that we will pivot from the original request
here and can mark this as wontfix. Thank you all for this discussion
--
Ticket URL: <https://code.djangoproject.com/ticket/35365#comment:19>

[Django]

#35365: Add RFC 3834 Auto-Submitted header to emails by default
---------------------------------+------------------------------------
Reporter: Tobias Bengfort | Owner: cgracin
Type: New feature | Status: closed
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by Adam Johnson):

Thank you for the all the input Mike! I agree with wontfixing this.
--
Ticket URL: <https://code.djangoproject.com/ticket/35365#comment:20>