[Django] #17471: Using TLS and port 465 for SMTP email causing Django to hang sending.
Django
-------------------------------+--------------------
Reporter: dje | Owner: nobody
Type: Bug | Status: new
Component: Uncategorized | Version: 1.3
Severity: Normal | Keywords: smtp
Triage Stage: Unreviewed | Has patch: 1
Easy pickings: 1 | UI/UX: 0
-------------------------------+--------------------
There was a ticket opened three years ago to report this issue and it was
brushed off with no investigation:
https://code.djangoproject.com/ticket/9575. I've attached my patch which
completely resolved the problem on my local machine. The problem seems to
arise when using TLS, with username/password and port 465. I didn't have
any servers with a port other than 465 to test with, but the previous bug
report mentioned changing the port worked for them.[[BR]]
To reproduce the issue, I'd recommend using Gmail's servers (since they
are free):[[BR]]
EMAIL_HOST_USER = 'user'[[BR]]
EMAIL_HOST_PASSWORD = 'pass'[[BR]]
EMAIL_USE_TLS = True[[BR]]
EMAIL_HOST = 'smtp.gmail.com'[[BR]]
EMAIL_PORT = 465[[BR]]
The servers I've tested it against are not just Gmail, so this is clearly
not Google's issue. There may be an issue with the underlying Python SMTP
library, but patching this in Django was the quickest way for me to get my
project back up and running.
--
Ticket URL: <https://code.djangoproject.com/ticket/17471>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-----------------------------+--------------------------------------
Reporter: dje | Owner: nobody
Type: New feature | Status: new
Component: Core (Mail) | Version: 1.3
Severity: Normal | Resolution:
Needs tests: 0 | Patch needs improvement: 1
Changes (by kmtracey):
* needs_better_patch: => 1
* component: Uncategorized => Core (Mail)
* needs_tests: => 0
* needs_docs: => 0
* type: Bug => New feature
Comment:
Google's docs now say that port 465 is for SSL, while TLS/STARTTLS is 587
(they didn't used to be that specific as to which port was for what
protocol). What Django implements, when EMAIL_USE_TLS is True, is
starttls. So the proper port to use for Django speaking to GMail servers,
today, given that Django implements TLS and not SSL, is 587, not 465.
The patch as presented can't be used in Django as is. It re-interprets
Django's EMAIL_USE_TLS setting to mean "use an `smtplib.SMTP_SSL`
connection". Such a connection doesn't work for servers who implement TLS,
not SSL. For example if you use the patched code against GMail's server
port 587 you get an exception: SSLError: (1, '_ssl.c:503:
error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol'). So
if we were to change the code as is done in the patch, suddenly everyone
who had a working config talking to a TLS server would be broken.
It seems like what is needed to talk to an SSL-implementing server (like
GMail port 465) is yet another option (sigh) to tell the email code to use
an `smtplib.SMTP_SSL` connection rather than doing the starttls thing.
Note `smtplib.SMTP_SSL` is new in Python 2.6 so we cannot support this new
type of email security except when running under Python 2.6.
Changing this to a feature request rather than a bug, and changing the
summary to match. It's not a bug if things don't work correctly if they've
been misconfigured, and telling Django to use tls against an email server
who is expecting ssl and not tls is a misconfiguration. But what we are
missing on the Djagno side is apparently any way to do the ssl connection
instead of TLS, and if there are common servers that implement only SSL
and not TLS then Django likely should support talking to them.
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:1>
Django
-----------------------------+--------------------------------------
Component: Core (Mail) | Version: 1.3
Severity: Normal | Resolution:
Comment (by dje):
Thank you for the correction. I'll follow up with the providers I was
having trouble with to see if their documentation is wrong. One such
provider is Amazon's SES service, so they are becoming quite a common
server (I personally am developing two applications using them). They
indicate TLS, but it only works with the SSL connection.
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:2>
Django
-----------------------------+--------------------------------------
Type: Bug | Status: new
Severity: Normal | Resolution:
Needs tests: 0 | Patch needs improvement: 1
Changes (by dje):
* cc: dj.facebook@… (added)
* has_patch: 1 => 0
* type: New feature => Bug
Comment:
So I've spent a while reading through the RFC specs on TLSv1 and
discussing with the developers of a few of the providers I was having
issues with and it turns out that this is actually a *bug* and not a new
feature. TLSv1 has two operational modes: explicit and implicit when
forming the connection/handshake. Django incorrectly assumes that all TLS
connections are explicit connections. Explicit connections use STARTTLS
and are generally accepted on port 587. Implicit connections (which may
also fall back to SSL) are typically made over port 465 and should not use
the STARTTLS command.
I'm changing this back to a bug as it is a misinterpretation of the spec
and only half implemented.
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:3>
Django
-----------------------------+--------------------------------------
Type: Bug | Status: new
Severity: Normal | Resolution:
Changes (by dje):
* cc: dj.facebook@… (removed)
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:4>
Django
-----------------------------+------------------------------------
Type: Bug | Status: new
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Changes (by aaugustin):
* needs_docs: 0 => 1
* has_patch: 0 => 1
* needs_tests: 0 => 1
* stage: Unreviewed => Accepted
Comment:
I'll skip the bug vs. feature request debate; it doesn't really matter.
To sum up:
- Currently, Django supports only explicit connections (with STARTTLS);
- This ticket is about supporting implicit connections too;
- The change must be backwards compatible, which is likely to require yet
another setting.
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:5>
Django
-----------------------------+------------------------------------
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: smtp | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Changes (by dje):
* type: Bug => New feature
Comment:
Yeah, sorry about switching it back to a bug. It probably should have been
left as a feature. I've attached a patch that should maintain backwards
compatibility. The only parts I was unsure about was adding use_ssl to the
constructor and whether to call starttls if they set TLS and SSL to true.
Technically, they are supposed to be mutually exclusive, which is how I
coded it.
What's in the patch:
A new setting: EMAIL_USE_SSL
Updated documentation.
Added tests around USE_TLS and USE_SSL
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:6>
Django
-----------------------------+------------------------------------
Type: New feature | Status: new
Component: Core (Mail) | Version: 1.3
Severity: Normal | Resolution:
Keywords: smtp | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Easy pickings: 1 | UI/UX: 0
Changes (by sorin):
* cc: sorin (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:7>
Django
-----------------------------+------------------------------------
Type: New feature | Status: new
Component: Core (Mail) | Version: 1.3
Severity: Normal | Resolution:
Keywords: smtp | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Easy pickings: 1 | UI/UX: 0
Comment (by ezequielsz):
I just send a pull request for this, this is the link
https://github.com/django/django/pull/792
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:8>
Django
Reporter: dje | Owner: nobody
Type: New feature | Status: new
Component: Core (Mail) | Version: 1.3
Severity: Normal | Resolution:
django-ar | Needs documentation: 1
Has patch: 1 | Patch needs improvement: 1
Needs tests: 1 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
Changes (by fgallina):
* keywords: smtp => smtp sprints-django-ar
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:9>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: smtp sprints- | Triage Stage: Accepted
django-ar | Needs documentation: 1
Has patch: 1 | Patch needs improvement: 1
Needs tests: 1 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
* cc: charettes (added)
* version: 1.3 => master
Comment:
Reviewed the PR.
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:10>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Core (Mail) | Version: master
Severity: Normal | Resolution:
Keywords: smtp sprints- | Triage Stage: Accepted
django-ar | Needs documentation: 1
Has patch: 1 | Patch needs improvement: 1
Needs tests: 1 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
Comment (by areski):
During djangocon eu, did review of the patch and PR
please find a new rebase patch in attachment and a new PR
https://github.com/django/django/pull/1124
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:11>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Core (Mail) | Version: master
Severity: Normal | Resolution:
Keywords: smtp sprints- | Triage Stage: Accepted
django-ar | Needs documentation: 1
Has patch: 1 | Patch needs improvement: 1
Needs tests: 1 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
Comment (by claudep):
This is almost RFC, but on Python 3.2, I'm getting this output (but the
test succeeds anyway):
`.error: uncaptured python exception, closing channel <smtpd.SMTPChannel
connected 127.0.0.1:47904 at 0x38db710> (<class
'UnicodeDecodeError'>:'utf-8' codec can't decode byte 0x99 in position 12:
invalid start byte [/usr/lib/python3.2/asyncore.py|read|83]
[/usr/lib/python3.2/asyncore.py|handle_read_event|449]
[/usr/lib/python3.2/asynchat.py|handle_read|190]
[/usr/lib/python3.2/smtpd.py|collect_incoming_data|278]
[/home/claude/virtualenvs/djangogit32/lib/python3.2/encodings/utf_8.py|decode|16])`
If this is reproducible on other systems, it would be nice if this could
be silenced...
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:12>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Core (Mail) | Version: master
Severity: Normal | Resolution:
Keywords: smtp sprints- | Triage Stage: Accepted
django-ar | Needs documentation: 1
Needs tests: 1 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
* cc: senko (added)
* needs_better_patch: 1 => 0
Comment:
I've updated the previous patch to use a fake SMTPChannel which ignores
the UnicodeDecodeError.
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:13>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: closed
Component: Core (Mail) | Version: master
Keywords: smtp sprints- | Triage Stage: Accepted
django-ar | Needs documentation: 1
Has patch: 1 | Patch needs improvement: 0
Needs tests: 1 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"59ebe39812858ba37e83ab0ee886f676980d472d"]:
{{{
#!CommitTicketReference repository=""
revision="59ebe39812858ba37e83ab0ee886f676980d472d"
Fixed #17471 -- Added smtplib.SMTP_SSL connection option for SMTP backend
Thanks dj.facebook at gmail.com for the report and initial patch
and Areski Belaid and senko for improvements.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:14>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: closed
Component: Core (Mail) | Version: master
Severity: Normal | Resolution: fixed
Keywords: smtp sprints- | Triage Stage: Accepted
django-ar | Needs documentation: 1
Has patch: 1 | Patch needs improvement: 0
Needs tests: 1 | UI/UX: 0
Easy pickings: 1 |
-------------------------------------+-------------------------------------
Comment (by timo):
#13142 is a duplicate which contains some additional settings for the
cert/key files.
--
Ticket URL: <https://code.djangoproject.com/ticket/17471#comment:15>