[Python-Dev] urllib request HTTPS memory leak?

[V-ron P]

Hello Python Dev,

One of my HTTPS servers leaked over 8 GB of memory over the past week. After troubleshooting the issue, I managed to narrow it down to the minimal test case below:

https://gist.github.com/hydrogen-mvm/d588521c2138e0f98aa94b18e7dedfdb  

(Note that the 100 threads are not necessary, but they help exaggerate and accelerate the memory leaking.)

The memory leak happens if I send a HTTPS request, regardless if it's GET or POST.  The most bizarre part is the destination address does -not- need to exist and it still leaks! (In the script above it points to a non-existent HTTPS server on your localhost.)

This leak does not happen for HTTP (plaintext) requests, which suggests that the bug might be SSL-related.

My OS is Windows 7 and my Python version is:

Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32

A friend of mine tested the script on his Windows 10 machine and also experienced the same memory leaking effect.

Please confirm if this is in fact a bug in Python or this is a bug in my environment. Thanks.

[Christian Heimes]

Yes, we are aware of a memory leak on Windows. A recent change has
caused a memory leak and performance regression in the Windows-only part
of the ssl module. The bug is fixed in 3.7 branch but not yet released.

See https://bugs.python.org/issue37702 and
https://bugs.python.org/issue35941 for more details.

As a workaround I suggest that you create a single SSLContext with
ssl.create_default_context() and reuse the context in all HTTP queries.
You can share the context across threads w/o locking and use it for all
HTTPS connections. A single SSLContext is also more efficient.

Christian

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/CYKQCXVPJBFGUDMEE7QFQ5EWQF3LUJDG/