[Django] #32740: Running colorama.init() at import time causes deployment error
Django
-----------------------------------------+---------------------------------
Reporter: Leon Matthews | Owner: nobody
Type: Uncategorized | Status: new
Component: Core (Other) | Version: 3.2
Severity: Normal | Keywords: apache mod_wsgi
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+---------------------------------
[https://code.djangoproject.com/ticket/31216 Ticket 31216] added terminal
colour output on Windows. Unfortunately, there is a small problem with it
that causes deployments to fail under Linux/Apache/mod_wsgi, leaving me
stuck on Django 3.1.
I've tracked it down to line 16 of
[https://github.com/django/django/blob/b1a4b1f0bdf05adbd3dc4dde14228e68da54c1a3/django/core/management/color.py#L16
django/code/management/color.py]. There, the function `colorama.init()` is
being called, and being called at import time (with caveats, see below).
My deployment setup uses Apache ''mod_wsgi'' with the setting
''WSGIRestrictStdout on'' (its recommended default). During import of the
logging system `colorama.init()` accesses `sys.stdout` which is
[https://modwsgi.readthedocs.io/en/master/configuration-
directives/WSGIRestrictStdout.html prohibited by this setting] causing a
dreaded 500 error.
Running code during import is [https://www.benkuhn.net/importtime/ best
avoided if possible]. I understand that `colorama.init()` must be called
at some point, but could it please be moved into a function instead?
Would the function `colour_style()` in the same module be suitable? Doing
so would make it possible for me use my logging configuration to avoid the
function call.
Some notes:
* My (production) logging configuration doesn't use colour, but the import
chain forces a run of `colorama.init()` anyway.
* A work-around would be simply not install colorama, as `init()` is only
called it's import succeeds. Unfortunately, colorama is being installed
into my project's virtualenv by a 3rd-party dependency, and I don't want
to drop that package. Also, I use colorama in the custom management
commands of a couple of my projects.
* I apologise for not testing my deployments during the 3.2 beta. I'm
kicking myself, because I did test upgrades of my main projects for the
beta and release candidates. I fixed all the deprecation warnings and made
sure my test suite passed - but I didn't try a test deployment. I fell
prey to the classic ''it works on my computer'' blunder!
--
Ticket URL: <https://code.djangoproject.com/ticket/32740>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Leon Matthews | Owner: nobody
Type: Uncategorized | Status: new
Component: Core (Other) | Version: 3.2
Keywords: apache mod_wsgi | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Leon Matthews):
* Attachment "django-ticket-32740.txt" added.
Traceback from server
Django
Reporter: Leon Matthews | Owner: nobody
Type: Uncategorized | Status: new
Component: Core (Other) | Version: 3.2
Keywords: apache mod_wsgi | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* cc: MinchinWeb (added)
Comment:
Hello again. I'm supposing we should fix this, but I'm a little intrigued
by the last line of that traceback:
{{{
File "/srv/websites/digitaladvisor.nz/env/lib/python3.8/site-
packages/colorama/ansitowin32.py", line 59, in closed
26 return stream.closed
}}}
The docs you link have this for justification:
> A well behaved Python WSGI application should never attempt to write any
data directly to sys.stdout or use the print statement without directing
it to an alternate file object.
But [https://docs.python.org/3.9/library/io.html#io.IOBase.closed `closed`
is just checking the status of the stream]. It's not writing data, so
shouldn't trigger an error. To that extent it kind of looks like a bug in
`mod_wsgi`. 🤔
> could it please be moved into a function instead?
An earlier iteration of the PR did have this call nested inside a
function.
--
Ticket URL: <https://code.djangoproject.com/ticket/32740#comment:1>
Django
Reporter: Leon Matthews | Owner: nobody
Type: Uncategorized | Status: new
Component: Core (Other) | Version: 3.2
Keywords: apache mod_wsgi | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carlton Gibson):
I've opened [https://github.com/GrahamDumpleton/mod_wsgi/issues/675 an
issue on `mod_wsgi`] to see what Graham's thoughts are there.
Even if we were to nest the call to `colorama.init()`, the same path would
be hit, just later, and there'd still be a compatibility issue with
`mod_wsgi`.
(I don't think that requiring the logging config to avoid the call is
particularly robust...)
--
Ticket URL: <https://code.djangoproject.com/ticket/32740#comment:2>
Django
Reporter: Leon Matthews | Owner: nobody
Type: Uncategorized | Status: new
Component: Core (Other) | Version: 3.2
Keywords: apache mod_wsgi | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carlton Gibson):
Mariusz suggested in conversation that we might catch the exception,
something like:
{{{
try:
import colorama
except ImportError:
HAS_COLORAMA = False
else:
try:
colorama.init()
except:
# Would we want to at least log something here?
HAS_COLORAMA = False
else:
HAS_COLORAMA = True
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32740#comment:3>
Django
Reporter: Leon Matthews | Owner: nobody
Component: Core (Other) | Version: 3.2
Keywords: apache mod_wsgi | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz):
* type: Uncategorized => Bug
* stage: Unreviewed => Accepted
Comment:
What about avoiding calling `colorama.init()` on non-Windows platforms?
--
Ticket URL: <https://code.djangoproject.com/ticket/32740#comment:4>
Django
Reporter: Leon Matthews | Owner: nobody
Component: Core (Other) | Version: 3.2
Keywords: apache mod_wsgi | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* severity: Normal => Release blocker
Comment:
[https://github.com/django/django/pull/14386 PR with the try-catch
thought...] — Not at all sure how best to test that. Any thoughts welcome.
> What about avoiding calling colorama.init() on non-Windows platforms?
Yes... maybe… not sure.
There's no reason to have colorama installed at all if you're not on
Windows. So I'd prefer to push back a bit before we do too much: why is it
getting installed at all?; why is mod_wsgi raising for just the `closed`
check?; what do colorama say on this, since presumably they'd like to be
safe on other platforms? 🤔
We can work around this, but I'm not sure it's our issue. It probably
counts as a Release Blocker if we're going to address it though.
--
Ticket URL: <https://code.djangoproject.com/ticket/32740#comment:5>
Django
Reporter: Leon Matthews | Owner: nobody
Component: Core (Other) | Version: 3.2
Keywords: apache mod_wsgi | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carlton Gibson):
Hi Leon.
[https://github.com/GrahamDumpleton/mod_wsgi/issues/675#issuecomment-840181867
Graham replied on `mod_wsgi`]:
> For now try setting:
> WSGIRestrictedStdout Off
He also points out there where `colorama` can catch the OSError
([https://github.com/tartley/colorama/issues/304 Matching colorama issue])
Can you give the PR on https://github.com/django/django/pull/14386 a go.
That should at least enable you to get past this with
`WSGIRestrictedStdout` still on.
We're looking into whether we can move the `colorama.init()` off the
import-time path. (It's meant to be a no-op on other platforms, but is
hitting this OSError issue from mod_wsgi...)
--
Ticket URL: <https://code.djangoproject.com/ticket/32740#comment:6>
Django
Reporter: Leon Matthews | Owner: nobody
Component: Core (Other) | Version: 3.2
Keywords: apache mod_wsgi | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Leon Matthews):
Replying to [comment:6 Carlton Gibson]:
> Can you give the PR on https://github.com/django/django/pull/14386 a go.
> That should at least enable you to get past this with
`WSGIRestrictedStdout` still on.
Thank you Carlton, I'm back at work now, so I'll check that out and get
back to you soon.
> We're looking into whether we can move the `colorama.init()` off the
import-time path.
I agree that that would be cleanest approach. It would be entirely
reasonable to hit this error if my production configuration used the
feature (as my development configuration does).
--
Ticket URL: <https://code.djangoproject.com/ticket/32740#comment:7>
Django
Reporter: Leon Matthews | Owner: nobody
Component: Core (Other) | Version: 3.2
Keywords: apache mod_wsgi | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Leon Matthews):
Replying to [comment:6 Carlton Gibson]:
> Can you give the PR on https://github.com/django/django/pull/14386 a go.
> That should at least enable you to get past this with
`WSGIRestrictedStdout` still on.
That seems to have done the trick!
BTW, it's a moot point now but I found out why ''colorama'' was getting
installed in the first place. It wasn't a 3rd-party dependency like I
thought it was. It turns out that Ubuntu 20.04 LTS installs a bunch of
[https://bugs.launchpad.net/ubuntu/+source/python-virtualenv/+bug/1904945
libraries into every Python virtualenv by default].
Thank you so much for your work on this Carlton, it's very much
appreciated!
--
Ticket URL: <https://code.djangoproject.com/ticket/32740#comment:8>
Django
Reporter: Leon Matthews | Owner: Carlton
| Gibson
Type: Bug | Status: assigned
Component: Core (Other) | Version: 3.2
Keywords: apache mod_wsgi | Triage Stage: Ready for
| checkin
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* owner: nobody => Carlton Gibson
* status: new => assigned
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/32740#comment:9>
Django
Reporter: Leon Matthews | Owner: Carlton
| Gibson
Type: Bug | Status: assigned
Keywords: apache mod_wsgi | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/32740#comment:10>
Django
Reporter: Leon Matthews | Owner: Carlton
| Gibson
Component: Core (Other) | Version: 3.2
Keywords: apache mod_wsgi | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson <carlton.gibson@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"c2e6047c725e26987c87e2be59f2ab4bf9828fa5" c2e6047c]:
{{{
#!CommitTicketReference repository=""
revision="c2e6047c725e26987c87e2be59f2ab4bf9828fa5"
Fixed #32740 -- Caught possible exception when initializing colorama.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32740#comment:11>
Django
Reporter: Leon Matthews | Owner: Carlton
| Gibson
Type: Bug | Status: closed
Keywords: apache mod_wsgi | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carlton Gibson <carlton.gibson@…>):
In [changeset:"a173202dd475a454164bf4eeb528f183a85481c5" a173202d]:
{{{
#!CommitTicketReference repository=""
revision="a173202dd475a454164bf4eeb528f183a85481c5"
[3.2.x] Fixed #32740 -- Caught possible exception when initializing
colorama.
Backport of c2e6047c725e26987c87e2be59f2ab4bf9828fa5 from main
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32740#comment:12>