[Django] #23265: Django don't start on Windows with Python 2.7 (but only in august!)
Django
--------------------------------------------+--------------------
Reporter: SpaceFox | Owner: nobody
Type: Uncategorized | Status: new
Component: Core (Management commands) | Version: 1.6
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------------+--------------------
On Windows / Python 2.7, if Django is set to use the French locale
("fra"), an UTF8 encoding error prevents it to start with this stack
trace:
{{{
Validating models...
0 errors found
Unhandled exception in thread started by <function wrapper at
0x0000000003F33B38>
Traceback (most recent call last):
File "C:\Users\SpaceFox\.virtualenvs\zdsenv\lib\site-
packages\django\utils\autoreload.py", line 93, in wrapper
fn(*args, **kwargs)
File "C:\Users\SpaceFox\.virtualenvs\zdsenv\lib\site-
packages\django\core\management\commands\runserver.py", line 104,
in inner_run
now = now.decode('utf-8')
File "C:\Users\SpaceFox\.virtualenvs\zdsenv\lib\encodings\utf_8.py",
line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfb in position 2:
invalid start byte
}}}
How to reproduce this:
- Windows (tested on Windows 8.1 x64), all updates OK
- Python 2.7 (Python 2.7.8 (default, Jun 30 2014, 16:08:48) [MSC v.1500 64
bit (AMD64)] on win32)
- Django 1.6.5
- In settings.py, set locale to "fra": {{{locale.setlocale(locale.LC_TIME,
'fra')}}}
- Try to launch Django with {{{python manage.py runserver}}}. Kabooom!
The problem comes for the django/core/management/commands/runserver.py
line 102 to 104 :
{{{
now = datetime.now().strftime('%B %d, %Y - %X')
if six.PY2:
now = now.decode('utf-8')
}}}
These 3 lines suppose the strftime method returns an UTF-8 string... this
is false when Django runs on Windows with Python 2.7.
In French language, 3 month names have non-ASCII characters: "février",
"août" and "décembre" (February, August and December).
With described parameters, {{{now}}} is set as "ao¹t 09, 2014 - 00:56:41".
The crap that replaces the "û" character in "août" contains the 0xfb
character, which is illegal in UTF-8. This create the crash.
The problem is the same in February and December due to the 0xe9 byte in
{{{now}}}, caused by the "é" character.
With other non-ASCII characters, this may "work" as long as there is no
illegal byte in the given string: the output of this function will be
broken (not what is expected) but there will be no exception thrown.
I don't know how to correct this, therefore I can't provide any patch.
--
Ticket URL: <https://code.djangoproject.com/ticket/23265>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
August and December!)
-------------------------------------+-------------------------------------
Reporter: SpaceFox | Owner: nobody
Type: Uncategorized | Status: new
commands) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by SpaceFox):
* needs_better_patch: => 0
* needs_docs: => 0
* needs_tests: => 0
Comment:
Replying to [ticket:23265 SpaceFox]:
--
Ticket URL: <https://code.djangoproject.com/ticket/23265#comment:1>
Django
August and December!)
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
commands) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by aaugustin):
This problem was introduced in [cb1614f7b30f336db2a807b43696e20fdab7b78c]
to implement feature request #18611. It affects Django 1.5+.
A first attempt at fixing it was made in #21358 for Django 1.6+ but it was
incomplete.
Apparently we should use CP1252 for decoding in that case:
{{{
>>> print 'ao\xfbt'.decode('cp1252')
août
}}}
Which raises the more general question of what encoding to use. The best
answer I found was on StackOverflow:
http://stackoverflow.com/questions/19412915/how-determine-encoding-of-
datetime-strftime-in-python
Can you provide the output of `locale.getlocale(locale.LC_TIME)` and of
`locale.getpreferredencoding()` on your system?
--
Ticket URL: <https://code.djangoproject.com/ticket/23265#comment:2>
Django
August and December!)
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
commands) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by SpaceFox):
Replying to [comment:2 aaugustin]:
> Can you provide the output of `locale.getlocale(locale.LC_TIME)` and of
`locale.getpreferredencoding()` on your system?
{{{
>>> locale.getlocale(locale.LC_TIME)
('fr_FR', 'cp1252')
>>> locale.getpreferredencoding()
'cp1252'
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23265#comment:3>
Django
August and December!)
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
commands) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
I'd suggest to use the `django.utils.encoding.get_system_encoding`
function instead of the hardcoded 'utf-8'.
--
Ticket URL: <https://code.djangoproject.com/ticket/23265#comment:4>
Django
August and December!)
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
commands) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
SpaceFox, could you test if this patch solves your issue?
{{{
diff --git a/django/core/management/commands/runserver.py
b/django/core/management/commands/runserver.py
index 503cff2..dfff57d 100644
--- a/django/core/management/commands/runserver.py
+++ b/django/core/management/commands/runserver.py
@@ -11,6 +11,7 @@ import socket
from django.core.management.base import BaseCommand, CommandError
from django.core.servers.basehttp import run,
get_internal_wsgi_application
from django.utils import autoreload
+from django.utils.encoding import get_system_encoding
from django.utils import six
naiveip_re = re.compile(r"""^(?:
@@ -101,7 +102,7 @@ class Command(BaseCommand):
self.validate(display_num_errors=True)
now = datetime.now().strftime('%B %d, %Y - %X')
if six.PY2:
- now = now.decode('utf-8')
+ now = now.decode(get_system_encoding())
self.stdout.write((
"%(started_at)s\n"
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23265#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: SpaceFox | Owner: nobody
Type: Bug | Status: new
Component: Core (Management | Version: 1.6
commands) | Resolution:
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by timgraham):
* type: Uncategorized => Bug
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/23265#comment:6>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Core (Management | Version: 1.6
commands) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by SpaceFox):
claudep,
The server starts with your patch and works perfectly.
The only detail is that now the server tries to process UTF-8 characters
as cp1252, which displays strange characters in Windows Powershell:
ao├╗t 13, 2014 - 19:56:34
I don't know if this detail is a problem for Django Team - this is OK for
me.
--
Ticket URL: <https://code.djangoproject.com/ticket/23265#comment:7>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Core (Management | Version: 1.6
commands) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by claudep):
Thanks for checking.
I think the output problem is another issue, probably due to
`OutputWrapper.write` using `force_str` with default encoding of `utf-8`.
In that case, using `get_system_encoding` might be more problematic as the
output may be redirected to some file where the desired encoding might be
still `utf-8`. I'd like to get more opinions about the output encoding of
non-ASCII content from management commands on Windows.
But I plan to fix ASAP the decoding issue originally reported.
--
Ticket URL: <https://code.djangoproject.com/ticket/23265#comment:8>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Core (Management | Version: 1.6
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"055d95fce0668e11f2dae48d2439f378349d2524"]:
{{{
#!CommitTicketReference repository=""
revision="055d95fce0668e11f2dae48d2439f378349d2524"
Fixed #23265 -- Used system-specific encoding in runserver
Thanks SpaceFox for the report.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23265#comment:9>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Core (Management | Version: 1.6
commands) | Resolution: fixed
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by Claude Paroz <claude@…>):
In [changeset:"63ccf64079f833ff93a3fe9d158df2ec99015147"]:
{{{
#!CommitTicketReference repository=""
revision="63ccf64079f833ff93a3fe9d158df2ec99015147"
[1.7.x] Fixed #23265 -- Used system-specific encoding in runserver
Thanks SpaceFox for the report.
Backport of 055d95fce066 from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23265#comment:10>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Core (Management | Version: 1.6
commands) | Resolution: fixed
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by Claude Paroz <claude@…>):
In [changeset:"99b5567796b2a4b96547a51e630f8c5b23d78531"]:
{{{
#!CommitTicketReference repository=""
revision="99b5567796b2a4b96547a51e630f8c5b23d78531"
[1.6.x] Fixed #23265 -- Used system-specific encoding in runserver
Thanks SpaceFox for the report.
Backport of 055d95fce066 from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23265#comment:11>