[Django] #23317: StaticFileStorage Encoding error on '\xf4'
Django
--------------------------------------+--------------------------------
Reporter: achedeuzot | Owner: nobody
Type: Bug | Status: new
Component: File uploads/storage | Version: 1.7-rc-2
Severity: Normal | Keywords: UnicodeEncodeError
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+--------------------------------
While using the `collectstatic` command with `django-pipeline` and a i18n
js library (containing lots of foreign characters), I stumbled on an
encoding error:
{{{
Traceback (most recent call last):
File "./dev_manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-packages/django/core/management/__init__.py", line 385, in
execute_from_command_line
utility.execute()
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-packages/django/core/management/__init__.py", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **options.__dict__)
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-packages/django/core/management/base.py", line 337, in execute
output = self.handle(*args, **options)
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-packages/django/core/management/base.py", line 532, in handle
return self.handle_noargs(**options)
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-
packages/django/contrib/staticfiles/management/commands/collectstatic.py",
line 168, in handle_noargs
collected = self.collect()
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-
packages/django/contrib/staticfiles/management/commands/collectstatic.py",
line 114, in collect
for original_path, processed_path, processed in processor:
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-packages/pipeline/storage.py", line 32, in post_process
packager.pack_stylesheets(package)
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-packages/pipeline/packager.py", line 94, in pack_stylesheets
variant=package.variant, **kwargs)
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-packages/pipeline/packager.py", line 105, in pack
self.save_file(output_filename, content)
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-packages/pipeline/packager.py", line 117, in save_file
return self.storage.save(path, ContentFile("\xf4"))
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-packages/django/core/files/storage.py", line 51, in save
name = self._save(name, content)
File
"/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4
/site-packages/django/core/files/storage.py", line 228, in _save
_file.write(chunk)
UnicodeEncodeError: 'ascii' codec can't encode character '\xf4' in
position 0: ordinal not in range(128)
}}}
After some enquiries, the same error could be reproduced using the
following lines:
{{{
from django.contrib.staticfiles.storage import StaticFilesStorage
from django.core.files.base import ContentFile
storage = StaticFilesStorage()
storage.save("test-output.txt", ContentFile("ô"))
}}}
I was able to "solve" the issue by using `bytearray`:
{{{
storage.save("test-output.txt", ContentFile(bytearray("ô", 'utf-8')))
}}}
Also, another solution is/was to add `LANG=en_US.utf-8` into the
environment.
The line that trows the error is the following:
{{{
mode = 'wb' if isinstance(chunk, bytes) else 'wt'
_file = os.fdopen(fd, mode)
_file.write(chunk) # <= this one
}}}
We can see that if I send bytes, the write() happens without errors while
if I send a string, the write() fails.
How come Django fails on a "standard" utf-8 string ? Is this a Django
issue or from something else (OS write(), ...)?
--
Ticket URL: <https://code.djangoproject.com/ticket/23317>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: achedeuzot | Owner: nobody
Type: Bug | Status: new
uploads/storage | Resolution:
Severity: Normal | Triage Stage:
Keywords: UnicodeEncodeError | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by areski):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
I cannot reproduce this bug, I tested the following with Python2.7 and
Python3.4:
{{{
import os, django
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
django.setup()
from django.contrib.staticfiles.storage import StaticFilesStorage
from django.core.files.base import ContentFile
storage = StaticFilesStorage()
storage.save("test-output.txt", ContentFile("ô"))
}}}
I work on Ubuntu14 and by default LANG is set, I tried to unset it but no
luck.
--
Ticket URL: <https://code.djangoproject.com/ticket/23317#comment:1>
Django
Reporter: achedeuzot | Owner: nobody
Type: Bug | Status: new
uploads/storage | Resolution:
Severity: Normal | Triage Stage:
Keywords: UnicodeEncodeError | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by achedeuzot):
Additional info:
- MacOSX 10.9.4
- Python 3.4.1
- Env: no LANG nor LC_ALL nor LC_CTYPE (set but empty)
- Django 1.7r2
Thanks for your update @areski !
--
Ticket URL: <https://code.djangoproject.com/ticket/23317#comment:2>
Django
Reporter: achedeuzot | Owner: nobody
Type: Bug | Status: closed
Component: File | Version: 1.7-rc-2
uploads/storage | Resolution: invalid
Severity: Normal | Triage Stage:
Keywords: UnicodeEncodeError | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* resolution: => invalid
Comment:
I think it's a problem with your configuration, see
[https://docs.djangoproject.com/en/dev/faq/troubleshooting/#i-m-getting-a
-unicodedecodeerror-what-am-i-doing-wrong this FAQ entry].
--
Ticket URL: <https://code.djangoproject.com/ticket/23317#comment:3>