'ascii' codec can't encode character u'\xe4'

[kenneth gonsalves]

hi,

I have an application which runs on a webfaction hosting. I also have
the same app running on my devel machine. The code in the two machines
are pulled from the same repo. Django versions are identical. Python
versions are identical. I try to upload a file called Lisäinfo
kummeille.pdf in the dev machine. No problem. When I try the same file
in webfaction, I get the above error. I have checked and find I can use
python to open and save a file of the same name from the python prompt
in the webfaction shell. In templates, admin and and generated pdfs,
these non ascii characters are handled perfectly. Has anyone got any
clues about this? The full traceback is given below:

Environment:


Request Method: POST
Request URL: http://kenyakids.info/admin/web/download/add/

Django Version: 1.4a1
Python Version: 2.7.3
Installed Applications:
('django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.admin',
'django.contrib.staticfiles',
'sorl.thumbnail',
'south',
'kenyakids.web',
'modeltranslation',
'rosetta')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.middleware.doc.XViewMiddleware',
'django.contrib.messages.middleware.MessageMiddleware')


Traceback:
File
"/home/kenyakids/webapps/django/lib/python2.7/django/core/handlers/base.py" in get_response
111. response = callback(request,
*callback_args, **callback_kwargs)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/contrib/admin/options.py" in wrapper
367. return self.admin_site.admin_view(view)(*args,
**kwargs)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/utils/decorators.py" in _wrapped_view
91. response = view_func(request, *args, **kwargs)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/views/decorators/cache.py" in _wrapped_view_func
88. response = view_func(request, *args, **kwargs)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/contrib/admin/sites.py" in inner
192. return view(request, *args, **kwargs)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/utils/decorators.py" in _wrapper
25. return bound_func(*args, **kwargs)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/utils/decorators.py" in _wrapped_view
91. response = view_func(request, *args, **kwargs)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/utils/decorators.py" in bound_func
21. return func(self, *args2, **kwargs2)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/db/transaction.py"
in inner
209. return func(*args, **kwargs)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/contrib/admin/options.py" in add_view
955. self.save_model(request, new_object, form, False)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/contrib/admin/options.py" in save_model
709. obj.save()
File
"/home/kenyakids/webapps/django/lib/python2.7/django/db/models/base.py"
in save
464. self.save_base(using=using, force_insert=force_insert,
force_update=force_update)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/db/models/base.py"
in save_base
552. result = manager._insert([self], fields=fields,
return_id=update_pk, using=using, raw=raw)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/db/models/manager.py" in _insert
203. return insert_query(self.model, objs, fields, **kwargs)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/db/models/query.py"
in insert_query
1576. return
query.get_compiler(using=using).execute_sql(return_id)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/db/models/sql/compiler.py" in execute_sql
909. for sql, params in self.as_sql():
File
"/home/kenyakids/webapps/django/lib/python2.7/django/db/models/sql/compiler.py" in as_sql
872. for obj in self.query.objs
File
"/home/kenyakids/webapps/django/lib/python2.7/django/db/models/fields/files.py" in pre_save
249. file.save(file.name, file, save=False)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/db/models/fields/files.py" in save
86. self.name = self.storage.save(name, content)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/core/files/storage.py" in save
44. name = self.get_available_name(name)
File
"/home/kenyakids/webapps/django/lib/python2.7/django/core/files/storage.py" in get_available_name
70. while self.exists(name):
File
"/home/kenyakids/webapps/django/lib/python2.7/django/core/files/storage.py" in exists
230. return os.path.exists(self.path(name))
File "/usr/local/lib/python2.7/genericpath.py" in exists
18. os.stat(path)

Exception Type: UnicodeEncodeError at /admin/web/download/add/
Exception Value: 'ascii' codec can't encode character u'\xe4' in
position 71: ordinal not in range(128)


--
regards
Kenneth Gonsalves

[bruno desthuilliers]

On Thursday, June 14, 2012 12:35:40 PM UTC+2, lawgon wrote:

hi,

I have an application which runs on a webfaction hosting.

How is it deployed exactly ?
(sorry never used webfaction hosting for django - nor for anything else FWIW)
 

I also have
the same app running on my devel machine. The code in the two machines
are pulled from the same repo. Django versions are identical. Python
versions are identical. I try to upload a file called Lisäinfo
kummeille.pdf in the dev machine. No problem. When I try the same file
in webfaction, I get the above error. I have checked and find I can use
python to open and save a file of the same name from the python prompt
in the webfaction shell. In templates, admin and and generated pdfs,
these non ascii characters are handled perfectly. Has anyone got any
clues about this?

Each time I had this problem, it happened that the server process was using Apache's default locale, and that Apache default's locale was "C", not the utf8 locale configured for users. Not much of a problem for me since I have admin access but that may not be the case with webfaction :-/

HTH

[kenneth gonsalves]

On Thu, 2012-06-14 at 04:19 -0700, bruno desthuilliers wrote:
> > I have an application which runs on a webfaction hosting.
>
>
>
> How is it deployed exactly ?

apache + wsgi

> (sorry never used webfaction hosting for django - nor for anything
> else
> FWIW)

not my choice, client choice

>
>
> > I also have
> > the same app running on my devel machine. The code in the two
> machines
> > are pulled from the same repo. Django versions are identical.
> Python
> > versions are identical. I try to upload a file called Lisäinfo
> > kummeille.pdf in the dev machine. No problem. When I try the same
> file
> > in webfaction, I get the above error. I have checked and find I can
> use
> > python to open and save a file of the same name from the python
> prompt
> > in the webfaction shell. In templates, admin and and generated
> pdfs,
> > these non ascii characters are handled perfectly. Has anyone got
> any
> > clues about this?
>
>
>
>
> Each time I had this problem, it happened that the server process was
> using
> Apache's default locale, and that Apache default's locale was "C", not
> the
> utf8 locale configured for users. Not much of a problem for me since I
> have
> admin access but that may not be the case with webfaction :-/

I think that should be it - my dev server runs nginx (and on my own
server I have root and anyway do not use apache.

--
regards
Kenneth Gonsalves

[bruno desthuilliers]

On Thursday, June 14, 2012 1:58:28 PM UTC+2, lawgon wrote:

On Thu, 2012-06-14 at 04:19 -0700, bruno desthuilliers wrote:
> > I have an application which runs on a webfaction hosting.
>
>
>
> How is it deployed exactly ?

apache + wsgi

Ok, so that's the same setup we use... Are you/they using the WSGIDaemonProcess and WSGIProcessGroup directives ? (I bet they do if it's shared hosting and they know what's good for them). But anyway: I did not test the damn thing thoroughly but IIRC the problem appeared whether running daemon or embedded mode - the django process env is stil inherited from the Apache process. Using os.setenv in the .wsgi script is _supposed_ to cure the problem but our experience (here again I did not have time to really explore the problem :-/) is that it often doesn't work (can't even tell if it ever worked at all), so we alway end up editing apache conf - which _is_ a know working solution.

Your best bet here is to first try the os.setenv trick, and if it doesn't solve the problem contact webfaction and see what you / they can do.

HTH (and please report the solution)

 

[Paulo Almeida]

Hi,

Have you tried using SetEnv directives in .htaccess to change the locale? I'm not sure how that would interact with wsgi, but maybe something like this:

http://drumcoder.co.uk/blog/2010/nov/12/apache-environment-variables-and-mod_wsgi/

Of course you'd have to do it in .htaccess instead of Virtual Hosts.

Paulo

[Hendrik Speidel]

Hi,

as an alternative you can provide your own Storage implementation like
outlined here:

http://source.mihelac.org/2011/02/6/rename-uploaded-files-ascii-character-set-django/

Note: you may also need to override get_available_name to get it working
if i remember correctly.

hendrik

On 06/14/2012 05:38 PM, Paulo Almeida wrote:
> Hi,
>
> Have you tried using SetEnv directives in .htaccess to change the
> locale? I'm not sure how that would interact with wsgi, but maybe
> something like this:
>
> http://drumcoder.co.uk/blog/2010/nov/12/apache-environment-variables-and-mod_wsgi/
>
> Of course you'd have to do it in .htaccess instead of Virtual Hosts.
>
> Paulo
>
> On Thursday, June 14, 2012 11:35:40 AM UTC+1, lawgon wrote:
>
> hi,
>
> I have an application which runs on a webfaction hosting. I also have
> the same app running on my devel machine. The code in the two machines
> are pulled from the same repo. Django versions are identical. Python

> versions are identical. I try to upload a file called Lis�info

> 249. file.save(file.name <http://file.name>, file,

> save=False)
> File
> "/home/kenyakids/webapps/django/lib/python2.7/django/db/models/fields/files.py"
> in save

> 86. self.name <http://self.name> = self.storage.save(name,

> content)
> File
> "/home/kenyakids/webapps/django/lib/python2.7/django/core/files/storage.py"
> in save
> 44. name = self.get_available_name(name)
> File
> "/home/kenyakids/webapps/django/lib/python2.7/django/core/files/storage.py"
> in get_available_name
> 70. while self.exists(name):
> File
> "/home/kenyakids/webapps/django/lib/python2.7/django/core/files/storage.py"
> in exists
> 230. return os.path.exists(self.path(name))
> File "/usr/local/lib/python2.7/genericpath.py" in exists
> 18. os.stat(path)
>
> Exception Type: UnicodeEncodeError at /admin/web/download/add/
> Exception Value: 'ascii' codec can't encode character u'\xe4' in
> position 71: ordinal not in range(128)
>
>
> --
> regards
> Kenneth Gonsalves
>

> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/django-users/-/3B3Y--yCilsJ.
> To post to this group, send email to django...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.

[kenneth gonsalves]

On Thu, 2012-06-14 at 23:46 +0200, Hendrik Speidel wrote:
> as an alternative you can provide your own Storage implementation like
> outlined here:
>
> http://source.mihelac.org/2011/02/6/rename-uploaded-files-ascii-character-set-django/
>
>

I have told the client to use only ascii for the time being - but that
is not a solution - for example assume that the same files are
accessed/uploaded by other applications.
--
regards
Kenneth Gonsalves

[Xavier Ordoquy]

Hi there,

I have faced this kind of error a couple of times when the local wasn't correctly set on the system.
Within the python interpreter it was C (ie ascii) where it should have been UTF8.
Up to now, all have been resolved by correctly setting the locals system wide and within the web server (see apache setenv as mentioned in earlier post).

Regards,
Xavier Ordoquy,
Linovia.

> --
> You received this message because you are subscribed to the Google Groups "Django users" group.

[kenneth gonsalves]

On Thu, 2012-06-14 at 08:26 -0700, bruno desthuilliers wrote:
> > apache + wsgi
> >
>
>
> Ok, so that's the same setup we use... Are you/they using the
> WSGIDaemonProcess and WSGIProcessGroup directives ? (I bet they do if
> it's
> shared hosting and they know what's good for them).

yes

> But anyway: I did not
> test the damn thing thoroughly but IIRC the problem appeared whether
> running daemon or embedded mode - the django process env is stil
> inherited
> from the Apache process. Using os.setenv in the .wsgi script is
> _supposed_
> to cure the problem but our experience (here again I did not have time
> to
> really explore the problem :-/) is that it often doesn't work (can't
> even
> tell if it ever worked at all), so we alway end up editing apache conf
> -
> which _is_ a know working solution.

they give users a small snippet of httpd.conf which we can edit - I put
this line:

SetEnv Locale UTF8

and it is working. Thanks for the hints, I now remember that I had
solved this problem long ago, but it is so long since I have used
apache, that i had forgotten
--
regards
Kenneth Gonsalves