TypeError: decoding Unicode is not supported

[sohesado]

Hi

I'm currently developing a django project and i've encountered a
problem regarding Unicode string manipulation.

There is a backend python module in my project that perform's mainly
shutil (shell utilities) tasks.

I get an TypeError: "decoding Unicode is not supported" exception
while running the following method

---------------------------------------------------------------------------------------------------------
def create_base(name, reg):
path = unicode(ROOT + name, 'utf-8') <<<< error at this line
shutil.copytree(ROOT + 'matrix', path)
shutil.copystat(ROOT + 'matrix', path)

f = codecs.open(path + '/conf/local.php', "w", "UTF-8")
tmp = unicode("<?php\n$conf['title'] = " + name + ";\n", 'utf-8')
f.write(tmp)
if not reg:
f.write("$conf['disableactions'] = 'register';\n")
-------------------------------------------------------------------------------------------------------

The weird thing that puzzles me is that when i test the module in a
python shell , the method runs flawlessly which yields me to the
assumption that it's a django thing.
By the way i use apache+mod-python as a web-server

I can't get adequate information regarding this type of exception. I'm
stuck , please help.

[Alec Shaner]

unicode gives me nightmares.

Taking django out of the picture for a moment:
>>> unicode('foo', 'utf-8')
u'foo'
>>> unicode(u'foo', 'utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>

TypeError: decoding Unicode is not supported

So I would assume when you run it inside django you're already using a unicode string. Do you need to pass the encoding argument ('utf-8') to unicode?

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[sohesado]

Well your example clarifies the problem. I removed the unicode calls.

The initial problem was an UnicodeEncodeError exception. I tried to
solve it by encoding the strings to utf-8. It seems that's not the
case.
So i'm at square one.

Note that..
-with python manage.py runserver the app runs flawlessly.
-With apache+mod-python i get an UnicodeEncodeError.

So , it is an apache localization issue?

My /etc/apache2/envvars contains

------------------------------------------------------------------------------------------------
export APACHE_RUN_USER=www-data
export APACHE_RUN_GROUP=www-data
export APACHE_PID_FILE=/var/run/apache2.pid

## The locale used by some modules like mod_dav
export LANG='el_GR.UTF8' <- also tried el_GR.UTF-8 variation
export LC_ALL='el_GR.UTF8'
## Uncomment the following line to use the system default locale
instead:
#. /etc/default/locale

export LANG
---------------------------------------------------------------------------------------------------------

Have you got any clues what may cause the problem?

Thank you for your help.

On Aug 3, 11:31 pm, Alec Shaner <asha...@chumpland.org> wrote:
> unicode gives me nightmares.
>
> Taking django out of the picture for a moment:>>> unicode('foo', 'utf-8')
> u'foo'
> >>> unicode(u'foo', 'utf-8')
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: decoding Unicode is not supported
>
> So I would assume when you run it inside django you're already using a
> unicode string. Do you need to pass the encoding argument ('utf-8') to
> unicode?
>

> > django-users...@googlegroups.com<django-users%2Bunsu...@googlegroups.com>
> > .

[Alec Shaner]

I'm no expert on encodings so I can only suggest some alternatives that might get around the issue:

First, maybe you can find something here: http://docs.python.org/howto/unicode

You could try the literal method of generating a unicode string:

path = u'%s%s' % (ROOT, name)

Where exactly do you get the UnicodeEncodeError exception when you remove the unicode calls?

To unsubscribe from this group, send email to django-users...@googlegroups.com.

[sohesado]

On Aug 4, 6:48 pm, Alec Shaner <asha...@chumpland.org> wrote:
> I'm no expert on encodings so I can only suggest some alternatives that
> might get around the issue:
>

> First, maybe you can find something here:http://docs.python.org/howto/unicode

i've read , pretty much everything, available at docs.python.org
regarding unicode.

> You could try the literal method of generating a unicode string:
>
> path = u'%s%s' % (ROOT, name)

tried it. The problem persists

> Where exactly do you get the UnicodeEncodeError exception when you remove
> the unicode calls?

I get the the exception at this line -> shutil.copytree(ROOT +
'matrix', path)

And the traceback...

Traceback:
File "/usr/local/lib/python2.6/dist-packages/django/core/handlers/
base.py" in get_response
100. response = callback(request,
*callback_args, **callback_kwargs)
File "/usr/local/lib/python2.6/dist-packages/django/contrib/auth/
decorators.py" in _wrapped_view
25. return view_func(request, *args, **kwargs)
File "/home/sohesado/codein/wikitool/creators/views.py" in create_wiki
42. doku_script.create_wiki(w.name, tmpl,
request.POST['access_policy'], reg)
File "/home/sohesado/codein/wikitool/creators/doku_script.py" in
create_wiki
10. create_base(name, reg)
File "/home/sohesado/codein/wikitool/creators/doku_script.py" in
create_base
16. shutil.copytree(ROOT + u'matrix', path)
File "/usr/lib/python2.6/shutil.py" in copytree
146. os.makedirs(dst)
File "/usr/lib/python2.6/os.py" in makedirs
157. mkdir(name, mode)

Exception Type: UnicodeEncodeError at /wikitool/creators/create_wiki/
Exception Value: 'ascii' codec can't encode characters in position
19-25: ordinal not in range(128)

> > > > django-users...@googlegroups.com<django-users%2Bunsubscribe@google groups.com>
> > <django-users%2Bunsu...@googlegroups.com<django-users%252Bunsubscribe@g ooglegroups.com>

>
> > > > .
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/django-users?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Django users" group.
> > To post to this group, send email to django...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > django-users...@googlegroups.com<django-users%2Bunsubscribe@google groups.com>

[Alec Shaner]

Ah, maybe that makes more sense then. It does indeed sound like locale issue with your apache + mod_python envrionment. I see an old django ticket that sounds pretty similar:

http://code.djangoproject.com/ticket/8965

Sounds like you're still getting ASCII as the default encoding despite the env settings in apache?

Out of curiosity I created a view to dump the encoding in my django dev app by calling locale.getpreferredencoding(). In both apache+mod_wsgi and django development server it returns UTF-8. My linux box is configured with UTF-8 as the default, so I haven't done anything special with apache. If you put that in your code and it has UTF-8 in both cases then this issue is beyond me.

To unsubscribe from this group, send email to django-users...@googlegroups.com.

[sohesado]

i've found the solution. There was missing some language support
packages. It's kind of embarrassing :)...

My initial apache configuration was ok. The fact that manage.py
runserver just worked , was misleading.

Thanks for your help , Alec. I've learned a couple of things dealing
with this issue.