UnicodeDecodeError in template rendering

[cesco]

Hi,

I'm using sqlite3 as db for my django project.
In order to load the data in the db I first generated a unicode string
(which contains danish characters "ø, æ, å") and dumped such a string
to a file (in json format) as follows:
f_hdl = file(json_offers_path, 'w')
f_hdl.write(offer_entry.encode('utf-8'))
f_hdl.close()

I then loaded the data in the database using the command:
python manage.py loaddata path/to/json/file

I could run the local server for development without any problem. In
the homepage I have to display some information that may contain
danish characters. If the page I have to display doesn't contain
danish characters I don't get any error.
If it does contain them I get the following error:

Exception Type: UnicodeEncodeError
Exception Value: 'ascii' codec can't encode character u'\xe6' in
position 27: ordinal not in range(128)
Exception Location: C:\Python25\lib\site-packages\django\utils
\encoding.py in force_unicode, line 39
Unicode error hint: The string that could not be encoded/decoded was:
roanlæg

Do you have any suggestion on how to solve the problem?

Many thanks
Francesco

[Kai Kuehne]

Hi,

Do you use __str__ in your models.py?
If yes, change it to __unicode__ and it should work.

> Many thanks
> Francesco

Kai

[Malcolm Tredinnick]

The full traceback would be more useful than just the last line, since
that will explain where you are coming from to get to this point. What
is most important to know is (a) what is the data that it is trying to
render (both type and content). Try to find where you load that data and
print out it's type and repr() -- don't print str(), since that
disguises problems by calling conversion function, (b) what piece of
code is loading this data or trying to display it.

You seem to be suggesting that you don't see the problem with the
development server but in some other set up you do see it. What are the
details of this other setup? What else has changed between when you
didn't see the error and when it started to appear? For example, have
you changed the database backend?

Malcolm

--
Many are called, few volunteer.
http://www.pointy-stick.com/blog/

[cesco]

Thanks for your replies.

Just to avoid any doubt, I'm using __unicode__ instead of __str__ and
I'm using the latest development version of django (post unicode
merge).

> What is most important to know is (a) what is the data that it is
> trying to render (both type and content).

It's trying to render an object of the class
'my_project.offers.models.Offer', which contains, among other fields,
a "name" field (that is models.CharField, which in some cases contains
danish characters) and a picture field (that is models.ImageField).

In the page I'm trying to render I display
1. a thumbnail of such image using the patch from http://code.djangoproject.com/ticket/4115
which is located in django.contrib.thumbnails.
2. the name of the picture/offer which may contain danish characters.
Every time I reload the page, 6 offers objects are pulled randomly
from all the available ones. If those objects contain a danish
character in their name (like 'æ') then I get the UnicodeEncodeError
(please, note the previous title of this post was UnicodeDecodeError
while it's actually a UnicodeEncodeError). If the objects don't
contain any danish character then I don't get any problem.
If I print to a file the repr() of the object then I get the right
danish characters. Only in the template rendering for some reason
there is this problem.

I'm using sqlite3 with the development server and python 2.5

Following are the information I had given before (which I repeat here
for completeness) and the full traceback I get from the template
error.

Exception Type: UnicodeEncodeError
Exception Value: 'ascii' codec can't encode character u'\xe6' in
position 27: ordinal not in range(128)
Exception Location: C:\Python25\lib\site-packages\django\utils
\encoding.py in force_unicode, line 39
Unicode error hint: The string that could not be encoded/decoded was:
roanlæg

Traceback (most recent call last):
File "C:\Python25\lib\site-packages\django\template\__init__.py" in
render_node
754. result = node.render(context)
File "C:\Python25\lib\site-packages\django\template\defaulttags.py" in
render
134. nodelist.append(node.render(context))
File "C:\Python25\lib\site-packages\django\template\defaulttags.py" in
render
134. nodelist.append(node.render(context))
File "C:\Python25\lib\site-packages\django\template\loader_tags.py" in
render
96. return self.template.render(context)
File "C:\Python25\lib\site-packages\django\template\__init__.py" in
render
181. return self.nodelist.render(context)
File "C:\Python25\lib\site-packages\django\template\__init__.py" in
render
739. return ''.join([force_unicode(b) for b in bits])
File "C:\Python25\lib\site-packages\django\utils\encoding.py" in
force_unicode
39. s = unicode(str(s), encoding, errors)

UnicodeEncodeError at /

'ascii' codec can't encode character u'\xe6' in position 27: ordinal
not in range(128)

I'm struggling with this problem from quite some time. Any help would
be very appreciated.

Thanks
Francesco

[Malcolm Tredinnick]

On Sun, 2007-08-19 at 00:27 -0700, cesco wrote:
> Thanks for your replies.
>
> Just to avoid any doubt, I'm using __unicode__ instead of __str__ and
> I'm using the latest development version of django (post unicode
> merge).
>
> > What is most important to know is (a) what is the data that it is
> > trying to render (both type and content).
>
> It's trying to render an object of the class
> 'my_project.offers.models.Offer', which contains, among other fields,
> a "name" field (that is models.CharField, which in some cases contains
> danish characters) and a picture field (that is models.ImageField).
>
> In the page I'm trying to render I display
> 1. a thumbnail of such image using the patch from http://code.djangoproject.com/ticket/4115
> which is located in django.contrib.thumbnails.

This might be the problem. Looking at the last patch on #4115, it hasn't
been updated to handle the Unicode changes on trunk. It provides a
__str__ method (which isn't wrong), but it doesn't ensure that the
output of get_url() is UTF-8 encoded. Maybe there should be a call to
django.utils.encoding.iri_to_uri() in there or maybe more changes are
needed.

[...]

This is interesting (the last line). It means that whatever "s" is here,
it doesn't have a __unicode__ method. Again, this isn't necessarily bad,
but it's not clear that it should be raising an encoding exception.

So, have a look at the debug screen and click on the "local variables"
link just below this last line in the extended traceback. What is "s"
here (in particular, it would be good know what type(s) is)? Also, what
is the value of 'encoding'?

If 's' doesn't look very interesting (i.e. it claims to be a string or
something like that), what is the current value of 'b' (and it's type)
in the previous line of the traceback? Again, using the "locals" link on
the debug page should be able to tell you this.

If you can't see the type of s, print it out to a file by changing that
line to something like:

try:

s = unicode(str(s), encoding, errors)

except:
# print type(s) and repr(s) to a file
raise

I'm not completely sure why the "ascii" encoding comes into play here
unless there is no explicit __str__ method on whatever 's' is. So this
should help us get the necessary information to go a bit further.

Find out that information and report back; we should be able to work
this out.

Regards,
Malcolm

--
If Barbie is so popular, why do you have to buy her friends?
http://www.pointy-stick.com/blog/

[cesco]

Thanks a lot Malcolm, I think we are getting there:-)

> So, have a look at the debug screen and click on the "local variables"
> link just below this last line in the extended traceback. What is "s"
> here (in particular, it would be good know what type(s) is)? Also, what
> is the value of 'encoding'?

Here are the values.
encoding: u'utf-8'
errors: u'strict'
s: <django.contrib.thumbnails.base.Thumbnail object at 0x019D4510>
strings_only: False

> what is the current value of 'b' (and it's type)
> in the previous line of the traceback? Again, using the "locals" link on
> the debug page should be able to tell you this.

b: <django.contrib.thumbnails.base.Thumbnail object at 0x019D4510>

Hope I was able to report all the information. If the problem is, like
you suspect, the patch I applied, do you have any suggestion on how to
solve the problem? Maybe modify the patch itself following the
guidelines for transition to Unicode?

Thanks a ton
Francesco

[Malcolm Tredinnick]

On Sun, 2007-08-19 at 08:19 +0000, cesco wrote:
> Thanks a lot Malcolm, I think we are getting there:-)
>
> > So, have a look at the debug screen and click on the "local variables"
> > link just below this last line in the extended traceback. What is "s"
> > here (in particular, it would be good know what type(s) is)? Also, what
> > is the value of 'encoding'?
>
> Here are the values.
> encoding: u'utf-8'
> errors: u'strict'
> s: <django.contrib.thumbnails.base.Thumbnail object at 0x019D4510>
> strings_only: False

[...]

>
> Hope I was able to report all the information. If the problem is, like
> you suspect, the patch I applied, do you have any suggestion on how to
> solve the problem? Maybe modify the patch itself following the
> guidelines for transition to Unicode?

As suspected, it's the Thumbnail field that isn't handling non-ASCII
filenames well.

Use the instructions in [1] to port the URL display stuff (this is
unicode.txt in the source tree, as well). The simple 5-step plan for
porting to Unicode won't be quite enough in this case. Just giving the
model a unicode method probably isn't enough, because you are still
responsible for encoding the URL correctly.

[1]
http://www.djangoproject.com/documentation/unicode/#uri-and-iri-handling

Regards,
Malcolm

--
Monday is an awful way to spend 1/7th of your life.
http://www.pointy-stick.com/blog/

[Michael Radziej]

Hi,

On Sun, Aug 19, Malcolm Tredinnick wrote:

>
> On Sun, 2007-08-19 at 00:27 -0700, cesco wrote:
> > File "C:\Python25\lib\site-packages\django\template\__init__.py" in
> > render
> > 739. return ''.join([force_unicode(b) for b in bits])

^^^
Malcolm: shouldn't this be u''.join(...)?

Michael

--
noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg -
Tel +49-911-9352-0 - Fax +49-911-9352-100
http://www.noris.de - The IT-Outsourcing Company

Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Hansjochen Klenk -
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689