Unicode decode errors with Admin?

[davenaff]

Is this a bug, or am I doing something wrong?

A sample models.py:
class Article(models.Model):
headline = models.CharField(max_length=100)

def __unicode__(self):
return u'%s' %self.headline

admin.site.register(Article)

-----------------------
And the creation of an object:

>>> Article.objects.create(headline=u'\xc2\xae')

When I try to view the Article list page in the admin, I get:
TemplateSyntaxError at /admin/test/article/
Caught an exception while rendering: 'ascii' codec can't decode
byte ...: ordinal not in range(128). You passed in
DjangoUnicodeDecodeError('ascii',...

[davenaff]

Oh, its worth noting:
- r8321.
- MySQL backend
- utf8 /utf8_bin charset/collation

[Karen Tracey]

On Thu, Aug 14, 2008 at 6:03 PM, davenaff <dave...@gmail.com> wrote:

Oh, its worth noting:
-  r8321.
- MySQL backend
- utf8 /utf8_bin charset/collation

It's the utf8_bin collation.  MySQL makes it extraordinarily difficult to differentiate between character data with a binary collation set and truly binary data.  So to be safe, the MySQLdb connector (as of the 1.1.2 level) does not attempt to transform any data which may be binary into unicode, so the value of you get in your model is a bytestring, not a unicode object.  I thought we had a fix for this but it turns out not.
 

On Aug 14, 2:58 pm, davenaff <daven...@gmail.com> wrote:
> Is this a bug, or am I doing something wrong?
>
> A sample models.py:
> class Article(models.Model):
>     headline = models.CharField(max_length=100)
>
>     def __unicode__(self):
>         return u'%s' %self.headline

change this to return self.headline.decode('utf8') and it will work.  If you want to be safe in the face of different collations it would be better to test for what sort of thing self.headline is and only do the decode if it's not already unicode.  Ugly but that's the best workaround I can come up with at the moment.

Karen

[Karen Tracey]

On Thu, Aug 14, 2008 at 6:20 PM, Karen Tracey <kmtr...@gmail.com> wrote:

  So to be safe, the MySQLdb connector (as of the 1.1.2 level)

Sorry, I meant MySQLdb 1.2.2 here, not 1.1.2.

Karen

[davenaff]

Beautiful, thanks for the help and very thorough explanation.

This fix worked as expected.

Should I file a ticket for this?

On Aug 14, 3:20 pm, "Karen Tracey" <kmtra...@gmail.com> wrote:

[James Matthews]

I ran into a similar issue. i wrote my unicode message like this and it worked

def __unicode__(self):
    return self.article

--
http://www.goldwatches.com/

[Malcolm Tredinnick]

We have a function that already knows how to do all that testing:

django.utils.encoding.force_unicode()

Given any unicode, UTF-8 or object that knows how to turns itself into
unicode input, it returns a unicode string. Exactly for situations like
this.

Regards,
Malcolm

>

[Malcolm Tredinnick]

On Thu, 2008-08-14 at 16:24 -0700, davenaff wrote:
> Beautiful, thanks for the help and very thorough explanation.
>
> This fix worked as expected.
>
> Should I file a ticket for this?

There's not really any bug here (certainly not with Django, and probably
not with MySQLdb either(. We just can't do anything about it,
unfortunately. MySQL returns an inconsistent result. It's for cases like
that that force_unicode() exists and you should just use it if you're
using MySQL and want to use utf8_bin collation.

Regards,
Malcolm