Django encoding

[Lukáš]

Hey,

I have a little problem with django encoding. I have got everything in
the DB set as utf8. Now when I get them from db with django, the
encoding gets broken. For some reason django "thinks" (maybe knows)
the data is latin1, which wouldnt be a problem if I would tell him to
use latin1, which fixed encoding in templates.. Now here comes the
funny stuff.

But since then I wasnt able to edit anything, because I get the
classic decoding error. So I set it back to utf8, everything is
broken, but I can edit stuff. Now when I edit the broken string, it
will be saved, BUT when I get it out with PHP its, again, latin1.

So this is what happen:

get string from DB with PHP - its utf8
get string from DB with Django - its latin1
save with Django and get it with PHP - its latin1

I cant figure where the problem is. Maybe its the DB, but I dont know
"how" to fix it, because no matter what I do it will break either
Django or PHP. I have tried to convert the data to utf8, but it still
the same.

Here are some settings, maybe it helps:

DATABASE_OPTIONS = {
'use_unicode': True,
'charset': 'utf8',
}

Any ideas?

Regards

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[Karen Tracey]

On Wed, May 5, 2010 at 6:17 AM, Lukáš <luka...@gmail.com> wrote:

Hey,

I have a little problem with django encoding. I have got everything in
the DB set as utf8. Now when I get them from db with django, the
encoding gets broken. For some reason django "thinks" (maybe knows)
the data is latin1, which wouldnt be a problem if I would tell him to
use latin1, which fixed encoding in templates.. Now here comes the
funny stuff.

I don't know what you mean by saying Django '"thinks" (maybe knows) the data is latin1". Django does not assume or have a default charset of latin1 anywhere.

When Django communicates with the database, it sets the connection charset to utf-8. The database is responsible for converting between whatever charset is actually used for the data to/from utf-8 for the connection to Django. You can have, for example, a MySQL database that is using the latin1 charset. MySQL will convert the latin1 data to utf-8 when sending data on the Django connection, and similarly will convert utf-8 data received on the connection to latin1 for storage in the database. Django never sees latin1 data: it has set the connection to utf-8 so everything sent and received from the database is utf-8 encoded.

It's possible you have a mismatch between the actual encoding used in the database and the charset it is supposedly configured to be. For example, with MySQL and old Wordpress versions it is apparently easy to get into a situation where MySQL tables are created as latin1 but the actual encoding of the data is utf-8. But to track that down you are going to need to be a lot more specific about what your database is, how it was created and what programs were responsible for storing the data in it. Alternatively if you would give specifics of the "classic decoding error" you mention later that might provide a clue as to what and where, exactly, the mismatch is. You also mention you have tried to convert the data to utf8 and say it's still the same: if you would be very specific about what you mean by "tried to convert the data to utf8" that would help people help you more effectively.

Here are some settings, maybe it helps:

  DATABASE_OPTIONS = {
     'use_unicode': True,
     'charset': 'utf8',
   }

You don't need this DATABASE_OPTIONS setting (and should delete it to avoid confusion), unless you are using a very very old (pre 1.0) version of Django. If you are doing that, my advice is to update to a supported Django release.

Karen