unicode characters garbled on dumpdata / loaddata using postgres

[Wiley]

I was wondering if anyone else had trouble or a workaround for dumping
and loading data with mixed character sets using a postgres backend.
My data has quite a few Chinese characters, they run from my normal
installation fine, but when i dump and reload the data, it's all
corrupted. Any hints here?

Thanks so much, the community has been incredibly helpful in my first
experiences with Django!

[Ivan Sagalaev]

Wiley wrote:
> I was wondering if anyone else had trouble or a workaround for dumping
> and loading data with mixed character sets using a postgres backend.
> My data has quite a few Chinese characters, they run from my normal
> installation fine, but when i dump and reload the data, it's all
> corrupted. Any hints here?

Are you using latest trunk builds? There were still a couple of problems
with serialization in 0.96 release that were fixed on trunk by switching
Django to unicode [1]. Now all should work fine.

[1]:
http://code.djangoproject.com/wiki/UnicodeBranch#PortingApplicationsTheQuickChecklist

[Wiley]

I started with the .96 version locally and then backed up and restored
the postgres DB on my live server, which is running the SVN version.
The import worked fine, the front end works fine, but there is a
problem with the backend. It seems to be having a problem dealing
with some of the unicode characters in my database (others it seems to
be dealing with fine. This error always comes up in the object view
of objects with chinese characters in the title:

UnicodeEncodeError
Exception Value: 'ascii' codec can't encode characters in position
0-2: ordinal not in range(128)
Exception Location: /usr/lib/python2.5/site-packages/django/utils/
encoding.py in force_unicode, line 39
Python Executable: /usr/bin/python
Python Version: 2.5.1

Template error

In template /usr/lib/python2.5/site-packages/django/contrib/admin/
templates/admin/change_form.html, error at line 15
Caught an exception while rendering: 'ascii' codec can't encode
characters in position 0-2: ordinal not in range(128)

15 {% if add %}{% trans "Add" %} {{ opts.verbose_name|escape }}{%
else %}{{ original|truncatewords:"18"|escape }}{% endif %}

Frustratingly, it seems to read other parts of the Chinese database
just fine...I'm not sure what's going wrong here? My database and
application are both set to UTF8

> [1]:http://code.djangoproject.com/wiki/UnicodeBranch#PortingApplicationsT...

[Wiley]

So just to update on this, I think I might have found a bug in the SVN
release of django. I've created a brand new postgres database and a
simple test application (aka no dumping or loading data, and no
database backup and restore). When I try and enter Chinese characters
through the admin backend, I get the exact same errors as before.
When I switch over the the 0.96 release it works perfectly, I switch
back, boom Admin template error. I've replicated this with apache2
and mod_python several times on my mac and on a ubuntu feisty
server.

I'm new to web programming and open source communities. In a
situation where I believe I've found a bug, should I first post it
here and confirm? Should I post it on the http://code.djangoproject.com
trac page?

Thanks for the help!

[Russell Keith-Magee]

On 8/23/07, Wiley <wiley....@gmail.com> wrote:
>
> I'm new to web programming and open source communities. In a
> situation where I believe I've found a bug, should I first post it
> here and confirm? Should I post it on the http://code.djangoproject.com
> trac page?

If you're certain you have found an error, then opening a ticket would
be the correct course of action. If you're uncertain, post it here
first, and someone may be able to set you straight.

Either way, you will need to describe the following:

- Your exact setup (all the software and version numbers you can think
of that might be involved)
- The exact Django SVN revision in which you have observed the
problem. Saying 'svn trunk' doesn't help, as the contents of the trunk
can change several times a day.
- A precise set of instructions (including sample models, views and
data if required) for reproducing the problem. Which buttons do I
press, which links do I follow, which commands do I run, and in which
order, to make your problem appear?

If you want to earn extra points with the Django core developers, the
very best way to demonstrate an error is with a patch to the Django
test suite - that is, adding a unittest or doctest condition that
fails to the suite of tests that we run whenever we make a change to
Django. In your case, you'll probably be looking at the
tests/regressiontests/fixtures_regress tests. This is the directory
that contains the (fairly comprehensive) tests for loading and dumping
data using fixtures, including some unicode data.

However, this last bit is 'icing on the cake'. If you can't make sense
of the test suite, a simple set of manual instructions will be fine.

Yours,
Russ Magee %-)