Django and psycopg2 problems

[Jacob Kaplan-Moss]

Howdy folks --

So it appears that the Django psycopg2 has a few issues that make it
operate differently than any other of the db backends. There's a
paste of the django test suite output at http://django.pastebin.com/
778189 which shows 6 failures, but actually there's only two things
wrong:

1. The psycopg2 backend returns strings as unicode objects instead of
raw strings. So for example::

File "/Users/jacob/WO/code/brave-new-world/django.trunk/tests/
modeltests/custom_columns/models.py", line ?, in
modeltests.custom_columns.models.__test__.API_TESTS
Failed example:
p.last_name
Expected:
'Smith'
Got:
u'Smith'

2. The backend returns datetimes with attached timezone info instead
of naieve datetimes::

File "/Users/jacob/WO/code/brave-new-world/django.trunk/tests/
modeltests/basic/models.py", line ?, in
modeltests.basic.models.__test__.API_TESTS
Failed example:
Article.objects.get(id__exact=8).pub_date
Expected:
datetime.datetime(2005, 7, 31, 12, 30, 45)
Got:
datetime.datetime(2005, 7, 31, 12, 30, 45,
tzinfo=<psycopg2.tz.FixedOffsetTimezone object at 0x17178f0>)

Now, I know that both behaviors are actually more correct than the
other backends, but these subtle differences between the psycopg2
adaptor and the other ones is really driving me nuts.

Are the any objections to changing the psycopg2 backend to behave the
same was as the other ones (knowing that it's actually a step
backwards)? Eventually the right thing to do would be to make the
*other* backends behave this way, but that's all caught up with
unicodification, and I'd like to move towards consistency first, and
then correctness when it's ready.

Jacob

[Adrian Holovaty]

On 8/28/06, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:
> Are the any objections to changing the psycopg2 backend to behave the
> same was as the other ones (knowing that it's actually a step
> backwards)? Eventually the right thing to do would be to make the
> *other* backends behave this way, but that's all caught up with
> unicodification, and I'd like to move towards consistency first, and
> then correctness when it's ready.

It's definitely a step backward, but it's probably better to be
consistent than "more correct" in this case. Fine by me --

Adrian

--
Adrian Holovaty
holovaty.com | djangoproject.com

[Jacob Kaplan-Moss]

Yeah, it sucks, but consistancy seems important to me, too. I've
checked in the "fix" as [3675]; it's pretty easy to remove later.

Jacob

[crankycoder]

Aww... this is a regression from ticket #2514 and changeset [3565].

I really *really* need to be able to save and load unicode data in a
non-painful way. I have to support at least English, Vietnamese, Thai
and a half dozen other scripts.

vic

[crankycoder]

I should have elaborated on my problem.

I basically need to be able to handle about a dozen languages all at
the same time. That basically means I need Unicode to work. There's
some patches in Trac that I've put up that handle all the cases that
I'm currently aware of - some of which certainly need to be cleaned up.

That said - the call to
"psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)" is
badly needed or else I have to end up modifying all the character type
columns that go to varchar, char, or text so that they do UTF8 encoding
on the way to becoming SQL and decoding UTF8 on the way out.

The problem is further complicated since search queries will be made
more complicated - I'll need to modify the way the filter() method
works to encode all unicode strings to utf8 when constructing the
SELECT statement.

I can obviously just maintain my own patchset to keep the UNICODE
setting turned on, but I don't understand what is so bad about having
it on in the first place.

vic

[Jacob Kaplan-Moss]

On Aug 31, 2006, at 12:31 AM, crankycoder wrote:
> I basically need to be able to handle about a dozen languages all at
> the same time. That basically means I need Unicode to work. There's
> some patches in Trac that I've put up that handle all the cases that
> I'm currently aware of - some of which certainly need to be cleaned
> up.
>
> That said - the call to
> "psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)" is
> badly needed or else I have to end up modifying all the character type
> columns that go to varchar, char, or text so that they do UTF8
> encoding
> on the way to becoming SQL and decoding UTF8 on the way out.
>
> The problem is further complicated since search queries will be made
> more complicated - I'll need to modify the way the filter() method
> works to encode all unicode strings to utf8 when constructing the
> SELECT statement.
>
> I can obviously just maintain my own patchset to keep the UNICODE
> setting turned on, but I don't understand what is so bad about having
> it on in the first place.

Essentially the problem is that this makes the psycopg2 backend
behave differently from all the other backends. Thus many framework
tests fail when using psycopg2, and code written for psycopg2 breaks
when running with a different backend.

It's totally a step backwards, but I think consistancy is more
important. I'd suggest if you need the unicode behavior that you
call the ``register_type`` function in your code yourself.

Jacob

[Ivan Sagalaev]

crankycoder wrote:
> I really *really* need to be able to save and load unicode data in a
> non-painful way. I have to support at least English, Vietnamese, Thai
> and a half dozen other scripts.

Why can't you use utf-8 for this? I support a two-language site
(Russian, English) with psycopg1 and never had problems with it.

Basically you need native unicode only if you work with individual
characters in strings.

[Victor Ng]

Fair enough - any chance we can get that unicode branch opened up soon?

vic