Django custom types / DjangoUnicodeDecodeError:

[Brandon]

Greetings,
My organization is evaluating Django as a possible frontend to a
legacy IBM Db2 Database. I've successfully connected the Django 1.2
beta with our database, but ran into an issue with a DB2 specific
column.

Our tables all have primary keys defined as `CHAR(13) FOR BIT DATA`,
which is essentially a binary type (db2 has a built-in function to
generate these in lieu of an auto-increment column). I created a
custom type in Django, and used the `binascii` module in python to
pack/unpack the binary data (this much is known to work). I have a
working SQLAlchemy example that works in largely the same way. Here's
a very simple example of what's going on:

class TechIdField(models.Field):
description = "A Python type to translate CHAR FOR BIT DATA
fields to strings"

def __init__ (self, *args, **kwargs):
super(TechIdField, self).__init__(*args, **kwargs)

def db_type(self, connection):
return 'CHAR(13) FOR BIT DATA'

def to_python(self, value):
return b2a_hex(value)

def get_db_prep_value(self, value):
return a2b_hex(value)

class TestModel(models.Model):
tech_id = TechIdField(primary_key=True)
pref_name = models.CharField(max_length=20)
pref_value = models.CharField(max_length=100)

class Meta:
db_table = 'test_model'

When I try to run a query in the Django shell, the database backend
bombs out because it attempts to convert the value that my custom type
just packed to a unicode string. Here's an example from a manage.py
shell session:

>>> from testapp import models;
>>>
models.TestModel.objects.filter(tech_id='20070618193649842630000000')
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/lib/python2.5/site-packages/django/db/models/
query.py", line 66, in __repr__
data = list(self[:REPR_OUTPUT_SIZE + 1])
File "/usr/local/lib/python2.5/site-packages/django/db/models/
query.py", line 81, in __len__
self._result_cache.extend(list(self._iter))
File "/usr/local/lib/python2.5/site-packages/django/db/models/
query.py", line 267, in iterator
for row in compiler.results_iter():
File "/usr/local/lib/python2.5/site-packages/django/db/models/
sql/compiler.py", line 685, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.5/site-packages/django/db/models/
sql/compiler.py", line 740, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.5/site-packages/django/db/backends/
util.py", line 22, in execute
sql = self.db.ops.last_executed_query(self.cursor, sql,
params)
File "/usr/local/lib/python2.5/site-packages/django/db/backends/
__init__.py", line 211, in last_executed_query
u_params = tuple([to_unicode(val) for val in params])
File "/usr/local/lib/python2.5/site-packages/django/db/backends/
__init__.py", line 209, in <lambda>
to_unicode = lambda s: force_unicode(s, strings_only=True)
File "/usr/local/lib/python2.5/site-packages/django/utils/
encoding.py", line 92, in force_unicode
raise DjangoUnicodeDecodeError(s, *e.args)
DjangoUnicodeDecodeError: 'utf8' codec can't decode byte 0x84 in
position 7: unexpected code byte. You passed in ' \x07\x06\x18\x196I
\x84&0\x00\x00\x00' (<type 'str'>)

I sifted through the trac tickets a bit and this is exactly what was
reported in #6416, which was unfortunately closed without being
resolved.

I'd love to keep Django in the mix as a possible framework for us.
Could anyone suggest a workaround? This would pose a pretty big
problem for us, unfortunately, since almost every query we make will
have one or more of these ID's. I'm obviously new to Django as a
whole, so it's quite likely there is a better way for me to go about
getting Django to understand and deal with this column type.

Any help would be much appreciated.

Cheers,
Brandon

[Karen Tracey]

What happens if you try running with debug off?

Karen

[Brandon]

No error. I'm guessing Django doesn't populate connection.queries
when it isn't in debug mode?

This would definitely work for production, but what other options do I
have for development? It would be a huge step up for us to be able to
work in an interactive shell and see what queries were run during
those actions.

[Karen Tracey]

On Tue, Mar 2, 2010 at 7:05 PM, Brandon <bhor...@gmail.com> wrote:

No error.  I'm guessing Django doesn't populate connection.queries
when it isn't in debug mode?

Right.
 

This would definitely work for production, but what other options do I
have for development? It would be a huge step up for us to be able to
work in an interactive shell and see what queries were run during
those actions.

Try changing this line:

     File "/usr/local/lib/python2.5/site-packages/django/db/backends/__init__.py", line 209, in <lambda>

       to_unicode = lambda s: force_unicode(s, strings_only=True)

to: 

       to_unicode = lambda s: force_unicode(s, strings_only=True, errors='replace')

(You might want to switch to running from a copy of Django not installed in site-packages rather than start changing the official installed version.)

That might avoid the problem, though the params data stored in connection queries won't necessarily be too helpful.

I think there's an argument to be made that errors='replace' ought to be what is specified there, just because the gathering of debug data should not cause an exception to be raised. So that might be a relatively simple fix that could be put in easily, if it is actually helpful for your case.

Karen

[Brandon]

Thanks for the pointers, Karen. That change prevents the exception
with Debug enabled, and lets us at least see the general queries that
were run, though the Id's are mangled, as you'd expect. This should
at least get me over the hump for now.

Thanks again!

On Mar 2, 9:02 pm, Karen Tracey <kmtra...@gmail.com> wrote: