Writing custom model fields with Multi-table inheritance

76 views
Skip to first unread message

gordyt

unread,
Dec 16, 2008, 4:53:34 PM12/16/08
to Django users
Howdy Folks!

I wanted to be able to have UUID primary keys for certain models.
I found the implementation done by the django_extensions project
(http://code.google.com/p/django-command-extensions/) and it works
fine
as is.

But I wanted to be able to have a UUIDField that stored its values in
an
actual 'uuid' column if we were using PostgreSQL. For all other
databases
it would use the same char(36) field as in the original
implementation.
This was for performance reasons, by the way.

So listed below was my attempt at creating a UUIDField (based on the
django_extensions one) that would do that. It works great in every
situation EXCEPT if I am working with a model that inherits from
another
model.

Before I go any further, let me say that I've posted all of this
on a web site so that it can be seen with nicer formatting than what
is
possible in a newsgroup post. The URL for that page is:

http://www.gordontillman.info/computers/41-django/94-django-uuidfield-problem

-- or --

http://tinyurl.com/6rnsya

I've included the complete UUIDField definition, some sample models,
and a small test case.

Here is an interesting part. The to_python() method of the UUIDField
is supposed to return an instance of uuid.UUID. This is my original
implementation of that method:

def to_python(self, value):
if not value:
return None
if isinstance(value, uuid.UUID):
return value
# attempt to parse a UUID
return uuid.UUID(smart_unicode(value))


This original implementation words great unless I'm working with a
model that inherits from a base model that uses the UUIDField as its
primary key. If I change the implementation to this then the test
case
passes, both for the base model and the inherited model. But of
course
now it is returning a string, not a uuid.UUID value.

def to_python(self, value):
if not value:
return None
if isinstance(value, uuid.UUID):
return smart_unicode(value)
else:
return value

I was wondering if anyone could suggest an improvement to my
UUIDField implementation so that (1) to_python() returns a proper
uuid.UUID instance and (2) I can still use 'uuid' columns in
databases that support it.

It's possible there is a bug in the part of the Django code that
deals with inherited models, but I'm sure it's way more likely
that there is a bug in my UUIDField!


---------- begin UUIDField ----------
import uuid

from django.forms.util import ValidationError
from django import forms
from django.db import models
from django.utils.encoding import smart_unicode
from django.utils.translation import ugettext_lazy

class UUIDVersionError(Exception):
pass

class UUIDField(models.Field):
"""Encode and stores a Python uuid.UUID in a manner that is
appropriate
for the given datatabase that we are using.

For sqlite3 or MySQL we save it as a 36-character string value
For PostgreSQL we save it as a uuid field

This class supports type 1, 2, 4, and 5 UUID's.
"""
__metaclass__ = models.SubfieldBase

_CREATE_COLUMN_TYPES = {
'postgresql_psycopg2': 'uuid',
'postgresql': 'uuid'
}

def __init__(self, verbose_name=None, name=None, auto=True,
version=1,
node=None, clock_seq=None, namespace=None, **kwargs):
"""Contruct a UUIDField.

@param verbose_name: Optional verbose name to use in place of
what
Django would assign.
@param name: Override Django's name assignment
@param auto: If True, create a UUID value if one is not
specified.
@param version: By default we create a version 1 UUID.
@param node: Used for version 1 UUID's. If not supplied, then
the
uuid.getnode() function is called to obtain it. This can
be slow.
@param clock_seq: Used for version 1 UUID's. If not supplied
a random
14-bit sequence number is chosen
@param namespace: Required for version 3 and version 5 UUID's.
@param name: Required for version4 and version 5 UUID's.

See Also:
- Python Library Reference, section 18.16 for more
information.
- RFC 4122, "A Universally Unique IDentifier (UUID) URN
Namespace"

If you want to use one of these as a primary key for a Django
model, do this::
id = UUIDField(primary_key=True)
This will currently I{not} work with Jython because PostgreSQL
support
in Jython is not working for uuid column types.
"""
self.max_length = 36
kwargs['max_length'] = self.max_length
if auto:
kwargs['blank'] = True
kwargs.setdefault('editable', False)

self.auto = auto
self.version = version
if version==1:
self.node, self.clock_seq = node, clock_seq
elif version==3 or version==5:
self.namespace, self.name = namespace, name

super(UUIDField, self).__init__(verbose_name=verbose_name,
name=name, **kwargs)

def create_uuid(self):
if not self.version or self.version==4:
return uuid.uuid4()
elif self.version==1:
return uuid.uuid1(self.node, self.clock_seq)
elif self.version==2:
raise UUIDVersionError("UUID version 2 is not supported.")
elif self.version==3:
return uuid.uuid3(self.namespace, self.name)
elif self.version==5:
return uuid.uuid5(self.namespace, self.name)
else:
raise UUIDVersionError("UUID version %s is not valid." %
self.version)

def db_type(self):
from django.conf import settings
return UUIDField._CREATE_COLUMN_TYPES.get
(settings.DATABASE_ENGINE,
"char(%s)" % self.max_length)

def to_python(self, value):
"""Return a uuid.UUID instance from the value returned by the
database."""
#
# This is the proper way... But this doesn't work correctly
when
# working with an inherited model
#
if not value:
return None
if isinstance(value, uuid.UUID):
return value
# attempt to parse a UUID
return uuid.UUID(smart_unicode(value))

#
# If I do the following (returning a String instead of a UUID
# instance), everything works.
#

#if not value:
# return None
#if isinstance(value, uuid.UUID):
# return smart_unicode(value)
#else:
# return value

def pre_save(self, model_instance, add):
if self.auto and add:
value = self.create_uuid()
setattr(model_instance, self.attname, value)
else:
value = super(UUIDField, self).pre_save(model_instance,
add)
if self.auto and not value:
value = self.create_uuid()
setattr(model_instance, self.attname, value)
return value


def get_db_prep_value(self, value):
"""Casts uuid.UUID values into the format expected by the back
end
for use in queries"""
if isinstance(value, uuid.UUID):
return smart_unicode(value)
return value


def value_to_string(self, obj):
val = self._get_val_from_obj(obj)
if val is None:
data = ''
else:
data = smart_unicode(val)
return data

def formfield(self, **kwargs):
defaults = {
'form_class': forms.CharField,
'max_length': self.max_length
}
defaults.update(kwargs)
return super(UUIDField, self).formfield(**defaults)
---------- end UUIDField ----------

---------- begin sample models ----------

from django.db import models
from fields import UUIDField

class Customer(models.Model):
MAX_NAME_LEN = 200
id = UUIDField(primary_key=True)
name = models.CharField(max_length=MAX_NAME_LEN)

class User(models.Model):
MAX_FIRST_NAME = 32
MAX_LAST_NAME = 32
MAX_USERNAME = 32

id = UUIDField(primary_key=True)
customer = models.ForeignKey(Customer)
first_name = models.CharField(max_length=MAX_FIRST_NAME)
last_name = models.CharField(max_length=MAX_LAST_NAME)
username = models.CharField(max_length=MAX_USERNAME)
password = models.CharField(max_length=128, blank=True)

class Teacher(User):
email = models.EmailField()
admin = models.BooleanField(default=False)
---------- end sample models ----------


---------- begin sample test ----------
import logging
import random
import unittest
import time
import sys
from models import *

class InheritanceTestCase(unittest.TestCase):
"""
python manage.py test hb7t.InheritanceTestCase
"""

def runTest(self):
c = Customer(name='cust1')
c.save()
u = User(customer=c, first_name='f', last_name='l',
username='fl',
password='p')
u.save()
self.assertEqual(1, Customer.objects.count())
self.assertEqual(1, User.objects.count())
self.assertEqual(0, Teacher.objects.count())

t = Teacher(customer=c, first_name='g', last_name='t',
username='gt',
password='p', email='g...@test.com')
# Everything passes up to this point, whether or not to_python
() returns
# a uuid.UUID value OR a string value
# But t.save() fails if to_python() returns a uuid.UUID value
t.save()
self.assertEqual(1, Customer.objects.count())
self.assertEqual(2, User.objects.count())
self.assertEqual(1, Teacher.objects.count())

c.delete()
self.assertEqual(0, Customer.objects.count())
self.assertEqual(0, User.objects.count())
self.assertEqual(0, Teacher.objects.count())

---------- end sample test ----------



ERROR: runTest (kindle.hb7t.tests.InheritanceTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/gordy/projects/kindle/../kindle/hb7t/tests.py", line
118, in runTest
t.save()
File "/usr/local/lib/python2.5/site-packages/django/db/models/
base.py", line 328, in save
self.save_base(force_insert=force_insert,
force_update=force_update)
File "/usr/local/lib/python2.5/site-packages/django/db/models/
base.py", line 375, in save_base
manager.filter(pk=pk_val).extra(select={'a': 1}).values
('a').order_by())):
File "/usr/local/lib/python2.5/site-packages/django/db/models/
query.py", line 191, in __nonzero__
iter(self).next()
File "/usr/local/lib/python2.5/site-packages/django/db/models/
query.py", line 185, in _result_iter
self._fill_cache()
File "/usr/local/lib/python2.5/site-packages/django/db/models/
query.py", line 618, in _fill_cache
self._result_cache.append(self._iter.next())
File "/usr/local/lib/python2.5/site-packages/django/db/models/
query.py", line 659, in iterator
for row in self.query.results_iter():
File "/usr/local/lib/python2.5/site-packages/django/db/models/sql/
query.py", line 203, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.5/site-packages/django/db/models/sql/
query.py", line 1756, in execute_sql
cursor.execute(sql, params)
ProgrammingError: can't adapt

Viktor

unread,
Feb 1, 2009, 2:36:54 PM2/1/09
to Django users
Hi,

did you got any response or did you managed to solve your problem?
I've just started to search for a UUIDField, and it would be great to
have proper uuid columns for postgres.

If so do you plan to commit it to django_extensions?

Thanks, Viktor

On 2008 dec. 16, 22:53, gordyt <gor...@gmail.com> wrote:
> Howdy Folks!
>
> I wanted to be able to have UUID primary keys for certain models.
> I found the implementation done by the django_extensions project
> (http://code.google.com/p/django-command-extensions/) and it works
> fine
> as is.
>
> But I wanted to be able to have a UUIDField that stored its values in
> an
> actual 'uuid' column if we were using PostgreSQL.  For all other
> databases
> it would use the same char(36) field as in the original
> implementation.
> This was for performance reasons, by the way.
>
> So listed below was my attempt at creating a UUIDField (based on the
> django_extensions one) that would do that.  It works great in every
> situation EXCEPT if I am working with a model that inherits from
> another
> model.
>
> Before I go any further, let me say that I've posted all of this
> on a web site so that it can be seen with nicer formatting than what
> is
> possible in a newsgroup post.  The URL for that page is:
>
> http://www.gordontillman.info/computers/41-django/94-django-uuidfield...
>             password='p', email=...@test.com')

gordyt

unread,
Feb 20, 2009, 10:21:19 AM2/20/09
to Django users
Howdy Viktor,

On Feb 1, 1:36 pm, Viktor <viktor.n...@gmail.com> wrote:
> Hi,
>
> did you got any response or did you managed to solve your problem?
> I've just started to search for a UUIDField, and it would be great to
> have proper uuid columns for postgres.
>
> If so do you plan to commit it to django_extensions?

Sorry for the delayed reply. I just noticed your message. I do have
my custom uuid field working, using native uuid datatype with
postgresql and char field for other databases.

There is one situation where I'm seeing a problem in the Django admin
interface and as soon as I can resolve it I will post the complete
solution.

The problem is seen when using a model that has a uuid primary key
that is edited inline (tabular or stacked) on the same page as a
parent model. I see this problem not just with my version of
UUIDField but also with the one that is currently in django-
extensions.

I'm trying to figure out what Django does differently when using it's
own automatically generated primary key as opposed to using a declared
primary key.

That is the only situation I have found that causes a problem. There
is no difficulty with the admin interface when working with models
that have uuid primary keys when they have their own admin edit page.

--gordon

Karen Tracey

unread,
Feb 20, 2009, 12:34:54 PM2/20/09
to django...@googlegroups.com
On Fri, Feb 20, 2009 at 10:21 AM, gordyt <gor...@gmail.com> wrote:

Howdy Viktor,

On Feb 1, 1:36 pm, Viktor <viktor.n...@gmail.com> wrote:
> Hi,
>
> did you got any response or did you managed to solve your problem?
> I've just started to search for a UUIDField, and it would be great to
> have proper uuid columns for postgres.
>
> If so do you plan to commit it to django_extensions?

Sorry for the delayed reply.  I just noticed your message.  I do have
my custom uuid field working, using native uuid datatype with
postgresql and char field for other databases.

There is one situation where I'm seeing a problem in the Django admin
interface and as soon as I can resolve it I will post the complete
solution.

The problem is seen when using a model that has a uuid primary key
that is edited inline (tabular or stacked) on the same page as a
parent model.  I see this problem not just with my version of
UUIDField but also with the one that is currently in django-
extensions.

I'm trying to figure out what Django does differently when using it's
own automatically generated primary key as opposed to using a declared
primary key.

This sounds like a problem that has been fixed.  Are you running with a recent enough trunk or 1.0.X branch checkout so that you have this fix:

http://code.djangoproject.com/changeset/9664

?

Karen

gordyt

unread,
Feb 20, 2009, 3:37:48 PM2/20/09
to Django users
> This sounds like a problem that has been fixed.  Are you running with a
> recent enough trunk or 1.0.X branch checkout so that you have this fix:
>
> http://code.djangoproject.com/changeset/9664

Karen I'm running build 9846. I'm going to put together a very
minimal example to illustrate the problem and will post a link to it
in this thread.

Thanks!

--gordy

gordyt

unread,
Feb 20, 2009, 6:14:32 PM2/20/09
to Django users
Karen I made a small sample to illustrate the problem and posted it
here:

http://dpaste.com/hold/123199/

It's an extremely simple test case and instructions are included in
the comments. I'm not sure if this error is related to the issue that
you told me about or if it is something new entirely.

Thanks a bunch!

--gordy

Karen Tracey

unread,
Feb 20, 2009, 11:20:00 PM2/20/09
to django...@googlegroups.com

It's a different problem (the one I was thinking of was a problem on initial save, not save of an existing object), but I think it is the same as this one:

http://code.djangoproject.com/ticket/8813

If you could attach the file you dpasted to that ticket and note that you see this problem in the admin, that would probably be useful, as recreating it in admin should be easier than recreating the user-defined form, view, and models involved in the initial report there.  (I have not had time to look in any detail at that problem, but it looks the same root cause based on the traceback.)

Karen

gordyt

unread,
Feb 21, 2009, 2:56:56 PM2/21/09
to Django users
> http://code.djangoproject.com/ticket/8813
>
> If you could attach the file you dpasted to that ticket and note that you
> see this problem in the admin, that would probably be useful, as recreating
> it in admin should be easier than recreating the user-defined form, view,
> and models involved in the initial report there.  (I have not had time to
> look in any detail at that problem, but it looks the same root cause based
> on the traceback.)

Thanks Karen!

I will do that first thing on Monday when I get back to the office. I
appreciate your looking into this!

--gordy
Reply all
Reply to author
Forward
0 new messages