Re: [Django] #14518: Field.to_python not called on foreign key IDs
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.2
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by avnimahajan@…):
* ui_ux: => 0
* easy: => 0
Comment:
I am also facing same problem. We have changed the default behavior of
having primary keys. Throughout our code it has been replaced with custom
UUIDFiled(given below). Till django 1.1 we were not facing any problem but
while migrating to django1.3, I started facing problem during
syncdb(duplicate key error while inserting in auth_permissions).
Since there is change in "django/contrib/auth/management/__init__.py"
create_permission code. Now it is referring to "content_type" which is a
Foreign key. In my case its value is not being to_python'd. Thus it fails
in comparison with a uuid.UUID type value (returned when referred as pk
which is always to_python'd).
Please update by when this bug will be solved?
class UUIDField(models.Field):
__metaclass__ = models.SubfieldBase
empty_strings_allowed = False
def __init__(self, *args, **kwargs):
if kwargs.get('primary_key', False):
kwargs['editable'] = False
kwargs['db_index'] = True
super(UUIDField, self).__init__(*args, **kwargs)
def db_type(self, connection):
return 'uuid'
def get_internal_type(self):
return 'UUIDField'
def to_python(self, value):
if (value is None) or isinstance(value, uuid.UUID):
return value
try:
return uuid.UUID(value)
except (AttributeError, TypeError, ValueError):
return value
def pre_save(self, obj, add):
old_val = getattr(obj, self.attname)
if (self.primary_key and add) and (old_val is None):
value = uuid.uuid4()
setattr(obj, self.attname, value)
return value
else:
return old_val
def get_db_prep_lookup(self, lookup_type, value,connection=None,
prepared=False):
if lookup_type == 'exact':
return [self.get_db_prep_value(value,connection=None,
prepared=False)]
elif lookup_type == 'in':
return [self.get_db_prep_value(v,connection=None,
prepared=False) for v in value]
elif lookup_type == 'isnull':
return []
raise TypeError(u"Field has invalid lookup type: %s" %
lookup_type)
def value_to_string(self, obj):
value = getattr(obj, self.attname)
if value is None:
return value
else:
return unicode(value)
def formfield(self, **kwargs):
defaults = {
'form_class': forms.RegexField,
'regex': UUID_REGEX,
'max_length': 47,
'error_messages': {'invalid': u"Enter only valid UUID."}
}
defaults.update(kwargs)
return super(UUIDField, self).formfield(**defaults)
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.2
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* cc: kitsunde@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.2
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* cc: info@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:6>
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.2
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by anubhav9042):
Well there have been some changes in the code and some things have
changed.
Here is my models.py
{{{
from django.db import models
class MyField(models.CharField):
__metaclass__ = models.SubfieldBase
def __init__(self, *args, **kwargs):
super(MyField, self).__init__(*args, **kwargs)
def to_python(self, value):
return "Value: %s" % value
class Foo(models.Model):
field = MyField(max_length=30, primary_key=True)
class Bar(models.Model):
foo = models.ForeignKey(Foo)
}}}
In shell, I get correct values here:
{{{
>>> m1 = Foo.objects.create(field="anubhavjoshi")
>>> m2 = Bar.objects.create(foo=m1)
>>> m2.foo_id
u'Value: Value: anubhavjoshi'
>>> m2.foo.field
u'Value: Value: anubhavjoshi'
}}}
But if I do:
{{{
>>> m1 = Foo.objects.create(field="anubhavjoshi")
>>> m2 = Bar.objects.create(foo=m1)
>>> m2 = Bar.objects.all()[0]
>>> m2.foo_id
u'Value: Value: anubhavjoshi'
>>> m2.foo.field
DoesNotExist: Foo matching query does not exist.
>>> m2.foo
DoesNotExist: Foo matching query does not exist.
}}}
Following traceback:
{{{
---------------------------------------------------------------------------
DoesNotExist Traceback (most recent call
last)
C:\Users\Anubhav\DjangoDev\apps\<ipython-input-7-3a4a21c18010> in
<module>()
----> 1 m2.foo
c:\users\anubhav\django\django\db\models\fields\related.pyc in
__get__(self, instance, instance_type)
562 qs = qs.filter(extra_filter, **params)
563 # Assuming the database enforces foreign keys,
this won't fail.
--> 564 rel_obj = qs.get()
565 if not self.field.rel.multiple:
566 setattr(rel_obj,
self.field.related.get_cache_name(), instance)
c:\users\anubhav\django\django\db\models\query.pyc in get(self, *args,
**kwargs)
374 raise self.model.DoesNotExist(
375 "%s matching query does not exist." %
--> 376 self.model._meta.object_name)
377 raise self.model.MultipleObjectsReturned(
378 "get() returned more than one %s -- it returned %s!" %
(
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:7>
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* cc: anubhav9042@… (added)
* version: 1.2 => master
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:8>
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by anubhav9042):
I am attaching a diff to fix what remains.
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:9>
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by anubhav9042):
This might cause other tests to fail but might give some insight as to why
this problem occured.
As queryset gets the correct value earlier on and is changed to empty qs
after that condition.
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:10>
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by Harm Geerts <hgeerts@…>):
The reason the first example works is because in
Bar.objects.create(foo=m1) the m1 Foo instance is cached on the Bar
instance, so m2.foo.field does not perform a database query.
The second example fails because m2 has to perform a database lookup, and
that's where things get fun.
Your MyField.to_python is called multiple times so the value stored in the
database is actually "Value: Value: anubhavjoshi".
When you access bar.foo the primary key value "Value: Value: anubhavjoshi"
is passed through to_python again and becomes "Value: Value: Value:
anubhavjoshi" and that primary key does not exist.
The reason your patch works is because you only had one Foo record in your
dataset.
If you had multiple Foo records your test would raise
MultipleObjectsReturned.
The only thing the patch does is disable related lookup filters which is
not what you want ;)
It may be an idea to add a note to the documentation about custom field
classes.
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:11>
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by anubhav9042):
Thanks for your explanation. As I said my patch wasn't correct, it only
was to tell which line caused errors.
Now that I tried the following:
{{{
def to_python(self, value):
return ".".join(value)
}}}
So the lookup which failed earlier passes, but as explained by Harm above,
{{{
>>> m2.foo_id
u'a...n...u...b...h...a...v'
>>> m2.foo.field
u'a.......n.......u.......b.......h.......a.......v'
}}}
But it should have been: `u'a.n.u.b.h.a.v'`
It seems that `to_python` is called too many times and that might be wrong
when it is defined in a way like in my examples..
Also if we remove `__metaclass__ = models.SubfieldBase` from our custom
field then `to_python` does gets called but something strange happens as
it appears it does not serves the purpose as always `u'anubhav'` is
returned by `m1.field` or `m2.foo_id`.
There might be problem in that metaclass implementation, but thats just a
wild guess.
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:12>
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by slurms):
@anubhav9042: looks like this is a problem with the way you're subclassing
the `CharField`. When `to_python` is called, it could well already be in
the right format, or it might be unset. `to_python` gets called every time
the field is assigned a value and you are responsible for ensuring it is
in the right format (https://docs.djangoproject.com/en/1.6/howto/custom-
model-fields/#django.db.models.Field.to_python). Notice the two lines at
the beginning of that example code. You will need to do something similar.
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:13>
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by antialiasis):
Is there any chance this might be fixed in some way at some point?
Like others, we've been using UUIDs as primary keys on a PostgreSQL
database, and this bug is completely breaking `prefetch_related`. The way
prefetching is implemented (at least in 1.7), instances are collected into
a dictionary with their `to_python`'d primary keys as the keys, and then
it tries to access them using the non-`to_python`'d foreign key ID field.
Needless to say, this results in a KeyError if `to_python` changes the
representation of the field at all (in our case, to strip PostgreSQL's
native hyphens).
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:14>
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by timgraham):
Yes, once we have a working patch and someone reviews it.
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:15>
Django
-------------------------------------+-------------------------------------
Reporter: wolever | Owner: nobody
Component: Database layer | Version: master
(models, ORM) | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by timgraham):
* status: new => closed
* resolution: => worksforme
Comment:
I can't reproduce the behavior described in the ticket as far back as
Django 1.2. `b.foo_id` and `b.foo.id` both return an integer. A look of
the follow up comments use `SubfieldBase` which is deprecated in favor of
`Field.from_db_value()`. If problems remain in newer versions of Django,
please open a new ticket with details.
--
Ticket URL: <https://code.djangoproject.com/ticket/14518#comment:16>