problems with get_or_create
Kenneth Gonsalves
I have a model like this:
name - unique
slno - not null
mode - not null
If the instance does not exist and I give all three values to get_or_create,
it works
if the instance exists, and I give values for slno and mode which are different
from the existing ones - I get a programming error as get_or_create tries to
create a new instance and falls foul of the unique constraint on name. It
works if I only give the value for name - it gets the instance and then I can
add the new values and save
if the instance does not exist and I only give the value for name, I get a
programming error that the not null constraints on slno and mode are violated.
so it looks to me that get_or_create will only work reliably when there are no
not null or unique constraints in the model. Or am I missing something?
--
regards
Kenneth Gonsalves
Senior Associate
NRC-FOSS
http://certificate.nrcfoss.au-kbc.org.in
Tom Evans
> hi,
>
> I have a model like this:
>
> name - unique
> slno - not null
> mode - not null
>
> If the instance does not exist and I give all three values to get_or_create,
> it works
>
> if the instance exists, and I give values for slno and mode which are different
> from the existing ones - I get a programming error as get_or_create tries to
> create a new instance and falls foul of the unique constraint on name. It
> works if I only give the value for name - it gets the instance and then I can
> add the new values and save
>
> if the instance does not exist and I only give the value for name, I get a
> programming error that the not null constraints on slno and mode are violated.
>
> so it looks to me that get_or_create will only work reliably when there are no
> not null or unique constraints in the model. Or am I missing something?
I think you are missing the optional kwarg 'initial', which is a
dictionary of initial values to use when the instance does not exist.
So not
foo, created = Foo.objects.get_or_create(name=name,
sino=sino,mode=mode)
but
foo, created = Foo.objects.get_or_create(name=name,
initial={'sino': sino, 'mode': mode, })
Cheers
Tom
Daniel Roseman
Actually the kwarg is 'defaults', not 'initial'.
See http://docs.djangoproject.com/en/1.1/ref/models/querysets/#get-or-create-kwargs
--
DR.
Tom Evans
>
> Actually the kwarg is 'defaults', not 'initial'.
>
> See http://docs.djangoproject.com/en/1.1/ref/models/querysets/#get-or-create-kwargs
> --
> DR.
>
Indeed :/
coffee++
Cheers
Tom
Malcolm Tredinnick
> hi,
>
> I have a model like this:
>
> name - unique
> slno - not null
> mode - not null
>
> If the instance does not exist and I give all three values to get_or_create,
> it works
>
> if the instance exists, and I give values for slno and mode which are different
> from the existing ones - I get a programming error as get_or_create tries to
> create a new instance and falls foul of the unique constraint on name. It
> works if I only give the value for name - it gets the instance and then I can
> add the new values and save
I don't understand what you are saying here. If the instance exists (as
in, the filter parameters passed to get would return a single item),
that instance is returned and the default parameters are never
considered. So the values you pass for slno and mode are irrelevant. If
the equivalent get() call would not return anything, then the instance
does not exist and it's clear that the default parameter must contain
correct data that can be used to create a new object -- including not
violating any unique constraints. Perhaps you could show the code call
you are making, rather than writing it out as a sentence.
In short: get_or_create() first runs the get(). If that returns
something, it is what is returned. If the get() fails with no object
being present, then the instance DOES NOT exist and a new one is
created, with all the same requirements as if you'd created a new object
from scratch (e.g. no unique constraint violations).
Malcolm
Tom Evans
He is describing this situation:
>>> User.objects.get(username='teva...@googlemail.com')
<User: teva...@googlemail.com>
>>> User.objects.get_or_create(username='teva...@googlemail.com')
(<User: teva...@googlemail.com>, False)
>>> User.objects.get(username='teva...@googlemail.com').first_name
u'Tom'
>>> User.objects.get(username='teva...@googlemail.com', first_name='Thomas')
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "....site-packages/django/db/models/manager.py", line 119, in get
return self.get_query_set().get(*args, **kwargs)
File "....django/db/models/query.py", line 305, in get
% self.model._meta.object_name)
DoesNotExist: User matching query does not exist.
>>> User.objects.get_or_create(username='teva...@googlemail.com', first_name='Thomas')
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "....site-packages/django/db/models/manager.py", line 122, in
get_or_create
return self.get_query_set().get_or_create(**kwargs)
File "....site-packages/django/db/models/query.py", line 343, in get_or_create
raise e
IntegrityError: (1062, "Duplicate entry '' for key 2")
IE, where Model.objects.get(**kwargs) does not return an object, but
Model.objects.create(**kwargs) would fail, which is fixed by knowing
about the 'defaults' kwarg to get_or_create:
>>> User.objects.get_or_create(username='teva...@googlemail.com',
... defaults={'first_name': 'Thomas', })
(<User: teva...@googlemail.com>, False)
Cheers
Tom
Malcolm Tredinnick
[...]
>
> He is describing this situation:
>
> >>> User.objects.get(username='teva...@googlemail.com')
> <User: teva...@googlemail.com>
> >>> User.objects.get_or_create(username='teva...@googlemail.com')
> (<User: teva...@googlemail.com>, False)
> >>> User.objects.get(username='teva...@googlemail.com').first_name
> u'Tom'
> >>> User.objects.get(username='teva...@googlemail.com', first_name='Thomas')
> Traceback (most recent call last):
> File "<console>", line 1, in <module>
> File "....site-packages/django/db/models/manager.py", line 119, in get
> return self.get_query_set().get(*args, **kwargs)
> File "....django/db/models/query.py", line 305, in get
> % self.model._meta.object_name)
> DoesNotExist: User matching query does not exist.
> >>> User.objects.get_or_create(username='teva...@googlemail.com', first_name='Thomas')
> Traceback (most recent call last):
> File "<console>", line 1, in <module>
> File "....site-packages/django/db/models/manager.py", line 122, in
> get_or_create
> return self.get_query_set().get_or_create(**kwargs)
> File "....site-packages/django/db/models/query.py", line 343, in get_or_create
> raise e
> IntegrityError: (1062, "Duplicate entry '' for key 2")
No, he's not, because Kenneth said that the instance exists. In your
example, the instance doesn't exist and thus a new object is attempted
to be created. The restrictions at that point are the same as when
creating a new object from scratch, as noted in my reply. The caller of
get_or_create() will always know what the various constraints are on the
model and must allow for that in their code.
Regards,
Malcolm
Tom Evans
<mal...@pointy-stick.com> wrote:
> No, he's not, because Kenneth said that the instance exists. In your
Actually, he described both situations where the instance exists, but
with different values for the non-key attributes of the model, and
where the instance doesn't exist.
pedant++
> example, the instance doesn't exist and thus a new object is attempted
> to be created. The restrictions at that point are the same as when
> creating a new object from scratch, as noted in my reply. The caller of
> get_or_create() will always know what the various constraints are on the
> model and must allow for that in their code.
>
> Regards,
> Malcolm
>
>
Well, quite. His problem was that he wasn't allowing for the
constraints in his code because he was unaware of the 'defaults' kwarg
to get_or_create(). With that addition, his problems when an instance
with the same unique key exists, and when no instance exists are both
fixed.
Cheers
Tom
Kenneth Gonsalves
> I don't understand what you are saying here. If the instance exists (as
> in, the filter parameters passed to get would return a single item),
> that instance is returned and the default parameters are never
> considered. So the values you pass for slno and mode are irrelevant. If
> the equivalent get() call would not return anything, then the instance
> does not exist and it's clear that the default parameter must contain
> correct data that can be used to create a new object -- including not
> violating any unique constraints. Perhaps you could show the code call
> you are making, rather than writing it out as a sentence.
>
> In short: get_or_create() first runs the get(). If that returns
> something, it is what is returned. If the get() fails with no object
> being present, then the instance DOES NOT exist and a new one is
> created, with all the same requirements as if you'd created a new object
> from scratch (e.g. no unique constraint violations).
>
sorry for not replying to this - for some reason all your recent mails have
gone to the spam folder! I only became aware of this mail when I saw a reply
to it. Here is what I mean - A simple model with two fields, 'name' is unique,
and 'num' is not null:
>>> Tm.objects.get_or_create(name='me',num=1)
(<Tm: me>, True)
>>> Tm.objects.get_or_create(name='me',num=2)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/lib/python2.6/django/db/models/manager.py", line 135, in
get_or_create
return self.get_query_set().get_or_create(**kwargs)
File "/usr/lib/python2.6/django/db/models/query.py", line 375, in
get_or_create
transaction.savepoint_rollback(sid, using=self.db)
File "/usr/lib/python2.6/django/db/transaction.py", line 242, in
savepoint_rollback
connection._savepoint_rollback(sid)
File "/usr/lib/python2.6/django/db/backends/__init__.py", line 61, in
_savepoint_rollback
self.cursor().execute(self.ops.savepoint_rollback_sql(sid))
File "/usr/lib/python2.6/django/db/backends/__init__.py", line 75, in cursor
cursor = self._cursor()
File "/usr/lib/python2.6/django/db/backends/postgresql/base.py", line 149,
in _cursor
cursor.execute("SET client_encoding to 'UNICODE'")
ProgrammingError: ERROR: current transaction is aborted, commands ignored
until end of transaction block
SET client_encoding to 'UNICODE'
Karen Tracey
Here is what I mean - A simple model with two fields, 'name' is unique,
and 'num' is not null:
>>> Tm.objects.get_or_create(name='me',num=1)
(<Tm: me>, True)
>>> Tm.objects.get_or_create(name='me',num=2)
If you want get_or_create to return the instance with name 'me', regardless of the value of num which that instance may have, then you need to include the num value in the defaults argument to get_or_create, NOT as a bare keyword argument. From http://docs.djangoproject.com/en/dev/ref/models/querysets/#get-or-create-kwargs:
"Any keyword arguments passed to get_or_create() -- except an optional one called defaults -- will be used in a get() call."
So with the arguments as you have passed them, get_or_create is looking for a model with name='me' and num=2, and since one does not exist, it is going to go down the path of attempting to create an instance with those values. Since you have specified at the database level that name must be unique, that is not going to work: attempting to create a 2nd row with an already-existing name value is going to raise an IntegrityError.
To accomplish what you want, you need to move num into the defaults argument, so that it will not be used on the get() lookup, but will be used if the get fails to find a match and an object needs to be created. That is:
Tm.objects.get_or_create(name='me', defaults={'num': 2})
Traceback (most recent call last):
[snip]
File "/usr/lib/python2.6/django/db/backends/__init__.py", line 61, in
_savepoint_rollback
self.cursor().execute(self.ops.savepoint_rollback_sql(sid))
File "/usr/lib/python2.6/django/db/backends/__init__.py", line 75, in cursor
cursor = self._cursor()
File "/usr/lib/python2.6/django/db/backends/postgresql/base.py", line 149,
in _cursor
cursor.execute("SET client_encoding to 'UNICODE'")
ProgrammingError: ERROR: current transaction is aborted, commands ignored
until end of transaction block
SET client_encoding to 'UNICODE'
Here you are running into a bug in the postgresql backend. Is there some reason you are using the old psycopg1 backend instead of the psycopg2 one? The latter is really a much better choice at this point.
Anyway, what has happened here is the get_or_create code has gone down the path of trying to create the new object, which has failed because of the unique constraint on name. It is trying to rollback to the previously-created savepoint, but that is failing due to a bug that was fixed yesterday in r12848 (trunk) and r12849 (1.1.X branch). So to avoid this you could either simply switch to the psycopg2 backend (which did not have the bug) or update to a latest trunk or 1.1.X Django code.
Note when you fix the call to get_or_create to properly specify which of the values must be used on the get() part of get_or_create(), you won't be going down this path as you are now, so you likely won't see this exception during basic testing even if you don't change backends or update to latest Django code. However, you could still go down this path legitimately -- if two threads "simultaneously" attempt to create an object with the same name, both may see that it does not exist and try to create it. Only one will succeed, and the other will go down this path. You'll need this path to work properly in order for the 2nd thread to be able to retrieve the object created by the other thread. So even if you notice that fixing the get_or_create call makes this other exception go away, you should still update the code you are using, if you want the call to behave correctly in all situations.
Karen
Kenneth Gonsalves
read the documentation, but did not absorb it properly. Use of pyscopg1 is due
to inertia. When I was first using django I had a lot of legacy code using
psycopg1 and it was too much of a bother to migrate. That code is no longer my
responsibility now, so I will probably shift. Am top posting as there is
nothing specific to reply to. Thanks again.
On Friday 26 Mar 2010 4:55:39 pm Karen Tracey wrote:
> On Fri, Mar 26, 2010 at 1:49 AM, Kenneth Gonsalves <law...@au-kbc.org>wrote:
> > Here is what I mean - A simple model with two fields, 'name' is unique,
> >
> > and 'num' is not null:
> > >>> Tm.objects.get_or_create(name='me',num=1)
> >
> > (<Tm: me>, True)
> >
> > >>> Tm.objects.get_or_create(name='me',num=2)
>
> If you want get_or_create to return the instance with name 'me', regardless
> of the value of num which that instance may have, then you need to include
> the num value in the defaults argument to get_or_create, NOT as a bare
> keyword argument. From
> http://docs.djangoproject.com/en/dev/ref/models/querysets/#get-or-create-kw
> args
>
>
--
Karen Tracey
thanks for the detailed reply - I now understand the issue. And yes, I did
read the documentation, but did not absorb it properly.
A number of people have tripped over this sort of use of get_or_create; there is an open ticket (#12579) to improve the doc here.