Circularly dependent fixtures fail with Postgres 9.2

382 views
Skip to first unread message

Yo-Yo Ma

unread,
Jun 16, 2013, 6:22:16 PM6/16/13
to django-d...@googlegroups.com
There doesn't appear to be a way to load fixtures from JSON (using Postgres - works fine in sqlite3) for the following models:


class Collection(models.Model):
    main_thing = models.OneToOneField(
        'things.Thing',
        null=True,
        on_delete=models.SET_NULL
    )

class Thing(models.Model):
    collection = models.ForeignKey(
        'collections.Collection'
    )


Here is the exception:

Problem installing fixture 'my_fixture.json': Could not load collections.Collection(pk=1): insert or update on table "collections_collection" violates foreign key constraint "main_thing_id_refs_id_3a4d3fef"
DETAIL:  Key (main_thing_id)=(1) is not present in table "things_thing".

I'm not sure if the issue is due to the unique constraint implied by a OneToOneField, or if it's just related to this issue: https://code.djangoproject.com/ticket/3615 (seems like that ticket and related ones have been closed for years, so possibly not related).

Any thoughts?

Note: I'm using @1.6a1

Russell Keith-Magee

unread,
Jun 16, 2013, 7:40:02 PM6/16/13
to django-d...@googlegroups.com

Circular dependencies *shouldn't* be a problem on PostgreSQL because all constraints are set DEFERABLE INITIALLY DEFERRED; that means no constrain checks should be performed are performed until the transaction boundary, so all circular references shouldn't be a problem. 

Ticket #3615 exists because MySQL's implementation of DEFERABLE INITIALLY DEFERRED under InnoDB is, to use the technical term, "Broken". It's unrelated to any problem you may have found in PostgreSQL, because PostgreSQL gets the underlying behaviour right. 

Beyond that, we need a specific test case to take this any further. As it stands, I'm not aware of any problems loading fixtures into PostgreSQL. If you are able to construct and provide a set of models (which you have done) and simple fixture (which you haven't) that fails reliably, we have a new bug on our hands, and you should open a ticket with all the details you can provide. Confirming whether this is a problem with the alpha, or an ongoing problem would also be helpful.

Yours,
Russ Magee %-)



--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Yo-Yo Ma

unread,
Jun 21, 2013, 4:32:33 PM6/21/13
to django-d...@googlegroups.com
Hi Russel,

Thanks for taking the time to explain that. I tried that same day to reproduce the issue in a testing env with the simplified models I typed above, but my hosting provider had some erroneous networking nonsense that ruined my test after I spent a couple hours setting everything up. I figured I'm come back to it... and here I am.

I didn't set up an entire test env and test app this time, just made a fresh database and ran my apps fixtures on it, but I did test my app again, using a fresh database without any data. The models and fixtures for which are as follows (minus most of the decimals, chars, and other non-FK-type fields, none of which should be related to this problem):


# account/models.py
class Account(models.Model):
    name = models.CharField(_(u'name'), max_length=255)


# orders/models.py
class Order(models.Model):
    account = models.ForeignKey('account.Account', verbose_name=_(u'account'))
    number = models.IntegerField(_(u'number'))
    bill_address = models.OneToOneField(
        'orders.OrderAddress',
        null=True,
        on_delete=models.SET_NULL,
        related_name='bill_address_order',
        verbose_name=_(u'bill to address')
    )

class OrderAddress(models.Model):
    account = models.ForeignKey('account.Account', verbose_name=_(u'account'))
    order = models.ForeignKey('orders.Order', verbose_name=_(u'order'))
    country = models.CharField(_(u'country'), max_length=2)


// orders/fixtures/test_data.json
[
    {
        "model": "orders.order",
        "pk": 1,
        "fields": {
            "account": 1,
            "number": 1,
            "bill_address": 1
        }
    },
    {
        "model": "orders.orderaddress",
        "pk": 1,
        "fields": {
            "account": 1,
            "order": 1,
            "country": "US",
        }
    }
]


(an Account with the primary key of 1 already exists at the time of ``loaddata``)


The error I get with `python manage.py loaddata test_data orders` is:

django.db.utils.IntegrityError: Problem installing fixture '/opt/myproject/apps/orders/fixtures/test_data.json': Could not load orders.Order(pk=1): insert or update on table "orders_order" violates foreign key constraint "bill_address_id_refs_id_3a4d3fef"
DETAIL:  Key (bill_address_id)=(1) is not present in table "orders_orderaddress".


The above fixtures load locally, and they load during test running (with Postgres) a number of times, but for some reason I get that error when using `manage.py loaddata ...`.

Yo-Yo Ma

unread,
Jun 21, 2013, 4:34:14 PM6/21/13
to django-d...@googlegroups.com
Pardon one typo: I meant `python manage.py loaddata test_data` in my previous post.

Yo-Yo Ma

unread,
Jun 23, 2013, 7:24:40 PM6/23/13
to django-d...@googlegroups.com
Hi again Russell,

I did a little digging. I'm not sure, but I may have uncovered the problem. A transaction block (using `commit_on_success_unless_managed`) is entered and exited during each fixture object loaded, due to the calls to the aforementioned method that exist in various model methods (namely, `save_base`, in this case). Because of this, the transaction is committed immedately after each object is loaded, despite the attempt to wrap `commit_on_success_unless_managed` around the context of the `loaddata` call in the management command.

The following are the results of my placing print statements (I know that's old-school - pdb is just too time consuming) inside `commit_on_success_unless_managed`. In each call, I added:

    print 'AUTOCOMMIT', connection.autocommit
    print 'IN ATOMIC BLOCK', connection.in_atomic_block
    for frame in inspect.stack():
        print frame[1], frame[3], frame[2]

as well as a print after the stack of whether atomic() was returned or _transaction_func() was returned (for easier reading):


AUTOCOMMIT False
IN ATOMIC BLOCK False

django/db/transaction.py commit_on_success_unless_managed 492
django/core/management/commands/loaddata.py handle 53
django/core/management/base.py execute 283
django/core/management/base.py run_from_argv 240
django/core/management/__init__.py execute 392
django/core/management/__init__.py execute_from_command_line 399
manage.py <module> 10

----RETURNING TRANSACTION FUNC

===========================================================

AUTOCOMMIT False
IN ATOMIC BLOCK False

django/db/transaction.py commit_on_success_unless_managed 492
django/db/models/base.py save_base 573
django/core/serializers/base.py save 165
django/core/management/commands/loaddata.py process_dir 225
django/core/management/commands/loaddata.py load_label 169
django/core/management/commands/loaddata.py loaddata 102
django/core/management/commands/loaddata.py handle 54
django/core/management/base.py execute 283
django/core/management/base.py run_from_argv 240
django/core/management/__init__.py execute 392
django/core/management/__init__.py execute_from_command_line 399
manage.py <module> 10

----RETURNING TRANSACTION FUNC

===========================================================

SAVEPOINT False
AUTOCOMMIT True
IN ATOMIC BLOCK False

|||||||||||||||||||||||||||||||||||||||||||||||
django/db/transaction.py commit_on_success_unless_managed 492
django/db/models/base.py save_base 573
django/core/serializers/base.py save 165
django/core/management/commands/loaddata.py process_dir 225
django/core/management/commands/loaddata.py load_label 169
django/core/management/commands/loaddata.py loaddata 102
django/core/management/commands/loaddata.py handle 54
django/core/management/base.py execute 283
django/core/management/base.py run_from_argv 240
django/core/management/__init__.py execute 392
django/core/management/__init__.py execute_from_command_line 399
manage.py <module> 10

----RETURNING ATOMIC

===========================================================


The remaining calls were exactly like call 3 (including "IN ATOMIC BLOCK False", despite the 3rd call having returned `atomic()`). My prima facie opinion is that `with atomic()` is needed in `loaddata`, instead of `with commit_on_success_unless_managed`, since the latter acts funky when nested calls occur (as see in save_base in the stacks printed above). However, the issue might be something that needs to be resolved in the transitioning-to-atomic code. I don't fully understand all of this yet, but it's a start.

Yo-Yo Ma

unread,
Jun 23, 2013, 7:35:45 PM6/23/13
to django-d...@googlegroups.com
Minor correction: I placed Atomic.__exit__ to verify - the transaction is commited every time *starting* on the second object (the third stack printed in the previous post) - it happens at https://github.com/django/django/blob/master/django/db/transaction.py#L288.

Yo-Yo Ma

unread,
Jun 25, 2013, 4:30:17 PM6/25/13
to django-d...@googlegroups.com
I should actually note, this bug affects all versions of Postgres, and presumably all other supported RDBMSs as well.

Aymeric Augustin

unread,
Jun 25, 2013, 4:59:27 PM6/25/13
to django-d...@googlegroups.com
Would you mind checking if the bug occurs in Django 1.5? If it doesn't, it's a regression introduced by the new transaction management in Django 1.6, and it's a release blocker.

-- 
Aymeric.


Yo-Yo Ma

unread,
Jun 25, 2013, 8:43:16 PM6/25/13
to django-d...@googlegroups.com
Hi Aymeric,

It does happen in 1.5, but the error is somehow slightly different (no traceback in 1.5 to find the root cause). I have 2 fields on the order model pointing to the address model. I included only the one in my above example because it was consistently the culprit (likely due to being defined above the other). In 1.5 the *other* field is consistently the culprit:

IntegrityError: Problem installing fixtures: insert or update on table "orders_order" violates foreign key constraint "shipping_rate_id_refs_id_84a732cf"
DETAIL:  Key (shipping_rate_id)=(2) is not present in table "shipping_shippingrate".

The error on 1.6 again is:


django.db.utils.IntegrityError: Problem installing fixture '/opt/myproject/apps/orders/fixtures/test_data.json': Could not load orders.Order(pk=1): insert or update on table "orders_order" violates foreign key constraint "bill_address_id_refs_id_3a4d3fef"
DETAIL:  Key (bill_address_id)=(1) is not present in table "orders_orderaddress".


Aymeric Augustin

unread,
Jun 26, 2013, 2:27:03 AM6/26/13
to django-d...@googlegroups.com
Thanks for taking the time to check. Based on Russell's message, I think that's a bug, but I'm not sure why it happens. Could you file a ticket in the tracker, so we don't forget about it?

Thank you,

--
Aymeric.

Yo-Yo Ma

unread,
Jun 27, 2013, 10:42:30 AM6/27/13
to django-d...@googlegroups.com
Aymeric,

No problem. Thanks for the replies. I just created the ticket @ https://code.djangoproject.com/ticket/20666

Yo-Yo Ma

unread,
Jun 28, 2013, 2:09:13 PM6/28/13
to django-d...@googlegroups.com
Aymeric,

Somebody (anonymous) said in the ticket that they had a problem in 1.6 and not in 1.5. I had a similar error on both, but there might be unrelated issues in both versions. Would that make it a release blocker? Sorry for not posting a reply on here sooner (the beta release today).


On Wednesday, June 26, 2013 2:27:03 AM UTC-4, Aymeric Augustin wrote:

Russell Keith-Magee

unread,
Jun 28, 2013, 6:33:14 PM6/28/13
to django-d...@googlegroups.com

If you can confirm that the problem exists on 1.6, but doesn't exist on 1.5, then yes, it is a release blocker. 

However, as I understand, you were reporting the same problem in 1.5. If it's the same problem, or a different error rising from the same root cause, then no, it isn't a release blocker.

Yours,
Russ Magee %-)

Reply all
Reply to author
Forward
0 new messages