Model's Foreign Key Id Attribute not being updated

[Michael Viveros]

I have a Question model and a Choice model where a Question can have many Choices so the Choice model has a Foreign Key mapping to a Question.

models.py:

from django.db import models

class Question(models.Model):

    text = models.CharField(max_length=200)

    title = models.CharField(max_length=200)

    total_response_count = models.IntegerField()

    topic = models.CharField(max_length=20)

    sequence_num = models.IntegerField()

    select_multiple = models.BooleanField(default=False)

    def __str__(self):

        return self.text

class Choice(models.Model):

    question = models.ForeignKey(Question, on_delete=models.CASCADE)

    text = models.CharField(max_length=200)

    response_count = models.IntegerField()

    response_percent = models.FloatField()

    def __str__(self):

        return self.text

I need to parse a csv file containing these models and insert them into a database. 

I want to be efficient and bulk create these models instead of having to call save() after creating each model.

I can bulk create the Questions but then when I try to bulk create the Choices I get an IntegrityError about the Choice's foreign key id attribute ("question_id") violating a not-null constraint.

After creating the Questions, the Question's id attribute gets updated but the Choice's question_id attribute does not get updated and it does not get re-evaluated when the Choice is created.

Here's a simplified version of what I want to do:

from censustest.models import Question,Choice

q=Question(text="Q1",title="",topic="",sequence_num=0,total_response_count=0)

c=Choice(text="C1",response_count=5,response_percent=0.3,question=q)

print("q.id %s, c.question %s, c.question_id %s" % (q.id, c.question, c.question_id)) # None,Q1,None

Question.objects.bulk_create([q])

print("After creating q: q.id %s, c.question %s, c.question_id %s" % (q.id, c.question, c.question_id)) # valid id,Q1,None

Choice.objects.bulk_create([c]) # ERROR: IntegrityError

Output:

Is this a bug with Django (the foreign key id attribute not being updated) or is there a better way to do it?

An alternative approach I thought of was to assign the Choice's Question after creating the Questions and this works.

from censustest.models import Question,Choice

q=Question(text="Q1",title="",topic="",sequence_num=0,total_response_count=0)

c=Choice(text="C1",response_count=5,response_percent=0.3)

print("q.id %s" % (q.id)) # None,Q1,None

Question.objects.bulk_create([q])

c.question=q

print("c.question %s, c.question_id %s" % (c.question, c.question_id)) # Q1,valid id

Choice.objects.bulk_create([c]) # works, no error

Output:

[C. Kirby]

The reason it isn't working in your first example is because the pk gets assigned to the object when it is saved to the database.  When you supply question=q to the choice, it is (correctly) getting a pk of null because q has not been saved.

[Michael Viveros]

Thanks for the reply Kirby.

I understand that the Question's pk is null when you pass in the Question to the Choice constructor. But when you save the Choice, the Question was saved earlier so the Question's pk is not null. But the Choice's question_id attribute (representing the Question's pk) is still null when you save the Choice. 

1. Should the Choice's question_id have been updated to it's current value when saving the Choice? 

I read some more about what happens when you save an object in Django and it seems like the Choice's question_id should be updated when the data is prepared for the db (step 3 from the link). 

I understand I can override the Choice class's save method to do this (see code below) but it seems like it should be something that is already done by the Django framework.

    def save(self, *args, **kwargs):
        self.question_id = self.question.id
        super(Choice, self).save(*args, **kwargs)

2. Is there a better way to bulk create objects in a one-to-many relationship (Questions and Choices)?

My idea (from the code snippet after models.py) was to bulk create the Questions and then bulk create the Choices but this has the problem mentioned above of the Choice's FK (question_id) not being updated even after the Questions were bulk created.

[C. Kirby]

On Monday, November 28, 2016 at 3:12:37 PM UTC+2, Michael Viveros wrote:

Thanks for the reply Kirby.

I understand that the Question's pk is null when you pass in the Question to the Choice constructor. But when you save the Choice, the Question was saved earlier so the Question's pk is not null. But the Choice's question_id attribute (representing the Question's pk) is still null when you save the Choice. 

1. Should the Choice's question_id have been updated to it's current value when saving the Choice? 

No. The preparing you linked handles type changes between python and the db, if the db cannot handle the type you have (like the referenced python datetime).
What is happening is no different than:

>>> a = 1
>>> b = a
>>> a = 2
>>> b
1
>>> a
2

I read some more about what happens when you save an object in Django and it seems like the Choice's question_id should be updated when the data is prepared for the db (step 3 from the link). 

I understand I can override the Choice class's save method to do this (see code below) but it seems like it should be something that is already done by the Django framework.

    def save(self, *args, **kwargs):
        self.question_id = self.question.id
        super(Choice, self).save(*args, **kwargs)

This won't work for 1 or 2 reasons, depending on versions. save() does not get called when using bulk_create(). Also, unless you are using postgres and django 1.10, pk's get saved to the db, but not assigned to the object when using bulk_create()

2. Is there a better way to bulk create objects in a one-to-many relationship (Questions and Choices)?

My idea (from the code snippet after models.py) was to bulk create the Questions and then bulk create the Choices but this has the problem mentioned above of the Choice's FK (question_id) not being updated even after the Questions were bulk created.

You pretty much can't do it inline like you want with bulk_create(), I don't think. If you have the question objects retrieved you can then assign them to the choices and you are all set. If you really want to do it how you are trying then your best bet would be to assign a primary_key field instead of using the AutoField primary key provided. If you use the question title, or generate a uuid for each question and set that as the primary_field then you can provide the primary key that you know for a question to the associated choices.

[Michael Viveros]

Thanks for the detailed responses, they helped clarify things a lot.

I will try the first option you mentioned about creating the Questions first (and saving them) and then creating the Choices using the Questions that were just created.

My back-up plan will be to use a different pk field for Question (like question text) that will be already known when I create Choices for a question.

[Rowan Seymour]

I ran into the same problem today in a script for creating large amounts of test data. It would be awkward to re-organize the code it such a way that related objects are assigned to fields only after they've been saved, so I made a utility method to a take a model instance which has fields which might be assigned related objects before they are saved, and "fix" those fields after the related objects have been saved:

def _resync_fk_fields(self, obj):
    for field in type(obj)._meta.local_concrete_fields:
        if isinstance(field, models.ForeignKey):
            rel_obj = getattr(obj, field.name)
            if rel_obj is not None:
                setattr(obj, field.get_attname(), rel_obj.id)

Seems to do the trick

-Rowan