[Django] #29574: Django Foreign Key Mismatch

[Django]

#29574: Django Foreign Key Mismatch
-----------------------------------------+------------------------
Reporter: josephbiko | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 2.1
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
Django Foreign key's do not track the models DB when the model is
inherited from abstract class.

{{{
class A(Model):
class Meta:
abstract = True
class B(A):
pass
class C(Model):
fk = foreignkey(B,on_delete=models.CASCADE)
}}}

now model B is in table 1 and C points to table 1.
{{{
class A(Model):
pass
class B(A):
pass
class C(Model):
fk = foreignkey(B,on_delete=models.CASCADE)
}}}
Now Model B is in table 2 with Model A, however C is still pointing to
table 1, which leads to a foreign key mismatch error when you try to add
an instance of B

--
Ticket URL: <https://code.djangoproject.com/ticket/29574>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#29574: Django Foreign Key Mismatch

-------------------------------+--------------------------------------
Reporter: josephbiko | Owner: josephbiko
Type: Uncategorized | Status: assigned
Component: Uncategorized | Version: 2.1
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------+--------------------------------------
Changes (by josephbiko):

* owner: nobody => josephbiko
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:1>

[Django]

#29574: Django Foreign Key Mismatch

-------------------------------+--------------------------------------
Reporter: josephbiko | Owner: (none)

Type: Uncategorized | Status: new
Component: Uncategorized | Version: 2.1

Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------+--------------------------------------
Changes (by josephbiko):

* owner: josephbiko => (none)
* status: assigned => new

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:2>

[Django]

#29574: Django Foreign Key Mismatch

----------------------------+--------------------------------------
Reporter: josephbiko | Owner: (none)

Type: Bug | Status: closed
Component: Migrations | Version: 2.1
Severity: Normal | Resolution: needsinfo

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+--------------------------------------
Changes (by Simon Charette):

* status: new => closed
* resolution: => needsinfo
* component: Uncategorized => Migrations
* type: Uncategorized => Bug

Comment:

Hello josephbiko,

I assume you meant that running `makemigrations` between your two model
scenarios doesn't generate the appropriate database alterations?

There's a few bugs (#23521, #25247) related to switching from concrete to
abstract inheritance but in the abstract to concrete case there shouldn't
be any problem as `C.fk`should be automatically repointed to `B.a_ptr`
when `A` is made concrete.

I'm afraid we'll need a more detailed reproduction test case in order to
reproduce your issue. Please include tracebacks and detail every steps
your perform to get there.

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:3>

[Django]

#29574: Django Foreign Key Mismatch

----------------------------+--------------------------------------
Reporter: josephbiko | Owner: (none)
Type: Bug | Status: closed
Component: Migrations | Version: 2.1
Severity: Normal | Resolution: needsinfo

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+--------------------------------------

Comment (by Simon Charette):

Upon further inspection I assume you had an existing `B` entry before
attempting to run the migration that adds the `B.a_ptr` primary key and it
failed because no rows were present in the newly added `A` table?

In this case I think you should be in charge of populating `A` with a data
migration (either using `RunPython` or `RunSQL`) instead of expecting
Django to do it for you.

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:4>

[Django]

#29574: Django Foreign Key Mismatch

----------------------------+--------------------------------------
Reporter: josephbiko | Owner: (none)

Type: Bug | Status: new

Component: Migrations | Version: 2.1
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+--------------------------------------
Changes (by josephbiko):

* cc: josephbiko (added)
* status: closed => new
* resolution: needsinfo =>

Old description:

> Django Foreign key's do not track the models DB when the model is
> inherited from abstract class.
>

> {{{
> class A(Model):
> class Meta:
> abstract = True
> class B(A):
> pass
> class C(Model):
> fk = foreignkey(B,on_delete=models.CASCADE)
> }}}
>
> now model B is in table 1 and C points to table 1.
> {{{
> class A(Model):
> pass
> class B(A):
> pass
> class C(Model):
> fk = foreignkey(B,on_delete=models.CASCADE)
> }}}
> Now Model B is in table 2 with Model A, however C is still pointing to
> table 1, which leads to a foreign key mismatch error when you try to add
> an instance of B

New description:

Django Foreign key's do not track the models DB when the model is
inherited from abstract class.

Steps to reproduce:

create models:"

{{{
class A(models.Model):
field = models.IntegerField()

class Meta:
abstract = True

class B(A):
field2 = models.IntegerField()

class C(models.Model):
fk = models.ForeignKey(B,on_delete=models.CASCADE)
}}}

migrate

change the models (by removeing the meta )to:
{{{
class A(models.Model):
field = models.IntegerField()

class B(A):
field2 = models.IntegerField()

class C(models.Model):
fk = models.ForeignKey(B,on_delete=models.CASCADE)
}}}

run migrations.

Now add an object of class B in the admin.

resulting error:
"foreign key mismatch - "models_c" referencing "models_b""

--

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:5>

[Django]

#29574: Django Foreign Key Mismatch

----------------------------+--------------------------------------
Reporter: josephbiko | Owner: (none)
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+--------------------------------------

Comment (by josephbiko):

Replying to [comment:4 Simon Charette]:

> Upon further inspection I assume you had an existing `B` entry before
attempting to run the migration that adds the `B.a_ptr` primary key and it
failed because no rows were present in the newly added `A` table?
>
> In this case I think you should be in charge of populating `A` with a
data migration (either using `RunPython` or `RunSQL`) instead of expecting
Django to do it for you.

No there was no data present before

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:6>

[Django]

#29574: Unable to create model instance after changing an abstract model to non-
abstract due to "foreign key mismatch" error
----------------------------+------------------------------------

Reporter: josephbiko | Owner: (none)
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+------------------------------------
Changes (by Tim Graham):

* stage: Unreviewed => Accepted

Comment:

Reproduced at 1e9b02a4c28142303fb4d33632e77ff7e26acb8b.

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:7>

[Django]

#29574: Unable to create model instance after changing an abstract model to non-
abstract due to "foreign key mismatch" error
----------------------------+------------------------------------
Reporter: josephbiko | Owner: (none)
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+------------------------------------

Comment (by Bhuvnesh):

hi, i was working around to find the real cause of this issue and i am
writing my observation here but i am not sure if they are fully correct or
not, it would be really helpful if someone could cross-check/confirm.
1. when {{{a_ptr}}} is created its a primary key hence has UNIQUE
constraint.
2. since {{{a_ptr}}} is a non nullable field it asks for a default value
to populate the existing rows(even for empty db). And as we provide
default value the UNIQUE constraint fails.
3. Now as UNIQUE constraints failed , it defies one of the rule of foreign
key constraints given [https://www.sqlite.org/foreignkeys.html here]
(Under the heading Required and Suggested Database Indexes) which says
{{{The parent key columns named in the foreign key constraint are not the
primary key of the parent table and are not subject to a unique constraint
using collating sequence specified in the CREATE TABLE}}}. This
subsequently leads to the {{{"foreign key mismatch - "models_c"
referencing "models_b""}}} error.

I'm using django dev version and its giving me error as soon as i migrate
second time.

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:8>

[Django]

#29574: Unable to create model instance after changing an abstract model to non-
abstract due to "foreign key mismatch" error
----------------------------+------------------------------------
Reporter: josephbiko | Owner: (none)
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+------------------------------------

Comment (by Bhuvnesh):

Ok, so i have found the cause (the one above is incorrect):
1. When we run migration for the first time 2 db tables are created (B and
C) , in which {{{C.fk}}} points to primary key of B i.e {{{id}}} .
2. Now when we run second migration, table A is created and B is recreated
with field {{{a_ptr}}} as primary key and all the old fields of B are
deleted(even{{{id}}}). No changes for table C.
3. Since {{{c.fk}}} still points to {{{b.id}}} (at physical level), it
throws {{{foreign_key_mismatch}}} error.
4. If we explicity define primary key in model B everything works as
expected.
{{{
bid = models.IntegerField(primary_key=True)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:9>

[Django]

#29574: Unable to create model instance after changing an abstract model to non-
abstract due to "foreign key mismatch" error
----------------------------+------------------------------------

Reporter: josephbiko | Owner: Bhuvnesh
Type: Bug | Status: assigned

Component: Migrations | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+------------------------------------
Changes (by Bhuvnesh):

* owner: (none) => Bhuvnesh

* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:10>

[Django]

#29574: Unable to create model instance after changing an abstract model to non-
abstract due to "foreign key mismatch" error
----------------------------+------------------------------------
Reporter: josephbiko | Owner: Bhuvnesh
Type: Bug | Status: assigned
Component: Migrations | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+------------------------------------

Comment (by Bhuvnesh):

WIP PR https://github.com/django/django/pull/16325

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:11>

[Django]

#29574: Unable to create model instance after changing an abstract model to non-
abstract due to "foreign key mismatch" error
----------------------------+------------------------------------

Reporter: josephbiko | Owner: (none)
Type: Bug | Status: new

Component: Migrations | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+------------------------------------
Changes (by Bhuvnesh):

* owner: Bhuvnesh => (none)

* status: assigned => new

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:12>

[Django]

#29574: Unable to create model instance after changing an abstract model to non-
abstract due to "foreign key mismatch" error

----------------------------+-------------------------------------
Reporter: josephbiko | Owner: Calvin Vu
Type: Bug | Status: assigned

Component: Migrations | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+-------------------------------------
Changes (by Calvin Vu):

* cc: Calvin Vu (added)
* owner: (none) => Calvin Vu

* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:13>

[Django]

#29574: Unable to create model instance after changing an abstract model to non-
abstract due to "foreign key mismatch" error
----------------------------+-------------------------------------
Reporter: josephbiko | Owner: Calvin Vu
Type: Bug | Status: assigned
Component: Migrations | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+-------------------------------------
Comment (by Calvin Vu):

I have reproduced the {{{ foreign key mismatch - "testapp_c" referencing
"testapp_b" }}} error but with slightly different steps:
1. Add the models to models.py
{{{

class A(models.Model):
field = models.IntegerField()

class Meta:
abstract = True

class B(A):

field2 = models.IntegerField()

class C(models.Model):
fk = models.ForeignKey(B,on_delete=models.CASCADE)
}}}

2. Make migrations and migrate
3. Make the A model concrete (by removing its Meta)
{{{

class A(models.Model):
field = models.IntegerField()

class B(A):
field2 = models.IntegerField()

class C(models.Model):
fk = models.ForeignKey(B, on_delete=models.CASCADE)
}}}

4. Make migrations which will present the following output:
{{{
It is impossible to add a non-nullable field 'a_ptr' to b without
specifying a default. This is because the database needs something to
populate existing rows.
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows with a
null value for this column)
2) Quit and manually define a default value in models.py.
Select an option:
}}}
5. Go with option 1 and provide an integer (I provided 0)
6. Migrate and you'll see the error:
{{{
foreign key mismatch - "testapp_c" referencing "testapp_b"
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:14>

[Django]

#29574: Unable to create model instance after changing an abstract model to non-
abstract due to "foreign key mismatch" error
----------------------------+-------------------------------------
Reporter: josephbiko | Owner: Calvin Vu
Type: Bug | Status: assigned
Component: Migrations | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+-------------------------------------
Comment (by Calvin Vu):

It seems like Bhuvnesh correctly identified the cause. A potential
solution would be to first set `C.fk` to `A.id` (which is going to be the
`id` of model 'AB', i.e., the separate tables A and B, which are connected
one-to-one with `B.a_ptr_id` when model A is made concrete), then change
model B to inherit from model A (to form model 'AB'), and then update any
existing data.
--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:15>

[Django]

#29574: Unable to create model instance after changing an abstract model to non-
abstract due to "foreign key mismatch" error
----------------------------+-------------------------------------
Reporter: josephbiko | Owner: Calvin Vu
Type: Bug | Status: assigned
Component: Migrations | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+-------------------------------------
Comment (by Calvin Vu):

Replying to [comment:15 Calvin Vu]:

> It seems like Bhuvnesh correctly identified the cause. A potential
solution would be to first set `C.fk` to `A.id` (which is going to be the
`id` of model 'AB', i.e., the separate tables A and B, which are connected
one-to-one with `B.a_ptr_id` when model A is made concrete), then change
model B to inherit from model A (to form model 'AB'), and then update any

existing data. Alternatively, we could also set `C.fk` to `B.a_ptr_id` as
it is also a primary key.
Nevermind, this solution probably won't work because we need B to inherit
from A to determine exactly which A instance to set C.fk to but when B
inherits from A we lose B.id so there would be no way to link C and 'AB'
--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:16>

[Django]

#29574: Unable to create model instance after changing an abstract model to non-
abstract due to "foreign key mismatch" error

----------------------------+------------------------------------
Reporter: josephbiko | Owner: (none)
Type: Bug | Status: new

Component: Migrations | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+------------------------------------
Changes (by Calvin Vu):

* owner: Calvin Vu => (none)

* status: assigned => new

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:17>

[Django]

#29574: Unable to create model instance after changing an abstract model to non-
abstract due to "foreign key mismatch" error
----------------------------+------------------------------------
Reporter: josephbiko | Owner: (none)
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

----------------------------+------------------------------------
Changes (by Calvin Vu):

* cc: Calvin Vu (removed)

--
Ticket URL: <https://code.djangoproject.com/ticket/29574#comment:18>