[Django] #36061: Custom data migration for M2M with through_fields not working

[Django]

#36061: Custom data migration for M2M with through_fields not working
-------------------------------------+-------------------------------------
Reporter: Brian Nettleton | Type: Bug
Status: new | Component:
| Migrations
Version: 4.2 | Severity: Normal
Keywords: through_fields | Triage Stage:
migration | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
When writing a data migration for a model with a field which uses through
fields the through_fields are not honored in the model retrieved from the
migration's app registry.

Here is an example of such a data migration using the models from the
Django documentation for through_fields. This data migration has an
assert at the end of the forwards function which fails. Note that issuing
the similar instructions in a Django shell works fine, the issue is
specific to retrieving a model in a migration using apps.get_model. The
models used and instructions for recreating the problem are also included
below.


{{{
# Generated by Django 4.2.17 on 2025-01-02 19:35

from django.db import migrations


def forwards(apps, schema_editor):
Person = apps.get_model('groups', 'Person')
Group = apps.get_model('groups', 'Group')
Group._meta.local_many_to_many[0].remote_field.through_field =
("group", "person")
Membership = apps.get_model('groups', 'Membership')

# Initialize some data in the database
sally, _ = Person.objects.get_or_create(name="Sally Forth")
steve, _ = Person.objects.get_or_create(name="Steve Smith")
alice, _ = Person.objects.get_or_create(name="Alice Adams")
grp1, _ = Group.objects.get_or_create(name="Group 1")
grp2, _ = Group.objects.get_or_create(name="Group 2")
admin, _ = Person.objects.get_or_create(name="Administrator")
Membership.objects.get_or_create(
group=grp1,
person=sally,
inviter=admin,
invite_reason="Initial setup via migration"
)
Membership.objects.get_or_create(
group=grp1,
person=steve,
inviter=admin,
invite_reason="Initial setup via migration"
)
Membership.objects.get_or_create(
group=grp1,
person=alice,
inviter=admin,
invite_reason="Initial setup via migration"
)

# Okay, now also put everyone whose name starts with an "S" and is in
Group 1 into Group 2
for s_member in grp1.members.filter(name__startswith="S"):
Membership.objects.get_or_create(
group=grp2,
person=s_member,
inviter=admin,
invite_reason="Initial setup 2: Put the initial 'S' people
from Group 1 into Group 2"
)

print(f"\n{grp2.members.count()=}\n")
assert grp2.members.count() == 2 # ===== FAILS =====


class Migration(migrations.Migration):

dependencies = [
('groups', '0001_initial'),
]

operations = [
migrations.RunPython(forwards),
]

}}}


The models used for this migration are as follows.


{{{
from django.db import models


class Person(models.Model):
name = models.CharField(max_length=50)


class Group(models.Model):
name = models.CharField(max_length=128)
members = models.ManyToManyField(
Person,
through="Membership",
through_fields=("group", "person"),
)


class Membership(models.Model):
group = models.ForeignKey(Group, on_delete=models.CASCADE)
person = models.ForeignKey(Person, on_delete=models.CASCADE)
inviter = models.ForeignKey(
Person,
on_delete=models.CASCADE,
related_name="membership_invites",
)
invite_reason = models.CharField(max_length=64)

}}}


Steps to reproduce:
1. Create a new empty project
2. Create a new empty app called groups and add to INSTALLED_APPS in
project settings.py
3. Add models above to groups/models.py
4. Make initial migrations
5. Run initial migrations
6. Create empty migration for groups app
7. Put migration above into empty migration file
8. Run migrate to reproduce the problem
--
Ticket URL: <https://code.djangoproject.com/ticket/36061>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36061: Custom data migration for M2M with through_fields not working
-------------------------------------+-------------------------------------

Reporter: Brian Nettleton | Owner: (none)

Type: Bug | Status: new
Component: Migrations | Version: 4.2

Severity: Normal | Resolution:
Keywords: through_fields | Triage Stage: Accepted
migration |

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Simon Charette):

* stage: Unreviewed => Accepted

Comment:

I find it hard to believe but `through_fields` never actually worked in
migrations since its introduction in
c627da0ccc12861163f28177aa7538b420a9d310 (#14549) as
`ManyToManyField.deconstruct`
[https://github.com/django/django/blob/c627da0ccc12861163f28177aa7538b420a9d310/django/db/models/fields/related.py#L2023-L2049
never special cased it]. While the lack of support for `through` (which is
by definition more common) was
[https://github.com/django/django/commit/6b078044745ee3c665051a886021c6fd1f6873b6
noticed during the introduction of migrations] `through_fields` was likely
missed as both features landed around the same time in the 1.7 release
cycle.

Brian, given you've already spent time investigating where
`through_fields` is stored (from your provided reproduction case) would
you be interested in submitting a patch? Adjusting
`ManyToManyField.deconstruct` to include `through_fields` if provided like
so

{{{#!diff
diff --git a/django/db/models/fields/related.py
b/django/db/models/fields/related.py
index 9ef2d29024..91018a27ff 100644
--- a/django/db/models/fields/related.py
+++ b/django/db/models/fields/related.py
@@ -1772,6 +1772,8 @@ def deconstruct(self):
kwargs["through"] = self.remote_field.through
elif not self.remote_field.through._meta.auto_created:
kwargs["through"] = self.remote_field.through._meta.label
+ if through_fields := getattr(self.remote_field, "through_fields",
None):
+ kwargs["through_fields"] = through_fields
# If swappable is True, then see if we're actually pointing to
the target
# of a swap.
swappable_setting = self.swappable_setting
}}}

With extra assertions in
[https://github.com/django/django/blob/51df0dff3c4f28016185a9e876ee5b3420712f99/tests/field_deconstruction/tests.py#L498-L550
the associated many-to-many field deconstruction test] should do.
--
Ticket URL: <https://code.djangoproject.com/ticket/36061#comment:1>

[Django]

#36061: Custom data migration for M2M with through_fields not working
-------------------------------------+-------------------------------------
Reporter: Brian Nettleton | Owner: (none)
Type: Bug | Status: new
Component: Migrations | Version: 4.2
Severity: Normal | Resolution:
Keywords: through_fields | Triage Stage: Accepted
migration |

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Brian Nettleton):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36061#comment:2>

[Django]

#36061: Custom data migration for M2M with through_fields not working
-------------------------------------+-------------------------------------
Reporter: Brian Nettleton | Owner: (none)
Type: Bug | Status: new
Component: Migrations | Version: 4.2
Severity: Normal | Resolution:
Keywords: through_fields | Triage Stage: Accepted
migration |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Brian Nettleton):

Ok, my first Django pull request!
https://github.com/django/django/pull/19006

Simon, thanks for the pointers on where to fix this.
--
Ticket URL: <https://code.djangoproject.com/ticket/36061#comment:3>

[Django]

#36061: Custom data migration for M2M with through_fields not working
-------------------------------------+-------------------------------------

Reporter: Brian Nettleton | Owner: Brian
| Nettleton
Type: Bug | Status: assigned

Component: Migrations | Version: 4.2
Severity: Normal | Resolution:
Keywords: through_fields | Triage Stage: Accepted
migration |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* owner: (none) => Brian Nettleton
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/36061#comment:4>

[Django]

#36061: Custom data migration for M2M with through_fields not working
-------------------------------------+-------------------------------------
Reporter: Brian Nettleton | Owner: Brian
| Nettleton
Type: Bug | Status: assigned
Component: Migrations | Version: 4.2
Severity: Normal | Resolution:

Keywords: through_fields | Triage Stage: Ready for
migration | checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Sarah Boyce):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/36061#comment:5>

[Django]

#36061: Custom data migration for M2M with through_fields not working
-------------------------------------+-------------------------------------
Reporter: Brian Nettleton | Owner: Brian
| Nettleton

Type: Bug | Status: closed
Component: Migrations | Version: 4.2
Severity: Normal | Resolution: fixed

Keywords: through_fields | Triage Stage: Ready for
migration | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce <42296566+sarahboyce@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"b13b8684a04d0bc1081104c5973c62c27dc673b0" b13b868]:
{{{#!CommitTicketReference repository=""
revision="b13b8684a04d0bc1081104c5973c62c27dc673b0"
Fixed #36061 -- Added migration support for
ManyToManyField.through_fields.

Added through_fields support to ManyToManyField.deconstruct.
Thanks to Simon Charette for pointers and the review.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36061#comment:6>

[Django]

#36061: Custom data migration for M2M with through_fields not working
-------------------------------------+-------------------------------------
Reporter: Brian Nettleton | Owner: Brian
| Nettleton
Type: Bug | Status: closed
Component: Migrations | Version: 4.2
Severity: Normal | Resolution: fixed
Keywords: through_fields | Triage Stage: Ready for
migration | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"b96e4c04b666b1044555d7a32f64e83cbad57b03" b96e4c04]:
{{{#!CommitTicketReference repository=""
revision="b96e4c04b666b1044555d7a32f64e83cbad57b03"
[5.2.x] Fixed #36061 -- Added migration support for

ManyToManyField.through_fields.

Added through_fields support to ManyToManyField.deconstruct.
Thanks to Simon Charette for pointers and the review.

Backport of b13b8684a04d0bc1081104c5973c62c27dc673b0 from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36061#comment:7>

[Django]

#36061: Custom data migration for M2M with through_fields not working
-------------------------------------+-------------------------------------
Reporter: Brian Nettleton | Owner: Brian
| Nettleton
Type: Bug | Status: closed
Component: Migrations | Version: 4.2
Severity: Normal | Resolution: fixed
Keywords: through_fields | Triage Stage: Ready for
migration | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by tales-aparecida):

Hi,

Could we have this mentioned in the release notes?

(I'm looking for the right place to post about this, but meanwhile here
looked like a great start)
--
Ticket URL: <https://code.djangoproject.com/ticket/36061#comment:8>

[Django]

#36061: Custom data migration for M2M with through_fields not working
-------------------------------------+-------------------------------------
Reporter: Brian Nettleton | Owner: Brian
| Nettleton
Type: Bug | Status: closed
Component: Migrations | Version: 4.2
Severity: Normal | Resolution: fixed
Keywords: through_fields | Triage Stage: Ready for
migration | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

> Could we have this mentioned in the release notes?

No, Django doesn't document bug fixes so if you want to refer to the
changes in a post you can either link to this ticket or the
[https://github.com/django/django/pull/19006 associated MR].
--
Ticket URL: <https://code.djangoproject.com/ticket/36061#comment:9>