[Django] #37234: bulk_create fails when related objects got PKs later

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------
Reporter: Filip Sedlák | Type: Bug
Status: new | Component: Database
| layer (models, ORM)
Version: 6.0 | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
This is a regression betwen Django 5.2 and 6.0.

Here's a short reproducer test that, when added to
`tests/bulk_create/tests.py` fails with an AssertionError within
`bulk_create`.

{{{
@skipUnlessDBFeature("can_return_rows_from_bulk_insert")
def test_pk_from_related_instance_saved_after_init(self):
country = Country(name="Syldavia", iso_two_letter="SW")
related = RelatedModel(country=country)
country.save()
RelatedModel.objects.bulk_create([related])
self.assertEqual(related.country_id, country.pk)
}}}

Failure:

{{{
File "/Users/krab/workspace/django/tests/bulk_create/tests.py", line
901, in test_pk_from_related_instance_saved_after_init
RelatedModel.objects.bulk_create([related])
^^^^^^^^^^^^^^^
File "/Users/krab/workspace/django/django/db/models/manager.py", line
87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^
File "/Users/krab/workspace/django/django/db/models/query.py", line 913,
in bulk_create
assert len(returned_columns) == len(objs_without_pk)
^^^
}}}

It looks like Django used to support this case intentionally, calling
`._prepare_related_fields_for_save(operation_name="bulk_create")` within
`bulk_create` but a recent change breaks that.

I've never contributed to Django, but I'm willing to work on a fix if
you're interested.
--
Ticket URL: <https://code.djangoproject.com/ticket/37234>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------

Reporter: Filip Sedlák | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: 6.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* cc: Simon Charette (added)
* severity: Normal => Release blocker
* stage: Unreviewed => Accepted

Comment:

Thank you for the test and ticket!
Regression in 7d9aab8da0a06787d649762a702e1a518d843a63 (refs #36260)

Marking as a release blocker as this was introduced in the latest feature
release. If you think you have a fix, you're very welcome to raise a PR!
You can review/test a PR from a different contributor. We will try to fix
this before the planned release next week (Tuesday 4th August)
--
Ticket URL: <https://code.djangoproject.com/ticket/37234#comment:1>

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------

Reporter: Filip Sedlák | Owner: Akshat
| Sparsh
Type: Bug | Status: assigned

Component: Database layer | Version: 6.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Akshat Sparsh):

* owner: (none) => Akshat Sparsh
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/37234#comment:2>

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------
Reporter: Filip Sedlák | Owner: Akshat
| Sparsh
Type: Bug | Status: assigned
Component: Database layer | Version: 6.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Akshat Sparsh):

* has_patch: 0 => 1

Comment:

[https://github.com/django/django/pull/21677 PR]

The full test suite passes on SQLite. The bulk_create tests also pass on
PostgreSQL 18 and MariaDB 11.8.
--
Ticket URL: <https://code.djangoproject.com/ticket/37234#comment:3>

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------
Reporter: Filip Sedlák | Owner: Akshat
| Sparsh
Type: Bug | Status: assigned
Component: Database layer | Version: 6.0
(models, ORM) |
Severity: Release blocker | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/37234#comment:4>

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------
Reporter: Filip Sedlák | Owner: Akshat
| Sparsh
Type: Bug | Status: assigned
Component: Database layer | Version: 6.0
(models, ORM) |
Severity: Release blocker | Resolution:

Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Jacob Walls):

* has_patch: 1 => 0
* needs_better_patch: 0 => 1
* needs_tests: 0 => 1
* stage: Ready for checkin => Accepted

Comment:

Left notes on an additional test case we might want.
--
Ticket URL: <https://code.djangoproject.com/ticket/37234#comment:5>

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------
Reporter: Filip Sedlák | Owner: Akshat
| Sparsh
Type: Bug | Status: assigned
Component: Database layer | Version: 6.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Akshat Sparsh):

* has_patch: 0 => 1

* needs_better_patch: 1 => 0
* needs_tests: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/37234#comment:6>

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------
Reporter: Filip Sedlák | Owner: Akshat
| Sparsh
Type: Bug | Status: assigned
Component: Database layer | Version: 6.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/37234#comment:7>

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------
Reporter: Filip Sedlák | Owner: Akshat
| Sparsh
Type: Bug | Status: assigned
Component: Database layer | Version: 6.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Akshat Sparsh):

* needs_better_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/37234#comment:8>

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------
Reporter: Filip Sedlák | Owner: Akshat
| Sparsh
Type: Bug | Status: assigned
Component: Database layer | Version: 6.0
(models, ORM) |
Severity: Release blocker | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/37234#comment:9>

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------
Reporter: Filip Sedlák | Owner: Akshat
| Sparsh
Type: Bug | Status: assigned
Component: Database layer | Version: 6.0
(models, ORM) |
Severity: Release blocker | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Jacob Walls):

* needs_better_patch: 0 => 1

* stage: Ready for checkin => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/37234#comment:10>

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------
Reporter: Filip Sedlák | Owner: Akshat
| Sparsh
Type: Bug | Status: assigned
Component: Database layer | Version: 6.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Akshat Sparsh):

* needs_better_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/37234#comment:11>

[Django]

#37234: bulk_create fails when related objects got PKs later
-------------------------------------+-------------------------------------
Reporter: Filip Sedlák | Owner: Akshat
| Sparsh
Type: Bug | Status: assigned
Component: Database layer | Version: 6.0
(models, ORM) |
Severity: Release blocker | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* stage: Accepted => Ready for checkin

Comment:

I restored the patch to the state suggested by Filip and agreed with
Simon. (The detour I took into expression assignment is something we can
more leisurely evaluate without backporting in #37239.)
--
Ticket URL: <https://code.djangoproject.com/ticket/37234#comment:12>