[Django] #36235: RelatedManager.all().get_or_create() does not work

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------
Reporter: Nick Pope | Type: Bug
Status: new | Component: Database
| layer (models, ORM)
Version: dev | Severity: Normal
Keywords: get_or_create, | Triage Stage:
related, manager | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
When accessing the queryset for a related manager, `.get_or_create()` and
`.update_or_create()` lose context of the related instance.

Calling on the related manager works as expected:

{{{#!python
publisher.books.get_or_create(name="The Very Hungry Caterpillar")
}}}

This is the case that was fixed by #3121 and #23611.

But calling on the queryset causes an `IntegrityError` to be raised:

{{{#!python
publisher.books.all().get_or_create(name="The Very Hungry Caterpillar")
}}}

This can be a subtle failure that is hard to understand, especially if
`publisher.books.all()` is assigned to a variable earlier.

The challenge here is that the overridden methods on the manager set
`kwargs[self.field.name] = self.instance`.

We'd need to be able to pass this information down to the queryset. It
looks like this might already be available in `_known_related_objects`
which is set by `RelatedManager._apply_rel_filters()`, so that would be a
good starting point for investigation.

If this is something we can't fix reliably, then we should update the
admonition under
[https://docs.djangoproject.com/en/stable/ref/models/querysets/#get-or-
create .get_or_create()] in the docs and probably update
[https://docs.djangoproject.com/en/stable/ref/models/querysets/#update-or-
create .update_or_create()] to make this and other issues related to use
through `RelatedManager` more clear.
--
Ticket URL: <https://code.djangoproject.com/ticket/36235>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------

Reporter: Nick Pope | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:

Keywords: get_or_create, | Triage Stage:
related, manager | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Nick Pope):

The following test can be added to `GetOrCreateTests` in
`tests/get_or_create/tests.py` below
`test_get_or_create_on_related_manager`:

{{{#!python
def test_get_or_create_on_related_queryset(self):
p = Publisher.objects.create(name="Acme Publishing")
# Create a book through the publisher.
book, created = p.books.all().get_or_create(name="The Book of Ed &
Fred")
self.assertTrue(created)
# The publisher should have one book.
self.assertEqual(p.books.count(), 1)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36235#comment:1>

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------
Reporter: Nick Pope | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: get_or_create, | Triage Stage:
related, manager | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Description changed by Nick Pope:

Old description:

> When accessing the queryset for a related manager, `.get_or_create()` and
> `.update_or_create()` lose context of the related instance.
>
> Calling on the related manager works as expected:
>
> {{{#!python
> publisher.books.get_or_create(name="The Very Hungry Caterpillar")
> }}}
>
> This is the case that was fixed by #3121 and #23611.
>
> But calling on the queryset causes an `IntegrityError` to be raised:
>
> {{{#!python
> publisher.books.all().get_or_create(name="The Very Hungry Caterpillar")
> }}}
>
> This can be a subtle failure that is hard to understand, especially if
> `publisher.books.all()` is assigned to a variable earlier.
>
> The challenge here is that the overridden methods on the manager set
> `kwargs[self.field.name] = self.instance`.
>
> We'd need to be able to pass this information down to the queryset. It
> looks like this might already be available in `_known_related_objects`
> which is set by `RelatedManager._apply_rel_filters()`, so that would be a
> good starting point for investigation.
>
> If this is something we can't fix reliably, then we should update the
> admonition under
> [https://docs.djangoproject.com/en/stable/ref/models/querysets/#get-or-
> create .get_or_create()] in the docs and probably update
> [https://docs.djangoproject.com/en/stable/ref/models/querysets/#update-

> or-create .update_or_create()] to make this and other issues related to

> use through `RelatedManager` more clear.

New description:

When accessing the queryset for a related manager, `.get_or_create()` and
`.update_or_create()` lose context of the related instance.

Calling on the related manager works as expected:

{{{#!python
publisher.books.get_or_create(name="The Very Hungry Caterpillar")
}}}

This is the case that was fixed by #3121 and #23611.

But calling on the queryset causes an `IntegrityError` to be raised:

{{{#!python
publisher.books.all().get_or_create(name="The Very Hungry Caterpillar")
}}}

This can be a subtle failure that is hard to understand, especially if
`publisher.books.all()` is assigned to a variable earlier.

The challenge here is that the overridden methods on the manager set
`kwargs[self.field.name] = self.instance`.

We'd need to be able to pass this information down to the queryset. It
looks like this might already be available in `_known_related_objects`
which is set by `RelatedManager._apply_rel_filters()`, so that would be a
good starting point for investigation.

We would also need to check whether something needs to be done to select
the correct database as `RelatedManager.get_or_create()` has handling for
this:

{{{#!python
db = router.db_for_write(self.model, instance=self.instance)

}}}

If this is something we can't fix reliably, then we should update the
admonition under
[https://docs.djangoproject.com/en/stable/ref/models/querysets/#get-or-
create .get_or_create()] in the docs and probably update
[https://docs.djangoproject.com/en/stable/ref/models/querysets/#update-or-
create .update_or_create()] to make this and other issues related to use
through `RelatedManager` more clear.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/36235#comment:2>

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------
Reporter: Nick Pope | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:

Keywords: get_or_create, | Triage Stage: Accepted
related, manager |

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* stage: Unreviewed => Accepted

Comment:

Thank you for the report and test!
--
Ticket URL: <https://code.djangoproject.com/ticket/36235#comment:3>

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------
Reporter: Nick Pope | Owner:

| JaeHyuckSa
Type: Bug | Status: assigned

Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: get_or_create, | Triage Stage: Accepted
related, manager |

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by JaeHyuckSa):

* has_patch: 0 => 1
* owner: (none) => JaeHyuckSa
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/36235#comment:4>

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------
Reporter: Nick Pope | Owner:
| JaeHyuckSa
Type: Bug | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: get_or_create, | Triage Stage: Accepted
related, manager |

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by JaeHyuckSa):

* has_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/36235#comment:5>

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------
Reporter: Nick Pope | Owner:
| JaeHyuckSa
Type: Bug | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: get_or_create, | Triage Stage: Accepted
related, manager |

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by JaeHyuckSa):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36235#comment:6>

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------
Reporter: Nick Pope | Owner:
| JaeHyuckSa
Type: Bug | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: get_or_create, | Triage Stage: Accepted
related, manager |
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36235#comment:7>

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------
Reporter: Nick Pope | Owner:
| JaeHyuckSa
Type: Bug | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: get_or_create, | Triage Stage: Accepted
related, manager |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Johanan Oppong Amoateng):

Replying to [comment:6 JaeHyuckSa]:

Are you till working on this?
--
Ticket URL: <https://code.djangoproject.com/ticket/36235#comment:8>

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------
Reporter: Nick Pope | Owner:
| JaeHyuckSa
Type: Bug | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: get_or_create, | Triage Stage: Accepted
related, manager |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Johanan Oppong Amoateng):

* cc: Johanan Oppong Amoateng (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/36235#comment:9>

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------
Reporter: Nick Pope | Owner:
| JaeHyuckSa
Type: Bug | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: get_or_create, | Triage Stage: Accepted
related, manager |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by JaeHyuckSa):

Hello! I forgot that Johanan had a ticket. At the moment, I don’t have
time to work on this, so please feel free to take it on if you’re able to.
--
Ticket URL: <https://code.djangoproject.com/ticket/36235#comment:10>

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------

Reporter: Nick Pope | Owner: Johanan
| Oppong Amoateng

Type: Bug | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: get_or_create, | Triage Stage: Accepted
related, manager |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Johanan Oppong Amoateng):

* owner: JaeHyuckSa => Johanan Oppong Amoateng

--
Ticket URL: <https://code.djangoproject.com/ticket/36235#comment:11>

[Django]

#36235: RelatedManager.all().get_or_create() does not work
-------------------------------------+-------------------------------------
Reporter: Nick Pope | Owner: Johanan
| Oppong Amoateng
Type: Bug | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: get_or_create, | Triage Stage: Accepted
related, manager |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Vasilis Vagenas):

Hello.
The problem can still be reproduced:
* Git commit hash: `e17ee4468875077b90b70bb6a589ebad7493f757`
* test method: the method `test_get_or_create_on_related_queryset`,
mentioned above by Nick Pope
* test output:

{{{

======================================================================
ERROR: test_get_or_create_on_related_queryset
(get_or_create.tests.GetOrCreateTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/vasilis/oss_projects/django/django/django/db/models/query.py", line
581, in get_or_create
return self.get(**kwargs), False
File
"/home/vasilis/oss_projects/django/django/django/db/models/query.py", line
435, in get
raise self.model.DoesNotExist(
get_or_create.models.Book.DoesNotExist: Book matching query does not
exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File
"/home/vasilis/oss_projects/django/django/django/db/backends/utils.py",
line 84, in _execute
return self.cursor.execute(sql, params)
File
"/home/vasilis/oss_projects/django/django/django/db/backends/sqlite3/base.py",
line 416, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: NOT NULL constraint failed:
get_or_create_book.publisher_id_column

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/lib/python3.10/unittest/case.py", line 59, in
testPartExecutor
yield
File "/usr/lib/python3.10/unittest/case.py", line 591, in run
self._callTestMethod(testMethod)
File "/usr/lib/python3.10/unittest/case.py", line 549, in
_callTestMethod
method()
File
"/home/vasilis/oss_projects/django/django/tests/get_or_create/tests.py",
line 143, in test_get_or_create_on_related_queryset

book, created = p.books.all().get_or_create(name="The Book of Ed &
Fred")

File
"/home/vasilis/oss_projects/django/django/django/db/models/query.py", line
588, in get_or_create
return self.create(**params), True
File
"/home/vasilis/oss_projects/django/django/django/db/models/query.py", line
453, in create
obj.save(force_insert=True, using=self.db)
File
"/home/vasilis/oss_projects/django/django/django/db/models/base.py", line
726, in save
self.save_base(using=using, force_insert=force_insert,
File
"/home/vasilis/oss_projects/django/django/django/db/models/base.py", line
763, in save_base
updated = self._save_table(
File
"/home/vasilis/oss_projects/django/django/django/db/models/base.py", line
868, in _save_table
results = self._do_insert(cls._base_manager, using, fields,
returning_fields, raw)
File
"/home/vasilis/oss_projects/django/django/django/db/models/base.py", line
906, in _do_insert
return manager._insert(
File
"/home/vasilis/oss_projects/django/django/django/db/models/manager.py",
line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File
"/home/vasilis/oss_projects/django/django/django/db/models/query.py", line
1268, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File
"/home/vasilis/oss_projects/django/django/django/db/models/sql/compiler.py",
line 1401, in execute_sql
cursor.execute(sql, params)
File
"/home/vasilis/oss_projects/django/django/django/db/backends/utils.py",
line 66, in execute
return self._execute_with_wrappers(sql, params, many=False,
executor=self._execute)
File
"/home/vasilis/oss_projects/django/django/django/db/backends/utils.py",
line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File
"/home/vasilis/oss_projects/django/django/django/db/backends/utils.py",
line 79, in _execute
with self.db.wrap_database_errors:
File "/home/vasilis/oss_projects/django/django/django/db/utils.py", line
90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File
"/home/vasilis/oss_projects/django/django/django/db/backends/utils.py",
line 84, in _execute
return self.cursor.execute(sql, params)
File
"/home/vasilis/oss_projects/django/django/django/db/backends/sqlite3/base.py",
line 416, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: NOT NULL constraint failed:
get_or_create_book.publisher_id_column

----------------------------------------------------------------------

}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36235#comment:12>