[Django] #26630: Defered constraint checks flush after `post_delete` signal

[Django]

#26630: Defered constraint checks flush after `post_delete` signal
----------------------------------------------+--------------------
Reporter: alexmadjar | Owner: nobody
Type: Bug | Status: new
Component: Database layer (models, ORM) | Version: 1.9
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------------------+--------------------
Repro steps:

1. Use a database that supports `can_defer_constraint_checks` (tested on
postgres)
1. Have a model (A) with a `null=True` ForeignKey or OneToOneField (F) to
another model (B)
1. Connect a function to listen to `post_delete` on (B) that pulls
objects of type (A) from the db (using a = A.objects.get... etc )
1. Try to access a.F for an object a that used to be associated with the
deleted instance of B that was just deleted

Expectation: a.F_id is `None`

Actually: a.F_id is the id of the deleted instance and a.F raises an
ObjectDoesNotExist exception

Note: doesn't repro on fields where `null=False` or on sqlite. My
diagnosis is that A's field isn't being properly cleared (or A objects
deleted in the case of on delete CASCADE) because the constraint checks
aren't being flushed until _after_ the `post_delete` signal fires

--
Ticket URL: <https://code.djangoproject.com/ticket/26630>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------

Reporter: alexmadjar | Owner: nobody
Type: Bug | Status: new

Component: Database layer | Version: 1.9
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by alexmadjar):

* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0

Comment:

Only tried on CASCADE deletes, I think it blames to this code -
https://github.com/django/django/blob/master/django/db/models/deletion.py#L15

--
Ticket URL: <https://code.djangoproject.com/ticket/26630#comment:1>

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------

Reporter: alexmadjar | Owner: nobody
Type: Bug | Status: new

Component: Database layer | Version: 1.9
(models, ORM) |
Severity: Normal | Resolution:

Keywords: | Triage Stage:
| Unreviewed

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------

Comment (by timgraham):

If you could provide a test case for Django's test suite, that would be
helpful.

--
Ticket URL: <https://code.djangoproject.com/ticket/26630#comment:2>

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------

Reporter: alexmadjar | Owner: nobody
Type: Bug | Status: new

Component: Database layer | Version: 1.9
(models, ORM) |
Severity: Normal | Resolution:

Keywords: | Triage Stage:
| Unreviewed

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by timgraham):

* Attachment "t26630.tar.gz" added.

sample app to reproduce

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------

Reporter: alexmadjar | Owner: nobody
Type: Bug | Status: new

Component: Database layer | Version: 1.9
(models, ORM) |
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by timgraham):

* stage: Unreviewed => Accepted

Comment:

Seems legitimate. I attached a sample project I used to reproduce. The
first test method shows the failure.

--
Ticket URL: <https://code.djangoproject.com/ticket/26630#comment:3>

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------
Reporter: Alex Madjar | Owner: Can
| Sarıgöl
Type: Bug | Status: assigned
Component: Database layer | Version: master

(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Can Sarıgöl):

* owner: nobody => Can Sarıgöl
* status: new => assigned
* version: 1.9 => master

Comment:

[https://github.com/django/django/pull/11780 PR]

--
Ticket URL: <https://code.djangoproject.com/ticket/26630#comment:4>

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------
Reporter: Alex Madjar | Owner: Can
| Sarıgöl
Type: Bug | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Can Sarıgöl):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/26630#comment:5>

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------
Reporter: Alex Madjar | Owner: Can
| Sarıgöl
Type: Bug | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by felixxm):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/26630#comment:6>

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------
Reporter: Alex Madjar | Owner: (none)
Type: Bug | Status: assigned
Component: Database layer | Version: dev

(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Can Sarıgöl):

* owner: Can Sarıgöl => (none)

--
Ticket URL: <https://code.djangoproject.com/ticket/26630#comment:7>

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------
Reporter: Alex Madjar | Owner: (none)
Type: Bug | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Comment (by Annabelle Wiegart):

I'm unable to reproduce the described behavior with the attached sample
project. The related Choice instance seems to be deleted immediately. The
post_delete callback function produces the error
`t26630.models.Choice.DoesNotExist: Choice matching query does not exist`.
Is there still a bug that needs to be fixed?
--
Ticket URL: <https://code.djangoproject.com/ticket/26630#comment:8>

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------
Reporter: Alex Madjar | Owner: (none)
Type: Bug | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Comment (by Jacob Walls):

Thanks, Annabelle, I'm seeing the same. However, I think this is a symptom
of the test project needing to be updated. If you connect a `post_delete`
signal on `Choice` as well, you can undo the effect of the optimization
from #30856 and trigger the failing code path again.

[https://dryorm.xterm.info/cascade-deletions-post_delete-signal-
interaction Here is a fiddle on DryORM]

I'm not sure exactly how to fix this, but we should probably examine the
solution space a bit more before closing this one.

If you're curious how I dug around for this, I noticed the original report
suggesting that one of the objects was not being cascade deleted, but as
you saw, it was. So I set a breakpoint and noticed we were hitting a "fast
delete" path, which led me to #30856 driving more traffic into that path.
--
Ticket URL: <https://code.djangoproject.com/ticket/26630#comment:9>

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------
Reporter: Alex Madjar | Owner: (none)
Type: Bug | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Comment (by Annabelle Wiegart):

Thank you for your clarifications, Jacob. i looked at the DryORM example
and a bit into the Collector class and the "fast delete" path.

I'm a bit unsure about the expected behavior though. The ticket
description says "Expectation: a.F_id is None". Since the "fast delete"
path leads to the deletion of `a` not being deferred, I suppose the
expected behavior would be exactly this, even if a `post_delete` signal is
attached to `Choice` as well?
--
Ticket URL: <https://code.djangoproject.com/ticket/26630#comment:10>

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------
Reporter: Alex Madjar | Owner: (none)
Type: Bug | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Annabelle Wiegart):

* cc: Annabelle Wiegart (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/26630#comment:11>

[Django]

#26630: Defered constraint checks flush after `post_delete` signal

-------------------------------------+-------------------------------------
Reporter: Alex Madjar | Owner: (none)

Type: Bug | Status: closed

Component: Database layer | Version: dev
(models, ORM) |

Severity: Normal | Resolution: duplicate

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Jacob Walls):

* resolution: => duplicate
* status: assigned => closed

Comment:

I'm also a little unsure. I think the gist of the report is that doing
database queries during a `post_delete` listener is problematic, because
you can query `Choice` rows by `question_id` and get misleading results
(as those rows are in the middle of being cascade deleted).

I'm reading the original report to suggest reordering the `post_delete`
signal to run after constraints are flushed (and all cascade deletes have
happened). I doubt that would be backward compatible.

In #30835 something similar was suggested, and Simon explained you can use
`on_commit()` to defer execution of the callback (to a point when all
cascade deletions have been made). I think I'd suggest rewriting the
signal handler that way to avoid this problem with unstable `Choice`
results.

I'm inclined to close this as duplicate of #30835. As mentioned there, if
there's a proposal for a docs tweak in `post_delete` we could consider it.
(In that case, do reopen #30835!)

Thanks for helping out with triage.
--
Ticket URL: <https://code.djangoproject.com/ticket/26630#comment:12>