[Django] #35408: Optimize post-migrate permission creation

[Django]

#35408: Optimize post-migrate permission creation
-------------------------------------+-------------------------------------
Reporter: Adam | Owner: Adam Johnson
Johnson |
Type: | Status: assigned
Cleanup/optimization |
Component: | Version: dev
contrib.auth |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 1
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
I have often seen `django.contrib.auth.management.create_permissions()`
take a significant amount of time in test run profiles. It can be
optimized by batching more of its operations, including making
`ContentTypeManager.get_for_models()` use batch creation.

For a comparison, I profiled 1518 of Django’s 1518 tests in modules called
“models”:

{{{
$ python -m cProfile -o profile runtests.py --parallel 1 *model*

$ python -m pstats profile <<< 'sort cumtime
stats 10000' | less
}}}

Before optimization stats:

* Total 11,938,857 function calls taking 5.349 seconds.
* 88 calls to `create_permissions()` take 456ms, ~8.5% of the total time.

After optimization stats:

* Total 11,359,071 function calls taking 5.035 seconds.
* 88 calls to `create_permissions()` now take 239ms, ~4.7% of the toal
time.
* 217ms and 579,786 function calls saved.

Optimization is limited because the `post_migrate` signal runs once per
migrated app config, so there’s no chance to bulk create *all* content
types and permissions at once. If we introduced a new “all migrated apps”
signal, that could reduce runtime further by batching all creation.
--
Ticket URL: <https://code.djangoproject.com/ticket/35408>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35408: Optimize post-migrate permission creation
-------------------------------------+-------------------------------------

Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: contrib.auth | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Natalia Bidart):

* needs_better_patch: 0 => 1
* stage: Unreviewed => Accepted

Comment:

Accepting following an initial review of the patch which looks sensible.
Setting as patch needs improvement due to the comments raised by David and
Mariusz.
--
Ticket URL: <https://code.djangoproject.com/ticket/35408#comment:1>

[Django]

#35408: Optimize post-migrate permission creation
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: contrib.auth | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Adam Johnson):

I repeated the profiling with the latest version of the patch, on top of
the latest `main` commit. The numbers are similar.

Before optimization stats:

* Total 12,387,798 function calls taking 5.589 seconds.
* 88 calls to create_permissions() take 483ms, ~8.6% of the total time.

After optimization stats:

* Total 11,797,519 function calls taking 5.207 seconds.
* 88 calls to create_permissions() take 241ms, ~4.6% of the total time.
* 590,279 function calls and 242ms saved.
--
Ticket URL: <https://code.djangoproject.com/ticket/35408#comment:2>

[Django]

#35408: Optimize post-migrate permission creation
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: contrib.auth | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Adam Johnson):

* needs_better_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/35408#comment:3>

[Django]

#35408: Optimize post-migrate permission creation
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: contrib.auth | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/35408#comment:4>

[Django]

#35408: Optimize post-migrate permission creation
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: contrib.auth | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Description changed by Adam Johnson:

Old description:

> I have often seen `django.contrib.auth.management.create_permissions()`
> take a significant amount of time in test run profiles. It can be
> optimized by batching more of its operations, including making
> `ContentTypeManager.get_for_models()` use batch creation.
>
> For a comparison, I profiled 1518 of Django’s 1518 tests in modules
> called “models”:
>
> {{{
> $ python -m cProfile -o profile runtests.py --parallel 1 *model*
>
> $ python -m pstats profile <<< 'sort cumtime
> stats 10000' | less
> }}}
>

> Before optimization stats:
>
> * Total 11,938,857 function calls taking 5.349 seconds.
> * 88 calls to `create_permissions()` take 456ms, ~8.5% of the total time.
>
> After optimization stats:
>

> * Total 11,359,071 function calls taking 5.035 seconds.
> * 88 calls to `create_permissions()` now take 239ms, ~4.7% of the toal
> time.
> * 217ms and 579,786 function calls saved.
>
> Optimization is limited because the `post_migrate` signal runs once per
> migrated app config, so there’s no chance to bulk create *all* content
> types and permissions at once. If we introduced a new “all migrated apps”
> signal, that could reduce runtime further by batching all creation.

New description:

I have often seen `django.contrib.auth.management.create_permissions()`
take a significant amount of time in test run profiles. It can be
optimized by batching more of its operations, including making
`ContentTypeManager.get_for_models()` use batch creation.

For a comparison, I profiled 1518 of Django’s tests in modules called

“models”:

{{{
$ python -m cProfile -o profile runtests.py --parallel 1 *model*

$ python -m pstats profile <<< 'sort cumtime
stats 10000' | less
}}}

Before optimization stats:

* Total 11,938,857 function calls taking 5.349 seconds.
* 88 calls to `create_permissions()` take 456ms, ~8.5% of the total time.

After optimization stats:

* Total 11,359,071 function calls taking 5.035 seconds.
* 88 calls to `create_permissions()` now take 239ms, ~4.7% of the toal
time.
* 217ms and 579,786 function calls saved.

Optimization is limited because the `post_migrate` signal runs once per
migrated app config, so there’s no chance to bulk create *all* content
types and permissions at once. If we introduced a new “all migrated apps”
signal, that could reduce runtime further by batching all creation.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/35408#comment:5>

[Django]

#35408: Optimize post-migrate permission creation
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson

Cleanup/optimization | Status: closed
Component: contrib.auth | Version: dev
Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce <42296566+sarahboyce@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"d2c5a30e5a8595d06c4a70ff0f66032fee0d3c8e" d2c5a30e]:
{{{#!CommitTicketReference repository=""
revision="d2c5a30e5a8595d06c4a70ff0f66032fee0d3c8e"
Fixed #35408 -- Optimized post-migrate permission creation.

co-authored-by: Mariusz Felisiak <felisiak...@gmail.com>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35408#comment:6>