[Django] #36620: Automate test coverage check on pull requests

[Django]

#36620: Automate test coverage check on pull requests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Type:
| Cleanup/optimization
Status: new | Component: Core
| (Other)
Version: dev | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
We could automate a significant part of checking test coverage on pull
requests. Instead of using a cloud service, there's an implementation
using `diff-cover` in a PR.

[https://github.com/django/django/pull/19777 PR]

[https://forum.djangoproject.com/t/rfc-prioritizing-tasks-for-automate-
processes-within-django-contribution-workflow-gsoc-2025/41960 Forum post]
--
Ticket URL: <https://code.djangoproject.com/ticket/36620>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36620: Automate test coverage check on pull requests
-------------------------------------+-------------------------------------

Reporter: Jacob Walls | Owner: Saurabh
Type: | Status: assigned
Cleanup/optimization |

Component: Core (Other) | Version: dev

Severity: Normal | Resolution:

Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* owner: (none) => Saurabh
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:1>

[Django]

#36620: Automate test coverage check on pull requests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Saurabh
Type: | Status: assigned
Cleanup/optimization |
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

I think the last time this was attempted the efforts stalled due the
difficulty of combining coverage reports from different suite runs. Some
part of the code is only covered by tests run on Postgres or a particular
Python version for example so there needs to be a coordinated job that
collects all of the `.coverage` artifacts and
[https://coverage.readthedocs.io/en/7.10.7/commands/cmd_combine.html then
combine them] otherwise the resulting coverage report will be lacking or
improperly reporting that some areas are not covered (e.g. if we only use
the SQLite test run and Postgres only changes are introduced).
--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:2>

[Django]

#36620: Automate test coverage check on pull requests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Saurabh
Type: | Status: assigned
Cleanup/optimization |
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jacob Walls):

(The linked implementation just treats that complexity as outside of MVP
and makes this caveat explicit in the posted comment.)
--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:3>

[Django]

#36620: Automate test coverage check on pull requests

--------------------------------------+------------------------------------

Reporter: Jacob Walls | Owner: Saurabh

Type: Cleanup/optimization | Status: assigned

Component: Core (Other) | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

--------------------------------------+------------------------------------
Changes (by Sarah Boyce):

* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:4>

[Django]

#36620: Automate test coverage check on pull requests
--------------------------------------+------------------------------------
Reporter: Jacob Walls | Owner: Saurabh
Type: Cleanup/optimization | Status: assigned
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 1
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
--------------------------------------+------------------------------------
Changes (by Sarah Boyce):

* needs_better_patch: 0 => 1
* needs_docs: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:5>

[Django]

#36620: Automate test coverage check on pull requests

-------------------------------------+-------------------------------------

Reporter: Jacob Walls | Owner: Saurabh

Type: | Status: assigned
Cleanup/optimization |

Component: Core (Other) | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Jacob Walls):

* needs_better_patch: 1 => 0
* needs_docs: 1 => 0
* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:6>

[Django]

#36620: Automate test coverage check on pull requests

--------------------------------------+------------------------------------

Reporter: Jacob Walls | Owner: Saurabh

Type: Cleanup/optimization | Status: assigned

Component: Core (Other) | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0

--------------------------------------+------------------------------------
Changes (by Jacob Walls):

* needs_better_patch: 0 => 1
* stage: Ready for checkin => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:7>

[Django]

#36620: Automate test coverage check on pull requests

-------------------------------------+-------------------------------------

Reporter: Jacob Walls | Owner: Saurabh

Type: | Status: assigned
Cleanup/optimization |

Component: Core (Other) | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Jacob Walls):

* needs_better_patch: 1 => 0

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:8>

[Django]

#36620: Automate test coverage check on pull requests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Saurabh

Type: | Status: closed

Cleanup/optimization |
Component: Core (Other) | Version: dev

Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls <jacobtylerwalls@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"a89183e63844a937aacd3ddb73c4952ef869d2cc" a89183e6]:
{{{#!CommitTicketReference repository=""
revision="a89183e63844a937aacd3ddb73c4952ef869d2cc"
Fixed #36620 -- Added coverage workflow to summarize coverage in pull
requests.

Part of GSoC 2025. Thanks Lily for mentorship, and Sarah Boyce and
Jacob Walls for reviews.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:9>

[Django]

#36620: Automate test coverage check on pull requests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Saurabh
Type: | Status: closed
Cleanup/optimization |
Component: Core (Other) | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jacob Walls <jacobtylerwalls@…>):

In [changeset:"e4c4a178aa642f8493b7ae2c0ad58527af51f67e" e4c4a17]:
{{{#!CommitTicketReference repository=""
revision="e4c4a178aa642f8493b7ae2c0ad58527af51f67e"
Reverted "Fixed #36620 -- Added coverage workflow to summarize coverage in
pull requests."

This reverts commit a89183e63844a937aacd3ddb73c4952ef869d2cc.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:10>

[Django]

#36620: Automate test coverage check on pull requests

--------------------------------------+------------------------------------

Reporter: Jacob Walls | Owner: Saurabh

Type: Cleanup/optimization | Status: new

Component: Core (Other) | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0

--------------------------------------+------------------------------------
Changes (by Jacob Walls):

* needs_better_patch: 0 => 1
* resolution: fixed =>

* stage: Ready for checkin => Accepted

* status: closed => new

Comment:

Revision is being progressed here:
https://github.com/django/django/pull/20302
--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:11>

[Django]

#36620: Automate test coverage check on pull requests
--------------------------------------+------------------------------------
Reporter: Jacob Walls | Owner: Saurabh

Type: Cleanup/optimization | Status: closed

Component: Core (Other) | Version: dev

Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
--------------------------------------+------------------------------------

Changes (by GitHub <noreply@…>):

* resolution: => fixed
* status: new => closed

Comment:

In [changeset:"26b0e2bb92caf2d16cabe455792350f20d6f42ca" 26b0e2bb]:
{{{#!CommitTicketReference repository=""
revision="26b0e2bb92caf2d16cabe455792350f20d6f42ca"
Fixed #36620 -- Fixed workflow to summarize coverage in PRs.

Follow-up to a89183e63844a937aacd3ddb73c4952ef869d2cc, which was
reverted in e4c4a178aa642f8493b7ae2c0ad58527af51f67e because a change
to the workflow trigger resulted in the PR branch not being checked out.

We used this opportunity to reimplement the coverage tracing and coverage
commenting in a two-workflow pattern with more granular permissions.

To reduce duplicative workflows, we removed the existing python test
workflow
on PRs, at least until we run more distinct configurations on GitHub
actions. The
run with coverage tracing enabled is sufficient for now. The existing
workflow still
runs on pushes to main. We can revisit when adding more test
configurations.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:12>

[Django]

#36620: Automate test coverage check on pull requests

-------------------------------------+-------------------------------------

Reporter: Jacob Walls | Owner: Saurabh

Type: | Status: closed
Cleanup/optimization |

Component: Core (Other) | Version: dev
Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Jacob Walls):

* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:13>

[Django]

#36620: Automate test coverage check on pull requests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Saurabh
Type: | Status: closed
Cleanup/optimization |
Component: Core (Other) | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by GitHub <noreply@…>):

In [changeset:"1dfd5aa2b5beb2d5354527eae4f2589b828ac06c" 1dfd5aa]:
{{{#!CommitTicketReference repository=""
revision="1dfd5aa2b5beb2d5354527eae4f2589b828ac06c"
Refs #36620 -- Removed PR number null check from coverage_comment
workflow.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:14>

[Django]

#36620: Automate test coverage check on pull requests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Saurabh
Type: | Status: closed
Cleanup/optimization |
Component: Core (Other) | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by GitHub <noreply@…>):

In [changeset:"fd4c5fa3edb1fa3f968085a7cb8b3e19f5646105" fd4c5fa3]:
{{{#!CommitTicketReference repository=""
revision="fd4c5fa3edb1fa3f968085a7cb8b3e19f5646105"
Refs #36620 -- Fixed PR number extraction in coverage_comment workflow.

Passing the PR number as an artifact is more reliable in cross-fork
workflows.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:15>

[Django]

#36620: Automate test coverage check on pull requests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Saurabh
Type: | Status: closed
Cleanup/optimization |
Component: Core (Other) | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jacob Walls <jacobtylerwalls@…>):

In [changeset:"e726254a380f2a35a2fcf71143e96cb5987d8102" e726254a]:
{{{#!CommitTicketReference repository=""
revision="e726254a380f2a35a2fcf71143e96cb5987d8102"
Refs #36620 -- Added contributor documentation for code coverage reports.

This was included in the original reverted patch:
a89183e63844a937aacd3ddb73c4952ef869d2cc
Follow-up to 26b0e2bb92caf2d16cabe455792350f20d6f42ca.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:16>

[Django]

#36620: Automate test coverage check on pull requests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Saurabh
Type: | Status: closed
Cleanup/optimization |
Component: Core (Other) | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Jacob Walls <jacobtylerwalls@…>):

In [changeset:"2e7133c58b24ebd32dc2030cc0665ce545c2670c" 2e7133c]:
{{{#!CommitTicketReference repository=""
revision="2e7133c58b24ebd32dc2030cc0665ce545c2670c"
[6.0.x] Refs #36620 -- Added contributor documentation for code coverage

reports.

This was included in the original reverted patch:
a89183e63844a937aacd3ddb73c4952ef869d2cc
Follow-up to 26b0e2bb92caf2d16cabe455792350f20d6f42ca.

Backport of e726254a380f2a35a2fcf71143e96cb5987d8102 from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:17>

[Django]

#36620: Automate test coverage check on pull requests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Saurabh
Type: | Status: closed
Cleanup/optimization |
Component: Core (Other) | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Jacob Walls <jacobtylerwalls@…>):

In [changeset:"2351c1b12cc9cf82d642f769c774bc3ea0cc4006" 2351c1b1]:
{{{#!CommitTicketReference repository=""
revision="2351c1b12cc9cf82d642f769c774bc3ea0cc4006"
Refs #36620 -- Ran coverage tests workflow on forks.

We can continue to limit the coverage comment workflow to django/django,
but now that this workflow is the main python test workflow, it should
run on forks by default. The other tests workflow (currently running
only JavaScript tests) may start running python tests again once we flesh
out the matrix, but since it was duplicating the coverage tests
configuration,
we temporarily removed it.

Follow-up to 26b0e2bb92caf2d16cabe455792350f20d6f42ca.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36620#comment:18>