[Django] #33213: Document coverage setup for parallel tests
Django
-----------------------------------------+------------------------
Reporter: Mr. Glass | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version:
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
Ticket #31431 requested django implement support coverage for parallel
tests. It was marked as wontfix, citing #4501 where a decision was made to
document how to run coverage rather than integrate it.
Unfortunately, the
[https://docs.djangoproject.com/en/dev/topics/testing/advanced/ coverage
documentation] does not mention parallel tests at all.
Since parallel tests use subprocesses,
[https://coverage.readthedocs.io/en/6.0.2/subprocess.html special setup]
is needed for coverage to be accurate. I have so far been unable to figure
this out, and the best information I've found is in this
[https://stackoverflow.com/a/42862546/502816 StackOverflow post].
The coverage section needs documentation covering parallel tests, and the
parallel tests section should probably mention that coverage needs special
setup and link to it.
I would be happy to update the documentation if someone can provide me
with the working answer.
--
Ticket URL: <https://code.djangoproject.com/ticket/33213>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Mr. Glass | Owner: nobody
Component: Documentation | Version:
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mr. Glass):
* type: Uncategorized => New feature
* component: Uncategorized => Documentation
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:1>
Django
Reporter: Mr. Glass | Owner: nobody
Component: Documentation | Version:
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* cc: Carlton Gibson (added)
* stage: Unreviewed => Accepted
Comment:
Tentatively accepted, I'm not sure if it's doable to capture coverage data
with parallel tests. You can try to ask on
[https://docs.djangoproject.com/en/dev/faq/help/ support channels].
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:2>
Django
Reporter: Mr. Glass | Owner: nobody
Component: Documentation | Version:
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Adam Johnson):
Coverage.py documents [https://coverage.readthedocs.io/en/latest/cmd.html
#cmd-run for 'run']:
> Coverage.py can measure multi-threaded programs by default. If you are
using more exotic concurrency, with the multiprocessing, greenlet,
eventlet, or gevent libraries, then coverage.py will get very confused.
Use the --concurrency switch to properly measure programs using these
libraries. Give it a value of multiprocessing, thread, greenlet, eventlet,
or gevent. Values other than thread require the C extension.
Plus it has a whole page on the topic:
https://coverage.readthedocs.io/en/latest/subprocess.html
I've found `--concurrency=multiprocessing` works fine.
Not sure Django needs to doc anything considering it's really extensively
documented there.
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:3>
Django
Reporter: Mr. Glass | Owner: nobody
Component: Documentation | Version:
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mr. Glass):
Replying to [comment:3 Adam Johnson]:
> Coverage.py documents
[https://coverage.readthedocs.io/en/latest/cmd.html#cmd-run for 'run']:
>
> > Coverage.py can measure multi-threaded programs by default. If you are
using more exotic concurrency, with the multiprocessing, greenlet,
eventlet, or gevent libraries, then coverage.py will get very confused.
Use the --concurrency switch to properly measure programs using these
libraries. Give it a value of multiprocessing, thread, greenlet, eventlet,
or gevent. Values other than thread require the C extension.
>
> Plus it has a whole page on the topic:
https://coverage.readthedocs.io/en/latest/subprocess.html
>
> I've found `--concurrency=multiprocessing` works fine.
>
> Not sure Django needs to doc anything considering it's really
extensively documented there.
Yeah --concurrency=multiprocessing didn't work for me, nor a bunch of
config file options I tried. I spent way more time than allotted in my
sprint trying to get this, but I could be making a simple mistake. I'll
try to hop on a help channel if I can buy time.
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:4>
Django
Reporter: Mr. Glass | Owner: nobody
Component: Documentation | Version:
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Adam Johnson):
I just set up a demo repo showing how to integrate coverage with django's
test runner here: https://github.com/adamchainz/django-coverage-example
Hope that helps.
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:5>
Django
Reporter: Mr. Glass | Owner: nobody
Component: Documentation | Version:
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak):
We could improve
[https://docs.djangoproject.com/en/stable/internals/contributing/writing-
code/unit-tests/#code-coverage docs] by removing ''"Coverage should be run
in a single process to obtain accurate statistics."'' and changing the
current commands to the:
{{{
coverage erase
coverage run --concurrency=multiprocessing ./runtests.py
--settings=test_sqlite
coverage combine && coverage html
}}}
However, this raises a warning for me:
{{{
coverage/inorout.py:530: CoverageWarning: Module django was previously
imported, but not measured (module-not-measured)
self.warn(msg, slug="module-not-measured")
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:6>
Django
Reporter: Mr. Glass | Owner: Martin Pauly
Type: New feature | Status: assigned
Component: Documentation | Version:
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Martin Pauly):
* owner: nobody => Martin Pauly
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:7>
Django
Reporter: Mr. Glass | Owner: Martin Pauly
Type: New feature | Status: assigned
Component: Documentation | Version:
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Martin Pauly):
* has_patch: 0 => 1
Comment:
Pull Request: [https://github.com/django/django/pull/15116]
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:8>
Django
Reporter: Mr. Glass | Owner: Martin Pauly
Type: New feature | Status: assigned
Component: Documentation | Version:
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:9>
Django
Reporter: Mr. Glass | Owner: Paolo
| Melchiorre
Type: New feature | Status: assigned
Component: Documentation | Version:
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* owner: Martin Pauly => Paolo Melchiorre
* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin
Comment:
[https://github.com/django/django/pull/16435 New PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:10>
Django
Reporter: Mr. Glass | Owner: Paolo
| Melchiorre
Component: Documentation | Version:
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by GitHub <noreply@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"69352d85fa8412865db9e0c7f177b333c0eac3e2" 69352d85]:
{{{
#!CommitTicketReference repository=""
revision="69352d85fa8412865db9e0c7f177b333c0eac3e2"
Fixed #33213 -- Doc'd testing code coverage in parallel and used it.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:11>
Django
Reporter: Mr. Glass | Owner: Paolo
| Melchiorre
Type: New feature | Status: closed
Component: Documentation | Version:
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by GitHub <noreply@…>):
In [changeset:"78da5ca0c1f2ab3201f8f6cd629e80d805ea023d" 78da5ca]:
{{{
#!CommitTicketReference repository=""
revision="78da5ca0c1f2ab3201f8f6cd629e80d805ea023d"
Reverted "Fixed #33213 -- Doc'd testing code coverage in parallel and used
it."
This reverts commit 69352d85fa8412865db9e0c7f177b333c0eac3e2.
Test coverage for async methods was no longer calculated with this
change.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:12>
Django
Reporter: Mr. Glass | Owner: Paolo
| Melchiorre
Component: Documentation | Version:
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* status: closed => new
* resolution: fixed =>
* stage: Ready for checkin => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:13>
Django
Reporter: Mr. Glass | Owner: Paolo
| Melchiorre
Type: New feature | Status: new
Component: Documentation | Version:
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* has_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:14>
Django
Reporter: Mr. Glass | Owner: Paolo
| Melchiorre
Component: Documentation | Version:
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by GitHub <noreply@…>):
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"070cbac0dbf6a09b55aad322137ab168b75bf56b" 070cbac]:
{{{
#!CommitTicketReference repository=""
revision="070cbac0dbf6a09b55aad322137ab168b75bf56b"
Restored multiprocessing concurrency on coverage.py settings
* Revert "Reverted "Fixed #33213 -- Doc'd testing code coverage in
parallel and used it.""
This reverts commit 78da5ca0c1f2ab3201f8f6cd629e80d805ea023d.
* Restored coverage multiprocess concurrency with threads
Investigating https://github.com/nedbat/coveragepy/issues/1585 revealed
that thread tracing gets disabled when passing
`concurrency = multiprocessing`. Adding `thread` restores it, and
ensures that the `auser()` is reported as covered since the test suite
uses `AsyncToSync` to execute this middleware (which spawns threads).
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33213#comment:15>