[Django] #29127: manage.py executting tagged tests with syntax error does not show them
Django
-------------------------------------+-------------------------------------
Reporter: Kryštof | Owner: nobody
Řeháček |
Type: Bug | Status: new
Component: Testing | Version: 2.0
framework | Keywords: tests, tagged-
Severity: Normal | tests, SyntaxError
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
When I execute tests with syntax error without any argument then it shows
`SyntaxError: invalid syntax` as expected but when I execute tagged tests,
then it shows that it ran 0 tests and it is OK and even error. So when
tests with syntax executed with the --tag="some_tag" argument and they
have syntax error then they complete with OK status.
--
Ticket URL: <https://code.djangoproject.com/ticket/29127>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: Kryštof Řeháček | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 2.0
Severity: Normal | Resolution:
Keywords: tests, tagged- | Triage Stage:
tests, SyntaxError | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
--
Ticket URL: <https://code.djangoproject.com/ticket/29127#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: Kryštof Řeháček | Owner: nobody
Component: Testing framework | Version: 2.0
Severity: Normal | Resolution:
tests, SyntaxError |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
Comment:
The problem may be in `django.test.runner.filter_tests_by_tags()`.
--
Ticket URL: <https://code.djangoproject.com/ticket/29127#comment:2>
Django
-------------------------------------+-------------------------------------
| Hones
Type: Bug | Status: assigned
Component: Testing framework | Version: 2.0
Severity: Normal | Resolution:
Keywords: tests, tagged- | Triage Stage: Accepted
tests, SyntaxError |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => Daniel Hones
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/29127#comment:3>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Testing framework | Version: 2.0
Severity: Normal | Resolution:
Keywords: tests, tagged- | Triage Stage: Accepted
tests, SyntaxError |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => new
* owner: Daniel Hones => (none)
Comment:
It looks like this behavior is ultimately due to Python's unittest
TestLoader and not Django's test framework. In case of a SyntaxError when
loading a test suite, it hits
[https://github.com/python/cpython/blob/master/Lib/unittest/loader.py#L439
this line], which creates a _FailedTest object from it without raising an
error. Then the {{{django.test.runner.filter_tests_by_tags()}}} sees a
_FailedTest object which won't have any tags and so it won't be run at
all. Since tests are tagged using a decorator, tests in a file with a
syntax error won't have any tags recognized for them anyway because the
file cannot be evaluated. I don't think there's a reasonable fix for this
and I'm not sure it should even be considered a bug, although the behavior
is a bit misleading.
--
Ticket URL: <https://code.djangoproject.com/ticket/29127#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: Kryštof Řeháček | Owner: (none)
Component: Testing framework | Version: 2.0
Severity: Normal | Resolution:
Keywords: tests, tagged- | Triage Stage: Accepted
tests, SyntaxError |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Chris Jerdonek):
I think a possible solution for this would be to make it so that
`unittest.loader._FailedTest` objects always match when tags are provided.
That would prevent the issue of tests falsely passing due to e.g. syntax
errors.
--
Ticket URL: <https://code.djangoproject.com/ticket/29127#comment:5>
Django
-------------------------------------+-------------------------------------
| Jerdonek
Type: Bug | Status: assigned
Component: Testing framework | Version: 2.0
Severity: Normal | Resolution:
Keywords: tests, tagged- | Triage Stage: Accepted
tests, SyntaxError |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: (none) => Chris Jerdonek
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/29127#comment:6>
Django
-------------------------------------+-------------------------------------
Reporter: Kryštof Řeháček | Owner: Chris
| Jerdonek
Type: Bug | Status: assigned
Component: Testing framework | Version: 2.0
Severity: Normal | Resolution:
Keywords: tests, tagged- | Triage Stage: Accepted
tests, SyntaxError |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
PR: https://github.com/django/django/pull/14195
--
Ticket URL: <https://code.djangoproject.com/ticket/29127#comment:7>
Django
-------------------------------------+-------------------------------------
Reporter: Kryštof Řeháček | Owner: Chris
| Jerdonek
Type: Bug | Status: assigned
Component: Testing framework | Version: 2.0
Severity: Normal | Resolution:
tests, SyntaxError | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/29127#comment:8>
Django
-------------------------------------+-------------------------------------
Reporter: Kryštof Řeháček | Owner: Chris
| Jerdonek
Component: Testing framework | Version: 2.0
Keywords: tests, tagged- | Triage Stage: Ready for
tests, SyntaxError | checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"038940cf5525c41464a1b9e9ba3801042320b0cc" 038940cf]:
{{{
#!CommitTicketReference repository=""
revision="038940cf5525c41464a1b9e9ba3801042320b0cc"
Fixed #29127 -- Prevented DiscoverRunner from hiding tagged test with
syntax errors.
This mades _FailedTest objects always match tags in DiscoverRunner.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29127#comment:9>