[Django] #32746: Unexpected test failures in 3.3.2 when running the whole testsuite
Django
---------------------------------------------+------------------------
Reporter: Jakub Kulík | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
We recently introduced Django 3.3.2 to Solaris (with Python 3.9) and we
are hitting unexpected test failures when running the full test suite on
SPARC. When I rerun those tests one by one however, they are successful.
My guess is that parts of the test suite are not independent and affect
each other.
Errors we are seeing are (although I am not sure that they are always the
same and they changed since 3.3.1):
{{{
ERROR: test_output_normal (test_runner.test_debug_sql.TestDebugSQL)
ERROR: test_output_verbose (test_runner.test_debug_sql.TestDebugSQL)
ERROR: test_setupclass_exception (test_runner.test_debug_sql.TestDebugSQL)
ERROR: test_lazy_base_url_init (file_storage.tests.FileSystemStorageTests)
FAIL: test_incorrect_timezone (settings_tests.tests.SettingsTests)
FAILED (failures=1, errors=4, skipped=1146, expected failures=4)
}}}
Running the test suite with `--reverse` results in many more failures
(attached).
This happens on SPARC only, Intel test results are clean (both with and
without `--reverse`). Our other Django 2.2.21 with Python 3.7 is clean as
well on both platforms.
I am not sure how to continue/investigate the issue further, but I am
happy to do so if guided a little.
I have yet to try Django 3.1 or Python 3.7 (but runtime is highly unlike
the problem).
--
Ticket URL: <https://code.djangoproject.com/ticket/32746>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Jakub Kulík | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.2
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Jakub Kulík):
* Attachment "reverse-failures.txt" added.
Test failures when running with --reverse
Django
Reporter: Jakub Kulík | Owner: nobody
Component: Testing framework | Version: 3.2
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* status: new => closed
* resolution: => needsinfo
Comment:
I cannot reproduce these failures, all tests works for me when run in
reverse, see also https://djangoci.com/job/main-reverse/. Can you provide
a subset of these that causes failures for you? flags passed to
`runtests.py`, Python version, database, etc.
--
Ticket URL: <https://code.djangoproject.com/ticket/32746#comment:1>
Django
Reporter: Jakub Kulík | Owner: nobody
Component: Testing framework | Version: 3.2
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Jakub Kulík):
I see this issue only with Django 3.2 on SPARC (with both Python 3.7 & 3.9
and both with and without `--reverse`). Intel in this case is without
issues, I also don't see this in Django 3.1 or 3.0 with all combinations
of Python runtime (3.7 and 3.9), architecture (SPARC and Intel), and order
(`--reverse` or not).
In each case, tests are executed with the following command:
`/usr/bin/env LC_ALL=en_US.UTF-8 /usr/bin/python3.x ./runtests.py
--settings test_sqlite --parallel=1 --verbosity=3 (--reverse)`
--
Ticket URL: <https://code.djangoproject.com/ticket/32746#comment:2>
Django
Reporter: Jakub Kulík | Owner: nobody
Component: Testing framework | Version: 3.2
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak):
Thanks for extra details, unfortunately I still cannot reproduce these
failures so it must be something Solaris-specific.
--
Ticket URL: <https://code.djangoproject.com/ticket/32746#comment:3>
Django
Reporter: Jakub Kulík | Owner: nobody
Component: Testing framework | Version: 3.2
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Jakub Kulík):
Unfortunately, it seems so... Do you have any pointers/ideas on how I
should debug this? I'll try to find and fix the issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/32746#comment:4>
Django
Reporter: Jakub Kulík | Owner: nobody
Component: Testing framework | Version: 3.2
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak):
Replying to [comment:4 Jakub Kulík]:
> Unfortunately, it seems so... Do you have any pointers/ideas on how I
should debug this? I'll try to find and fix the issue.
It would help to find test classes (or even a set of tests) that are
reproducing this issue without running the entire test suite. Maybe
`test_runner` is enough.
--
Ticket URL: <https://code.djangoproject.com/ticket/32746#comment:5>
Django
Reporter: Jakub Kulík | Owner: nobody
Component: Testing framework | Version: 3.2
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Jakub Kulík):
* resolution: needsinfo => invalid
Comment:
I am sorry, I just found the issue and it's entirely on our side.
For posterity: Django test suite can be fragile in a sense that when one
test fails, many unrelated may follow, and that's what happened here.
`test_incorrect_timezone` is the only one that is actually failing and the
cause is our internal Solaris issue with `/usr/share/zoneinfo`. Once
patched, the test suite is again 100% green.
--
Ticket URL: <https://code.djangoproject.com/ticket/32746#comment:6>