[Django] #33454: Django 4 TestRunner does not pick up tests correctly.
Django
---------------------------------------------+------------------------
Reporter: Thorbenl | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 4.0
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
Hey!
We use Django for an online webshop, and want to upgrade to Django 4.
However, since upgrading on a test branch, we have encountered a problem,
that did not appear before upgrading:
We serve two different markets, lets call them Market A and B. Since these
two markets can have different functionalities, our INSTALLED_APPS gets
populated like so:
Here is more information, and the error traces:
{{{
INSTALLED_APPS = [
...
'payments', # THIS INCLUDES `BasePayment` Model
....
]
}}}
{{{
MARKET_SPECIFIC_APPS = {
MARKET_B: [
'market_b.apps.MarketBConfig',
'market_b_payments.apps.MarketBPaymentsConfig'
],
MARKET_A: [
'market_a.apps.MarketAConfig',
'market_a_payments.apps.MarketAPaymentsConfig'
],
}
}}}
{{{
if MARKET in MARKET_SPECIFIC_APPS:
# If there is a market-specific app, add it to INSTALLED_APPS
INSTALLED_APPS += MARKET_SPECIFIC_APPS[MARKET]
}}}
{{{
======================================================================
ERROR [0.004s]: market_a.test_redirects (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: market_a.test_redirects
Traceback (most recent call last):
File
"/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py",
line 436, in _find_test_path
module = self._get_module_from_name(name)
File
"/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py",
line 377, in _get_module_from_name
__import__(name)
File "/e-commerce/market_a/test_redirects.py", line 5, in <module>
from market_a.models import AdvertisementIdMapping
File "/e-commerce/market_a/models.py", line 14, in <module>
class MigratedMissingData(models.Model):
File "/e-commerce/venv/lib/python3.9/site-
packages/django/db/models/base.py", line 113, in __new__
raise RuntimeError(
RuntimeError: Model class market_a.models.MigratedMissingData doesn't
declare an explicit app_label and isn't in an application in
INSTALLED_APPS.
======================================================================
ERROR [0.000s]: market_a_payments.models (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: market_a_payments.models
Traceback (most recent call last):
File
"/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py",
line 470, in _find_test_path
package = self._get_module_from_name(name)
File
"/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py",
line 377, in _get_module_from_name
__import__(name)
File "/e-commerce/market_a_payments/models/__init__.py", line 1, in
<module>
from .payment import Payment
File "/e-commerce/market_a_payments/models/payment.py", line 12, in
<module>
class Payment(BasePayment):
File "/e-commerce/venv/lib/python3.9/site-
packages/django/db/models/base.py", line 113, in __new__
raise RuntimeError(
RuntimeError: Model class market_a_payments.models.payment.Payments
doesn't declare an explicit app_label and isn't in an application in
INSTALLED_APPS.
======================================================================
ERROR [0.000s]: market_a_payments.tests (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: market_a_payments.tests
Traceback (most recent call last):
File
"/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py",
line 436, in _find_test_path
module = self._get_module_from_name(name)
File
"/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py",
line 377, in _get_module_from_name
__import__(name)
File "/e-commerce/market_a_payments/tests.py", line 20, in <module>
from market_a_payments.models import Payment
File "/e-commerce/market_a_payments/models/__init__.py", line 1, in
<module>
from .payment import Payment
File "/e-commerce/market_a_payments/models/payment.py", line 12, in
<module>
class Payment(BasePayment):
File "/e-commerce/venv/lib/python3.9/site-
packages/django/db/models/base.py", line 113, in __new__
raise RuntimeError(
RuntimeError: Model class market_a_payments.models.payment.Payments
doesn't declare an explicit app_label and isn't in an application in
INSTALLED_APPS.
}}}
all these test sit in an app called `market_a_payments`
This app should not be discovered when running tests for market b:
{{{
MARKET=MARKET_B python3 manage.py test --tag market_b --timing
}}}
I can try to share a bit of the test, and where it fails according to the
stack:
{{{
from urllib.parse import quote
from django.test import TestCase, Client
from core.mocks import Mocks
... other imports
MARKET_A = "market_a"
MARKET_B = "market_b"
def valid_market(market: str):
return market in [MARKET_A, MARKET_B]
def override_market(market):
if not valid_market(market):
raise Exception(f"{market} is not a valid market.")
return tag(market)
class TestRedirects(TestCase):
@override_market(MARKET_A)
def test_landing_page_redirects(self):
client = Client()
cases = {
....
}
for input, dst in cases.items():
with self.subTest(input):
response = client.get(input, secure=False, follow=True)
self.assertRedirects(response, dst, 301, 200)
}}}
Line 12 is actually `class TestRedirects(TestCase):`
Django 3.2.8 worked super fine, so i assume sth in the test runner must
have changed from that version to 4, as our code remained unchanged in
this test.
Annotating the `class` with the `@override_market(MARKET_A)` didnt do
anything either.
Test are run on circleci using `circleci/python:3.9.7-node-browsers`
image.
The overall goal here is that this test only gets run/discovered when the
specified market gets hit.
so `MARKET=MARKET_B python3 manage.py test --tag market_b --timing` should
not discover or run this test. taking off `--parallel` as suggested didnt
do anything either.
Minimal reproduction here:
https://github.com/Thorbenl/django4-testrunner.git
--
Ticket URL: <https://code.djangoproject.com/ticket/33454>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-----------------------------------+--------------------------------------
Reporter: Thorbenl | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 4.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--
Ticket URL: <https://code.djangoproject.com/ticket/33454#comment:1>
Django
-----------------------------------+--------------------------------------
Type: Bug | Status: new
Component: Testing framework | Version: 4.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
You need to debug the issue and explain where Django is at fault. See if
you can [https://docs.djangoproject.com/en/dev/internals/contributing
/triaging-tickets/#bisecting-a-regression bisect] to find the commit where
the behavior changed.
--
Ticket URL: <https://code.djangoproject.com/ticket/33454#comment:2>
Django
-----------------------------------+--------------------------------------
Reporter: Thorben Luepkes | Owner: nobody
Type: Bug | Status: closed
Component: Testing framework | Version: 4.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* cc: Chris Jerdonek (added)
* status: new => closed
* resolution: => invalid
Comment:
Thanks for the report. This behavior was changed intentionally in
038940cf5525c41464a1b9e9ba3801042320b0cc. Previously tests that failed to
load did not match tags. I was able to fix the attached project
(`django4-testrunner`) by:
- removing the
[https://github.com/Thorbenl/django4-testrunner/blob/main/django4_testrunner/__init__.py
__init__.py] file from `django4_testrunner`,
- changing imports from `django4_testrunner.default.market_settings` to
`default.market_settings`.
It looks like an issue in your code.
--
Ticket URL: <https://code.djangoproject.com/ticket/33454#comment:3>
Django
-----------------------------------+--------------------------------------
Type: Bug | Status: closed
Component: Testing framework | Version: 4.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Thorben Luepkes):
Replying to [comment:3 Mariusz Felisiak]:
> Thanks for the report. This behavior was changed intentionally in
038940cf5525c41464a1b9e9ba3801042320b0cc. Previously tests that failed to
load did not match tags. I was able to fix the attached project
(`django4-testrunner`) by:
> - removing the
[https://github.com/Thorbenl/django4-testrunner/blob/main/django4_testrunner/__init__.py
__init__.py] file from `django4_testrunner`,
> - changing imports from `django4_testrunner.default.market_settings` to
`default.market_settings`.
>
> It looks like an issue in your code.
This seems like a weird change to make to me?
Why would you change the way tags work?
Of course it is a problem in our code somehow, but cant there be a setting
so we can revert it?
I would really like to be able to share the whole codebase, but as you
might know, thats not really possible. We serve two different markets with
the same codebase, and have market specific tests: Main market runs 1000
tests, and market b just runs 27 tests that are specific to that market.
But due to the change to tags, the test-suit tries to include apps into
the test that arent in `INSTALLED_APPS`. I have dumped out the variable
during the test, and it does not have the apps where the error is thrown
even included.
--
Ticket URL: <https://code.djangoproject.com/ticket/33454#comment:4>
Django
-----------------------------------+--------------------------------------
Type: Bug | Status: closed
Component: Testing framework | Version: 4.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak):
Replying to [comment:4 Thorben Luepkes]:
> And `main_market_receipt` is an app NOT included during the test run,
when I dump out the `INSTALLED_APPS`
Have you tried the solution proposed in my previous
[https://code.djangoproject.com/ticket/33454?replyto=4#comment:3 comment]?
You can try to use one of
[https://code.djangoproject.com/wiki/TicketClosingReasons/UseSupportChannels
support channels] if you have further questions.
--
Ticket URL: <https://code.djangoproject.com/ticket/33454#comment:5>
Django
-----------------------------------+--------------------------------------
Type: Bug | Status: closed
Component: Testing framework | Version: 4.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Thorben Luepkes):
Replying to [comment:5 Mariusz Felisiak]:
> Replying to [comment:4 Thorben Luepkes]:
> > And `main_market_receipt` is an app NOT included during the test run,
when I dump out the `INSTALLED_APPS`
>
> Have you tried the solution proposed in my previous
[https://code.djangoproject.com/ticket/33454?replyto=4#comment:3 comment]?
You can try to use one of
[https://code.djangoproject.com/wiki/TicketClosingReasons/UseSupportChannels
support channels] if you have further questions.
>
Yes, I did. To no avail :)
Also, why should this behavior change from `3.2.11` to `4.0.1`? :) Is
there a different way apps get included now?
--
Ticket URL: <https://code.djangoproject.com/ticket/33454#comment:6>
Django
-----------------------------------+--------------------------------------
Type: Bug | Status: closed
Component: Testing framework | Version: 4.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak):
> Is there a different way apps get included now?
It's not about apps but about tags. In your case `main_market` and
`main_market_receipt` failed to load which should not happened, even when
they do not match tags. If you want to debug your issue with the help of
other Django folks you should use support channels, Trac is not one of
them.
--
Ticket URL: <https://code.djangoproject.com/ticket/33454#comment:7>
Django
-----------------------------------+--------------------------------------
Type: Bug | Status: closed
Component: Testing framework | Version: 4.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Thorben Luepkes):
Replying to [comment:7 Mariusz Felisiak]:
> > Is there a different way apps get included now?
>
> It's not about apps but about tags. In your case `main_market` and
`main_market_receipt` failed to load which should not happened, even when
they do not match tags. If you want to debug your issue with the help of
other Django folks you should use support channels, Trac is not one of
them.
I know this is closed now. And I am very sorry for the late reply. I have
now managed to create a little reproducible repo. I have extracted
everything I could into it and managed to reproduce this bug.
Could you please take a look at
https://github.com/Thorbenl/django4-testrunner again?
This is exactly our market setup, and now it might be easier for you guys
to assists on this. I am very sorry for the confusion again :)
--
Ticket URL: <https://code.djangoproject.com/ticket/33454#comment:8>
Django
-----------------------------------+--------------------------------------
Type: Bug | Status: closed
Component: Testing framework | Version: 4.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak):
Replying to [comment:8 Thorben Luepkes]:
> Could you please take a look at
https://github.com/Thorbenl/django4-testrunner again?
As [https://docs.djangoproject.com/en/stable/topics/testing/overview
/#running-tests documented]: ''"Test discovery is based on the unittest
module’s built-in test discovery. By default, this **will discover tests
in any file named “test*.py” under the current working directory.**"'' You
can pass a test label to ignore a specific market, e.g.
{{{
./manage.py test market_a_app --tag=market_a --timing --keepdb --verbosity
2
./manage.py test market_b_app --tag=market_b --timing --keepdb --verbosity
2
}}}
or you can use conditional imports:
{{{#!python
if settings.MARKET == MARKET_A:
from .models import AdvertisementIdMapping
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33454#comment:9>