[Django] #37299: Allow being able to run a specific PW test or class for multiple browsers

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------
Reporter: Varun Kasyap | Type:
Pentamaraju | Cleanup/optimization
Status: new | Component: Testing
| framework
Version: dev | Severity: Normal
Keywords: Playwright | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
you can run the full Playwright suite or a module against several browsers
at once. But, if you ask for one test class or one test, only the first
browser actually runs. The other browsers are ignored.

For example:
{{{
python tests/runtests.py --playwright=chromium,firefox,webkit --parallel=1
admin_changelist.tests.PlaywrightTests
Found 10 test(s).
}}}

But 10 * 3 = 30 tests should run.

One more example:
{{{
python tests/runtests.py --playwright=chromium,firefox,webkit --parallel=1
admin_changelist.tests.PlaywrightTests.test_add_row_selection
Found 1 test(s).
}}}

But three tests should run.

This issue is not seen when running the full module:
{{{
python tests/runtests.py --playwright=chromium,firefox,webkit --parallel=1
admin_changelist.tests
Found 30 test(s).
}}}

After the fix, the same command you already use to target a class or a
single test should honor every browser you pass.
--
Ticket URL: <https://code.djangoproject.com/ticket/37299>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------

Reporter: Varun Kasyap | Owner: Zubair
Pentamaraju | Hassan
Type: | Status: assigned
Cleanup/optimization |

Component: Testing framework | Version: dev

Severity: Normal | Resolution:

Keywords: Playwright | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Zubair Hassan):

* owner: (none) => Zubair Hassan
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/37299#comment:1>

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------

Reporter: Varun Kasyap | Owner: (none)
Pentamaraju |
Type: | Status: new

Cleanup/optimization |
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: Playwright | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Zubair Hassan):

* owner: Zubair Hassan => (none)
* status: assigned => new

--
Ticket URL: <https://code.djangoproject.com/ticket/37299#comment:2>

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------
Reporter: Varun Kasyap | Owner: (none)
Pentamaraju |
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: Playwright | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by AJ Collins):

Seems like this was confirmed as an existing issue before PlayWright in
[[https://github.com/django/django/pull/21596#discussion_r3559822589|PR
#21596]].
--
Ticket URL: <https://code.djangoproject.com/ticket/37299#comment:3>

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------

Reporter: Varun Kasyap | Owner: Zubair
Pentamaraju | Hassan
Type: | Status: assigned

Cleanup/optimization |
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: Playwright | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Zubair Hassan):

* owner: (none) => Zubair Hassan
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/37299#comment:4>

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------
Reporter: Varun Kasyap | Owner: Zubair
Pentamaraju | Hassan
Type: | Status: assigned
Cleanup/optimization |
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: Playwright | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by AJ Collins):

== Our debugging process (skip to summary if you want a shorter answer)
We have narrowed down the bug to `test/runner.py` in
`DiscoverRunner.load_tests_for_label` method:
- When running a single test, `tests` is assigned to
`self.test_loader.loadTestsFromName(label)` and returned. This is where it
**doesn't** find additional browsers.
- When running a full test module, `tests` is assigned to
`self.test_loader.discover(start_dir=label, **kwargs)` and returned. This
correctly adds the additional browser tests.


== Single Test (bug case)
We commented out all but one `PlaywrightTests` to make debugging easier.
When `tests = self.test_loader.loadTestsFromName(label)` runs, we get this
debugger output:
{{{
(Pdb) pp tests
<unittest.suite.TestSuite tests=[<admin_changelist.tests.PlaywrightTests
testMethod=test_add_row_selection>]>
}}}
`admin_changelist.tests.PlaywrightTests` is the first browser in the list
to run the tests with

== Full Test Module (happy case)
`tests = self.test_loader.loadTestsFromName(label)` returns no tests, so
`load_tests_for_label` continues, ultimately finding all tests properly
with `self.test_loader.discover`
{{{
(Pdb) pp tests
...<unittest.suite.TestSuite
tests=[<admin_changelist.tests.FirefoxPlaywrightTests
testMethod=test_add_row_selection>]> .. .<unittest.suite.TestSuite
tests=[<admin_changelist.tests.PlaywrightTests
testMethod=test_add_row_selection>]
}}}

== Summary
When you pass two browsers, a second test class with its own methods is
generated, in our case it's called
`admin_changelist.tests.FirefoxPlaywrightTests`. So when you specify a
single test (e.g.
`admin_changelist.tests.PlaywrightTests.test_add_row_selection`) it's only
going to run that one. It will not run the generated one:
`admin_changelist.tests.FirefoxPlaywrightTests.test_add_row_selection`...

A fun discovery is that you technically **can** run the other browser
tests, you just need to specify them as well
{{{
python tests/runtests.py --playwright=chromium,firefox --parallel=1
admin_changelist.tests.PlaywrightTests.test_add_row_selection
admin_changelist.tests.FirefoxPlaywrightTests.test_add_row_selection
--verbosity 2
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/37299#comment:5>

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------
Reporter: Varun Kasyap | Owner: Zubair
Pentamaraju | Hassan
Type: | Status: assigned
Cleanup/optimization |
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: Playwright | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Varun Kasyap Pentamaraju):

Thank you for the investigation Collins 👍
--
Ticket URL: <https://code.djangoproject.com/ticket/37299#comment:6>

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------
Reporter: Varun Kasyap | Owner: Zubair
Pentamaraju | Hassan
Type: | Status: assigned
Cleanup/optimization |
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: Playwright | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by AJ Collins):

This is a tough problem to solve technically, since the test script is
*technically* doing what the user asked. At the very least, I'd like to
see this documented in
https://docs.djangoproject.com/en/dev/internals/contributing/writing-code
/unit-tests/#running-the-playwright-tests .
--
Ticket URL: <https://code.djangoproject.com/ticket/37299#comment:7>

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------
Reporter: Varun Kasyap | Owner: Zubair
Pentamaraju | Hassan
Type: | Status: assigned
Cleanup/optimization |
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: Playwright | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by AJ Collins):

You're very welcome, I paired with Shubham Akhilesh Singh who deserves
credit for the investigation. Shoutout djangocon sprints! Replying to
[comment:6 Varun Kasyap Pentamaraju]:

> Thank you for the investigation Collins 👍

--
Ticket URL: <https://code.djangoproject.com/ticket/37299#comment:8>

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------
Reporter: Varun Kasyap | Owner: Zubair
Pentamaraju | Hassan
Type: | Status: assigned
Cleanup/optimization |
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: Playwright | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Varun Kasyap Pentamaraju):

Just in case you'd like to brainstorm more around Playwright, you can also
look into
https://github.com/django/django/pull/21596#issuecomment-5166852846
--
Ticket URL: <https://code.djangoproject.com/ticket/37299#comment:9>

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------
Reporter: Varun Kasyap | Owner: Zubair
Pentamaraju | Hassan
Type: | Status: assigned
Cleanup/optimization |
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: Playwright | Triage Stage:
| Unreviewed

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Zubair Hassan):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/37299#comment:10>

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------
Reporter: Varun Kasyap | Owner: Zubair
Pentamaraju | Hassan
Type: | Status: assigned
Cleanup/optimization |
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: Playwright | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Zubair Hassan):

{{{
python tests/runtests.py --playwright=chromium,firefox,webkit --parallel=1
admin_changelist.tests.PlaywrightTests

Found 30 test(s).

python tests/runtests.py --playwright=chromium,firefox,webkit --parallel=1
admin_changelist.tests.PlaywrightTests.test_add_row_selection

Found 3 test(s).
}}}

Now single test and class tests with multiple breowers is solved in the
above PR.
--
Ticket URL: <https://code.djangoproject.com/ticket/37299#comment:11>

[Django]

#37299: Allow being able to run a specific PW test or class for multiple browsers
-------------------------------------+-------------------------------------
Reporter: Varun Kasyap | Owner: Zubair
Pentamaraju | Hassan
Type: | Status: assigned
Cleanup/optimization |
Component: Testing framework | Version: dev
Severity: Normal | Resolution:

Keywords: Playwright | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Zubair Hassan):

* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/37299#comment:12>