[Django] #37072: assertWarnsMessage() also ignores entire warning category

[Django]

#37072: assertWarnsMessage() also ignores entire warning category
-------------------------------------+-------------------------------------
Reporter: Mike Edmunds | Type: Bug
Status: new | Component: Testing
| framework
Version: 6.0 | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
`assertWarnsMessage(category, message)` implicitly does the equivalent of
`ignore_warnings(category)`. As a result, tests for particular deprecation
warnings can overlook other, unrelated deprecation warnings coming from
the code being tested.

You would expect the second test case below to cause an error for
`RemovedInDjango70Warning: Unrelated deprecation`, but it actually passes:

{{{#!python
class SomeTests(SimpleTestCase):
def test_this_errors_as_expected(self):
# ERROR: RemovedInDjango70Warning: Unrelated deprecation
warnings.warn("Unrelated deprecation", RemovedInDjango70Warning)

def test_so_this_should_also_error_but_does_not(self):
with self.assertWarnsMessage(RemovedInDjango70Warning, "Expected
deprecation"):
warnings.warn("Expected deprecation",
RemovedInDjango70Warning)
# This gets ignored.
warnings.warn("Unrelated deprecation",
RemovedInDjango70Warning)

def test_duplicate_warnings_should_maybe_also_error(self):
with self.assertWarnsMessage(RemovedInDjango70Warning, "Expected
deprecation"):
warnings.warn("Expected deprecation",
RemovedInDjango70Warning)
# This also gets ignored. Maybe it shouldn't?
warnings.warn("Expected deprecation",
RemovedInDjango70Warning)
}}}

The problem is `assertWarnsMessage()` and `assertRaisesMessage()` try to
share `SimpleTestCase._assertFooMessage()`, which wraps `assertWarns()`
and `assertRaises()` respectively. But those methods aren't really
parallel: there can be only one error raised, but execution can continue
after a warning. `assertWarns(category)` captures ''all'' warnings in the
category.
--
Ticket URL: <https://code.djangoproject.com/ticket/37072>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#37072: assertWarnsMessage() also ignores entire warning category

-----------------------------------+--------------------------------------
Reporter: Mike Edmunds | Owner: (none)

Type: Bug | Status: new
Component: Testing framework | Version: 6.0

Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------------+--------------------------------------
Comment (by Mike Edmunds):

Here's a test case that could be added to tests/test_utils/tests.py to
verify a fix:

{{{#!python
class AssertWarnsMessageTests(SimpleTestCase):
...

def test_does_not_ignore_entire_category(self):
with (
self.assertRaisesMessage(RemovedInNextVersionWarning,
"Unexpected"),
self.assertWarnsMessage(RemovedInNextVersionWarning,
"Expected"),
):
warnings.warn("Expected", RemovedInNextVersionWarning)
warnings.warn("Unexpected", RemovedInNextVersionWarning)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/37072#comment:1>

[Django]

#37072: assertWarnsMessage() also ignores entire warning category
-----------------------------------+------------------------------------

Reporter: Mike Edmunds | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 6.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------

Changes (by Tim Graham):

* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/37072#comment:2>

[Django]

#37072: assertWarnsMessage() also ignores entire warning category
-----------------------------------+------------------------------------
Reporter: Mike Edmunds | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------

Comment (by Mike Edmunds):

See also #37076.

I'm thinking maybe `assertWarnsMessage()` should do something like:
- Capture all warnings while running the tested code
- Check that at least one captured warning matches the expected category
and message
- ''Optionally'' check that ''exactly'' one (or exactly a specified
`expected_count`) matches
- Re-emit any non-matching warnings that were captured, to be handled by
the original warning filters
--
Ticket URL: <https://code.djangoproject.com/ticket/37072#comment:3>