[Django] #34657: Testing assertions `assertContains` and `assertInHTML` should output the haystack on failure
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud | Owner: nobody
Colas |
Type: New | Status: new
feature |
Component: Testing | Version:
framework | Keywords: testcases, unit
Severity: Normal | tests, HTML, assertions, testing
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
If I use the vanilal Python `assertIn` or `assertRegex`, when the
assertion fails, it’s very simple to assess what went wrong from the test
error only:
{{{
self.assertIn("<b>hey</b>", "<p>Howdy!</p>")
AssertionError: '<b>hey</b>' not found in '<p>Howdy!</p>'
}}}
With Django’s `assertContains` and `assertInHTML`, this gets much harder:
{{{
self.assertInHTML("<b>hey</b>", "<p>Howdy!</p>")
File "/Users/thibaudcolas/Dev/django/html-testing-with-
django/.venv/lib/python3.11/site-packages/django/test/testcases.py", line
1076, in assertInHTML
self.assertTrue(
AssertionError: False is not true : Couldn't find '<b>
hey
</b>' in response
}}}
and:
{{{
self.assertContains(res, "<b>hey</b>")
File "/Users/thibaudcolas/Dev/django/html-testing-with-
django/.venv/lib/python3.11/site-packages/django/test/testcases.py", line
660, in assertContains
self.assertTrue(
AssertionError: False is not true : Couldn't find '<b>hey</b>' in response
}}}
---
In both cases, Django doesn’t display the haystack – so I have to waste a
lot of time going back to my HTML templates to check what they might be
outputting / or potentially load the same scenario in a browser / or
manually add `print` statements to my test cases. This is all very time-
consuming. Instead, it’d be much better if the haystack was just present.
Additionally for `assertInHTML` – it’s annoying that `needle` as displayed
in the failure message is the parser’s output, which is therefore broken
up over multiple lines. It’d be much nicer if the example above resulted
in `AssertionError: False is not true : Couldn't find '<b>hey</b>' in
response`.
---
Test suite I used to compare output for reference:
https://github.com/thibaudcolas/html-testing-with-
django/blob/39c82c410cce4bea71ac54be27be67057f4d8dd8/testing_tests/tests.py#L5
--
Ticket URL: <https://code.djangoproject.com/ticket/34657>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
haystack on failure
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Testing framework | Version:
Severity: Normal | Resolution:
Keywords: testcases, unit | Triage Stage:
tests, HTML, assertions, testing | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
New description:
and:
---
---
And link to Python’s `assertIn` / `assertNotIn` implementations:
https://github.com/python/cpython/blob/101d5ec7d7fe122fa81a377c8ab8b562d1add9ee/Lib/unittest/case.py#L1147-L1159
--
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:1>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version:
Severity: Normal | Resolution:
testing |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* keywords: testcases, unit tests, HTML, assertions, testing => HTML,
assertions, testing
* stage: Unreviewed => Accepted
Comment:
Tentatively accepting because this is related to the testing framework,
though I'm not 100% sure on procedure whether this specific case needs a
forum post since it's marked as a new feature. I would argue this is a
cleanup/optimization :thinking:
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:2>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version:
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Sarah Boyce):
I can see the value of this. The react testing library we use has a
setting `DEBUG_PRINT_LIMIT` (https://testing-library.com/docs/dom-testing-
library/api-debugging/#automatic-logging) with a default of 7000
characters but you can increase it if you need to see more printed. I
recommend that we have a similar setting 👍
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:3>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version:
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak):
Replying to [comment:2 Natalia Bidart]:
> Tentatively accepting because this is related to the testing framework,
though I'm not 100% sure on procedure whether this specific case needs a
forum post since it's marked as a new feature. I would argue this is a
cleanup/optimization :thinking:
Creating a new setting is always controversial (we already have many of
them) and users can already control it with `maxDiff`.
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:4>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version:
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Thibaud Colas):
Sorry, I didn’t realise feature requests require a forum post! I’ll know
for next time.
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:5>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version:
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Shamil Abdulaev):
I have fixed this bug and sent a PR
https://github.com/django/django/pull/17076 for consideration!)
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:6>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: nobody
Type: New feature | Status: new
Component: Testing framework | Version:
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
* has_patch: 0 => 1
* needs_tests: 0 => 1
* needs_docs: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:7>
Django
haystack on failure
-------------------------------------+-------------------------------------
| Abdulaev
Type: New feature | Status: assigned
Component: Testing framework | Version:
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Sarah Boyce):
* owner: nobody => Shamil Abdulaev
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:8>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Shamil
| Abdulaev
Component: Testing framework | Version:
Severity: Normal | Resolution: fixed
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:9>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Shamil
| Abdulaev
Component: Testing framework | Version:
Severity: Normal | Resolution:
testing |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: closed => new
* resolution: fixed =>
Comment:
Hi Shamil, we still need to review and merge in a PR before we mark a
ticket as fixed. In general I would leave the status alone as mostly the
fellows take care of this
I've written a couple of comments on the PR, once they're sorted we can
update the ticket to go into the review queue
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:10>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Shamil
| Abdulaev
Type: New feature | Status: new
Component: Testing framework | Version:
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Chinmoy):
Hey, is this still being worked upon? Can I assign myself to this ticket?
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:11>
Django
haystack on failure
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: Shamil Abdulaev => Chinmoy
* status: new => assigned
Comment:
Replying to [comment:11 Chinmoy]:
> Hey, is this still being worked upon? Can I assign myself to this
ticket?
Hi! Thank you for your interest in contributing. I checked and both
related PRs are closed so if you can work on this, it'll be certainly
welcomed! Thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:12>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Type: New feature | Status: assigned
Component: Testing framework | Version:
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Changes (by Natalia Bidart):
* needs_better_patch: 1 => 0
* has_patch: 1 => 0
* needs_tests: 1 => 0
* needs_docs: 1 => 0
Comment:
Resetting patch flags.
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:13>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Type: New feature | Status: assigned
Component: Testing framework | Version:
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Chinmoy (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:14>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Type: New feature | Status: assigned
Component: Testing framework | Version:
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Chinmoy):
Hey, sorry I missed this in my mail. Will submit a PR soon.
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:15>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Type: New feature | Status: assigned
Component: Testing framework | Version:
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
Here is the PR [https://github.com/django/django/pull/17204].
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:16>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Type: New feature | Status: assigned
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
* version: => dev
* needs_docs: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:17>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
* needs_docs: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:18>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_tests: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:19>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_tests: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:20>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"e99c7d8847e9006f877ab3cea47f1977652af71f" e99c7d88]:
{{{
#!CommitTicketReference repository=""
revision="e99c7d8847e9006f877ab3cea47f1977652af71f"
Refs #34657 -- Made assertInHTML() use unparsed needle in error messages.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:21>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:22>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: HTML, assertions, | Triage Stage: Accepted
testing |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by GitHub <noreply@…>):
In [changeset:"679212a47ac3e22a6fbb6ee3cd4c09f29aae8b5d" 679212a]:
{{{
#!CommitTicketReference repository=""
revision="679212a47ac3e22a6fbb6ee3cd4c09f29aae8b5d"
Refs #34657 -- Made msg_prefix handling in assertURLEqual()/assertInHTML
consistent with other assertions.
Co-authored-by: Chinmoy Chakraborty <chinm...@gmail.com>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:23>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
testing | checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:24>
Django
haystack on failure
-------------------------------------+-------------------------------------
Reporter: Thibaud Colas | Owner: Chinmoy
Component: Testing framework | Version: dev
Keywords: HTML, assertions, | Triage Stage: Ready for
testing | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"1dae65dc63ae84be5002c37b4ddae0b9220e8808" 1dae65dc]:
{{{
#!CommitTicketReference repository=""
revision="1dae65dc63ae84be5002c37b4ddae0b9220e8808"
Fixed #34657 -- Made assert(Not)Contains/assertInHTML display haystacks in
error messages.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/34657#comment:25>