[GSOC] Shifting to Py.Test and Improving the Test Suite
458 views
Skip to first unread message
Akshay Jaggi
Feb 27, 2014, 2:50:32 PM2/27/14
to django-d...@googlegroups.com
Sorry for starting a new thread but I thought a proposal should begin with a new thread.
(Please refer to the previous thread @ https://groups.google.com/forum/#!topic/django-developers/GO65Qndw4Hg )
I'm writing in points, for easier readability. This is a very brief summary I actually wrote down for myself.
Need for Improvement/ Current Problems
- Running of Test Cases is slow.
- Selection of what tests we want to run, and what all we don’t want to run is difficult
- We have better options available
Why Py.Test? (http://pytest.org)
- Widely Used
- Better reporting
- Distributed testing (speed up, especially on multi-core machines), Line Coverage, etc using plugins
- Third party plugins
Why Categorisation?
- Easy to manage tests
- User can specify own types suitable to him. For instance -> Phase-I, Phase-II, Regression, unit, module-1, system, integration,etc
- User can run tests in chunks (say of 2,3 test-classes belonging to a particular category) ->Faster run, as we run tests only pertaining to currently-being-modified module.
- Ease in specifying what tests to run
How will I go about it?
Currently SimpleTestCase is a subclass of unittest.TestCase. Two ways exist ->
- Make our own class, which gives all functionalities of unittest.TestCase in terms of Pytest constructs
- Modify SimpleTestCase, TransactionTestCase, TestCase to use Pytest Constructs
(Pytest anyway supports unittest TestCases, Why is this required? Difference in the way both define their constructs, causes incompatibility in certain cases. This results in false-positives. About 200 previously passing Tests are reported as failed when using pytest-django)
Redefine discovery and runner accordingly.
We don’t want broken tests. Check all Test Cases for false positives, especially those which do not have failure test cases defined.
Now add ability to categorise in SimpleTestCase -> just a list of tags, specified for each test-class. (or test function?)
Add ability in discovery to run only the tests, which have the tags as required.
Hence discovery can now be using paths, test-labels, pattern or tags. It can also be intersection or union of any combination of these.
Test cases to be run can be defined as defaults in a config file or through the command line.
I'll be glad to hear your views.
Thanks.
Akshay Jaggi
P.S. - Another way to categorise can be using a configuration file. We can specify groups, and add all test-labels that belong to that group in this file. This way, classification won't be declarative in code, but in a separate file.
Florian Apolloner
Feb 27, 2014, 2:59:56 PM2/27/14
to django-d...@googlegroups.com
Hi Akshay,
On Thursday, February 27, 2014 8:50:32 PM UTC+1, Akshay Jaggi wrote:
Why Py.Test? (http://pytest.org)
- Widely Used
So is nose and unittest, you'll need to add a bit more info to such statements.
- Better reporting
Better how exactly?
- Distributed testing (speed up, especially on multi-core machines), Line Coverage, etc using plugins
How would this work? We still have shared resources like the database where you can't just run 10 Test against in parallel.
Cheers,
Florian
Cheers,
Florian
Andrew Pashkin
Feb 27, 2014, 3:10:58 PM2/27/14
to django-d...@googlegroups.com
How would this work? We still have shared resources like the database where you can't just run 10 Test against in parallel.
- Distributed testing (speed up, especially on multi-core machines), Line Coverage, etc using plugins
There is django plugin for py.test exists, it is able to create separate DBs for each process.
Gwildor Sok
Feb 27, 2014, 3:17:10 PM2/27/14
to django-d...@googlegroups.com
Personally I'm a big fan of Py.test, simply because it's so simple and Pythonic to use. Simple functions with simple assert statements. That's all. For me this significantly lowers the threshold to write tests and requires less effort, which in the end results in way more tests written in Py.test than the testsuite Django is currently using.
This is a talk I watched when I was looking for alternative test suites and got me hooked: http://www.youtube.com/watch?v=DTNejE9EraI
This is a talk I watched when I was looking for alternative test suites and got me hooked: http://www.youtube.com/watch?v=DTNejE9EraI
Russell Keith-Magee
Feb 27, 2014, 6:42:17 PM2/27/14
to Django Developers
On Fri, Feb 28, 2014 at 4:17 AM, Gwildor Sok <gwild...@gmail.com> wrote:
Personally I'm a big fan of Py.test, simply because it's so simple and Pythonic to use. Simple functions with simple assert statements. That's all. For me this significantly lowers the threshold to write tests and requires less effort, which in the end results in way more tests written in Py.test than the testsuite Django is currently using.
This is an argument that I've never understood. Why is typing:
class MyTestCase(TestCase):
such a burden? Because once you've typed that, everything else in the class is "just a method". However, you also get the added benefit of test grouping, and if you've got common setup/teardown requirements, the class provides a place to put them.
I know py.test has a lot of fans, but I'm seriously unconvinced by the "it's so easy to write tests" argument.
For this proposal to be acceptable to core, it's going to need to start with a *comprehensive* analysis of why py.test is a better approach to take. Don't just assume everyone is on board.
Yours,
Russ Magee %-)
Andrew Pashkin
Apr 6, 2014, 11:24:47 AM4/6/14
to django-d...@googlegroups.com
Some Pytest advocacy:
1) Pytest has convenient tests collection options - you can just specify folder to run all tests in it. It is also possible to filter tests by regex, and select specific ones.
2) PyUnit:
class MyTestCase(TestCase):
def test_something(self):
expected_content = ...
response = self.client(...)
self.assertEqual(expected_content, response.content)
Pytest:
Pytest style is cleaner and complies PEP8. Assertions with assert are more readable. It is also possible to group tests, by placing them into class, without subclassing something.
4) In Pytest it is possible to define setup functions for modules, classes and specific tests.
5) Pytest also has fixture mechanism, which allows to use additional resources in tests by specifying fixture names as an arguments (like client) - it is good alternative to using setUp's and subclassing something.
6) pytest-xdist plugin provides parallel execution of tests.
7) Pytest has features for writing parametrized tests.
So Pytest is kind of cool and it is easy to migrate to it.
1) Pytest has convenient tests collection options - you can just specify folder to run all tests in it. It is also possible to filter tests by regex, and select specific ones.
2) PyUnit:
class MyTestCase(TestCase):
def test_something(self):
expected_content = ...
response = self.client(...)
self.assertEqual(expected_content, response.content)
Pytest:
def test_something(client):
expected_content = ...
response = self.client(...)
assert expected_content == response.content
response = self.client(...)
assert expected_content == response.content
Pytest style is cleaner and complies PEP8. Assertions with assert are more readable. It is also possible to group tests, by placing them into class, without subclassing something.
4) In Pytest it is possible to define setup functions for modules, classes and specific tests.
5) Pytest also has fixture mechanism, which allows to use additional resources in tests by specifying fixture names as an arguments (like client) - it is good alternative to using setUp's and subclassing something.
6) pytest-xdist plugin provides parallel execution of tests.
7) Pytest has features for writing parametrized tests.
So Pytest is kind of cool and it is easy to migrate to it.
Chris Wilson
Apr 6, 2014, 1:02:18 PM4/6/14
to django-d...@googlegroups.com
Hi Andrew,
I'm not a Django core contributor but just a user and occasional patcher.
I submit some comments on this proposal in the hope that they will be
useful.
On Sun, 6 Apr 2014, Andrew Pashkin wrote:
> Some Pytest advocacy:
> 1) Pytest has convenient tests collection options - you can just specify folder to run all tests in it. It is also possible to
> filter tests by regex, and select specific ones.
I do use pytest and it has a lot of good points. Running all tests in a
directory or a specific file is one of them, as is the better test
selection logic.
> class MyTestCase(TestCase):
> def test_something(self):
> expected_content = ...
> response = self.client(...)
> self.assertEqual(expected_content, response.content)
>
> Pytest:
>
> def test_something(client):
> expected_content = ...
> response = self.client(...)
> assert expected_content == response.content
But this is not one of them, at least not for me. I strongly dislike this
style of assertions:
> Pytest style is cleaner and complies PEP8. Assertions with assert are more readable.
* The difference between "self.assertEquals(a, b)" and "assert a == b" is
minimal for me.
* It hides a LOT of magic in how pytest goes back and evaluates the
assertion's arguments again if the assertion fails. Sometimes it gets
different results (if one of the tested methods has side effects) and then
the assertion message makes no sense, hides the problem, or pytest breaks.
* It makes it much harder to write custom assertions and get meaningful
display on error.
> It is also possible to group tests, by placing them into class without
> subclassing something.
Grouping tests with decorators allows tests to be members of multiple
suites, it's true. However this is not something that I've ever needed.
And losing the power of inheritance to add methods to my test classes, by
not putting tests into classes, is enough to kill this style completely
for me. I write a LOT of my own assertions:
https://github.com/aptivate/django-harness/tree/master/django_harness
> 4) In Pytest it is possible to define setup functions for modules, classes and specific tests.
I can do this by adding mixins to test classes. Works fine for me, and
less magical.
> 5) Pytest also has fixture mechanism, which allows to use additional
> resources in tests by specifying fixture names as an arguments (like
> client) - it is good alternative to using setUp's and subclassing
> something.
We have named fixtures in Django test classes already, so I don't see that
as much of an advantage. We can't easily add fixtures in mixins at the
moment. I've not needed that so far.
> 6) pytest-xdist plugin provides parallel execution of tests.
> 7) Pytest has features for writing parametrized tests.
>
> So Pytest is kind of cool and it is easy to migrate to it.
I don't think it's necessary or wise to rewrite the test suite to use
pytest-style assertions. But better integration of Django and Pytest, and
the ability to write tests with Pytest-style assertions if you like, would
certainly have my support (fwiw).
Cheers, Chris.
--
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Citylife House, Sturton Street, Cambridge, CB1 2QF, UK
Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.
I'm not a Django core contributor but just a user and occasional patcher.
I submit some comments on this proposal in the hope that they will be
useful.
On Sun, 6 Apr 2014, Andrew Pashkin wrote:
> Some Pytest advocacy:
> 1) Pytest has convenient tests collection options - you can just specify folder to run all tests in it. It is also possible to
> filter tests by regex, and select specific ones.
directory or a specific file is one of them, as is the better test
selection logic.
> class MyTestCase(TestCase):
> def test_something(self):
> expected_content = ...
> response = self.client(...)
> self.assertEqual(expected_content, response.content)
>
> Pytest:
>
> def test_something(client):
> expected_content = ...
> response = self.client(...)
> assert expected_content == response.content
style of assertions:
> Pytest style is cleaner and complies PEP8. Assertions with assert are more readable.
minimal for me.
* It hides a LOT of magic in how pytest goes back and evaluates the
assertion's arguments again if the assertion fails. Sometimes it gets
different results (if one of the tested methods has side effects) and then
the assertion message makes no sense, hides the problem, or pytest breaks.
* It makes it much harder to write custom assertions and get meaningful
display on error.
> It is also possible to group tests, by placing them into class without
> subclassing something.
Grouping tests with decorators allows tests to be members of multiple
suites, it's true. However this is not something that I've ever needed.
And losing the power of inheritance to add methods to my test classes, by
not putting tests into classes, is enough to kill this style completely
for me. I write a LOT of my own assertions:
https://github.com/aptivate/django-harness/tree/master/django_harness
> 4) In Pytest it is possible to define setup functions for modules, classes and specific tests.
less magical.
> 5) Pytest also has fixture mechanism, which allows to use additional
> resources in tests by specifying fixture names as an arguments (like
> client) - it is good alternative to using setUp's and subclassing
> something.
as much of an advantage. We can't easily add fixtures in mixins at the
moment. I've not needed that so far.
> 6) pytest-xdist plugin provides parallel execution of tests.
> 7) Pytest has features for writing parametrized tests.
>
> So Pytest is kind of cool and it is easy to migrate to it.
pytest-style assertions. But better integration of Django and Pytest, and
the ability to write tests with Pytest-style assertions if you like, would
certainly have my support (fwiw).
Cheers, Chris.
--
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Citylife House, Sturton Street, Cambridge, CB1 2QF, UK
Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.
Łukasz Rekucki
Apr 6, 2014, 1:07:20 PM4/6/14
to django-developers
On 6 April 2014 17:24, Andrew Pashkin <andrew....@gmx.co.uk> wrote:
> Some Pytest advocacy:
> 1) Pytest has convenient tests collection options - you can just specify
> folder to run all tests in it. It is also possible to filter tests by regex,
> and select specific ones.
Sounds good, but unittest's test discovery is not that bad and can be
> Some Pytest advocacy:
> 1) Pytest has convenient tests collection options - you can just specify
> folder to run all tests in it. It is also possible to filter tests by regex,
> and select specific ones.
easily improved.
>
> 2) PyUnit:
>
> class MyTestCase(TestCase):
> def test_something(self):
> expected_content = ...
> response = self.client(...)
> self.assertEqual(expected_content, response.content)
>
> Pytest:
>
> def test_something(client):
> expected_content = ...
> response = self.client(...)
> assert expected_content == response.content
>
> Pytest style is cleaner and complies PEP8.
unittest, so it's hard to argue that Pytest is more in line with
Python standard library recommendation (which what PEP8 is).
> Assertions with assert are more readable.
introspection" is rather scary. And you still need custom
methods/functions for non-trivial assertions like assertHTMLEquals or
assertQuerysetEquals.
> It is also possible to group tests, by placing them into class, without subclassing something.
> 4) In Pytest it is possible to define setup functions for modules, classes
> and specific tests.
https://docs.python.org/2/library/unittest.html#class-and-module-fixtures
> 6) pytest-xdist plugin provides parallel execution of tests.
shared resources (like the database - and no creating a new Oracle
instance in every subprocess is a non-starter!). So while having some
support for that is good, it's not something Django's test suite would
get for free.
> 5) Pytest also has fixture mechanism, which allows to use additional
> resources in tests by specifying fixture names as an arguments (like client)
> - it is good alternative to using setUp's and subclassing something.
> 7) Pytest has features for writing parametrized tests.
>
These two features are actually something worth looking at
>
(parametrized tests would allow cleaning up a lot of template tests),
but I'm guessing that for the core developers to get convinced you'd
have to show that this two will make a real difference (instead of
focusing on the less important stuff mentioned earlier).
Best regards,
Lucas Rekucki
Andrew Pashkin
Apr 6, 2014, 1:55:54 PM4/6/14
to django-d...@googlegroups.com
> ...Sometimes it gets different results (if one of the tested methods
> has side effects) and then the assertion message makes no sense, hides
> the problem, or pytest breaks.
>
> * It makes it much harder to write custom assertions and get
> meaningful display on error.
Can you give an examples for cases with messages/breakings and for
> the problem, or pytest breaks.
>
> * It makes it much harder to write custom assertions and get
> meaningful display on error.
custom assertions?
Regarding magic in Pytest - yeah - there is some =))
But if think deeper, inheritance and self is also magic, and Pytest just
uses its own magic (dependency injection) that is differs from OO magic =)
Actually it would be interesting to discuss cases where Pytest cant
handle (or handle with convennience) what PyUnit can and vice versa.
м.т - +7 (985) 898 57 59
Skype - waves_in_fluids
e-mail - andrew....@gmx.co.uk
Chris Wilson
Apr 6, 2014, 3:19:36 PM4/6/14
to django-d...@googlegroups.com
Hi Andrew,
On Sun, 6 Apr 2014, Andrew Pashkin wrote:
>> * It makes it much harder to write custom assertions and get meaningful
>> display on error.
>
> Can you give an examples for cases with messages/breakings and for custom
> assertions?
I don't have an example of breakage to hand, I ripped out the offending
code in disgust and replaced it with standard assertions that worked. If I
find another example I'll be sure to let you know before I do the same
again.
Regarding custom assertions, how would you write assert_not_redirected
from
https://github.com/aptivate/django-harness/blob/master/django_harness/fast_dispatch.py
as a PyTest assertion? Consider the following tests:
def not_redirected(response):
return (response.status_code != 302)
class RedirectViewTests(FastDispatchMixin, TestCase):
def test_redirect_view_with_custom_assertion(self):
response = self.fast_dispatch('my-redirecting-view')
self.assert_not_redirected(response)
def test_redirect_view_with_pytest_assertion(self):
response = self.fast_dispatch('my-redirecting-view')
assert not_redirected(response)
assert_not_redirected is from FastDispatchMixin in
https://github.com/aptivate/django-harness/blob/master/django_harness/fast_dispatch.py,
not_redirected is a simple assertion I wrote as a demo, since I don't use
pytest much myself for this reason. Now compare the output (reformatted
for readability):
____________________________________
RedirectViewTests.test_redirect_view_with_custom_assertion
____________________________________
main/tests/tests.py:71: in test_redirect_view_with_custom_assertion
> self.assert_not_redirected(response)
.ve/src/django-harness/django_harness/fast_dispatch.py:114: in
assert_not_redirected
> msg_prefix)
E AssertionError: 302 != 200 : unexpectedly redirected to
http://www.google.com/
____________________________________
RedirectViewTests.test_redirect_view_with_pytest_assertion
____________________________________
main/tests/tests.py:75: in test_redirect_view_with_pytest_assertion
> assert not_redirected(response)
E AssertionError: assert
not_redirected(<django.http.response.HttpResponseRedirect object at
0x493da10>)
I.e. my assertion outputs:
> AssertionError: 302 != 200 : unexpectedly redirected to
> http://www.google.com/
and the simple Python assertion outputs this:
> AssertionError: assert
> not_redirected(<django.http.response.HttpResponseRedirect object at
> 0x493da10>)
How would you generate a useful error message, bearing in mind that you
can't really access response._headers['location'][1] without first having
checked that the response is a redirect, which isn't the normal case?
> Regarding magic in Pytest - yeah - there is some =)) But if think
> deeper, inheritance and self is also magic, and Pytest just uses its own
> magic (dependency injection) that is differs from OO magic =)
OO "magic" is well known and understood by most Python developers.
Pytest's magic is (a) largely undocumented (afaik) and (b) relies on
understanding of Python bytecode which is definitely not common knowledge.
On Sun, 6 Apr 2014, Andrew Pashkin wrote:
>> * It makes it much harder to write custom assertions and get meaningful
>> display on error.
>
> Can you give an examples for cases with messages/breakings and for custom
> assertions?
code in disgust and replaced it with standard assertions that worked. If I
find another example I'll be sure to let you know before I do the same
again.
Regarding custom assertions, how would you write assert_not_redirected
from
https://github.com/aptivate/django-harness/blob/master/django_harness/fast_dispatch.py
as a PyTest assertion? Consider the following tests:
def not_redirected(response):
return (response.status_code != 302)
class RedirectViewTests(FastDispatchMixin, TestCase):
def test_redirect_view_with_custom_assertion(self):
response = self.fast_dispatch('my-redirecting-view')
self.assert_not_redirected(response)
def test_redirect_view_with_pytest_assertion(self):
response = self.fast_dispatch('my-redirecting-view')
assert not_redirected(response)
assert_not_redirected is from FastDispatchMixin in
https://github.com/aptivate/django-harness/blob/master/django_harness/fast_dispatch.py,
not_redirected is a simple assertion I wrote as a demo, since I don't use
pytest much myself for this reason. Now compare the output (reformatted
for readability):
____________________________________
RedirectViewTests.test_redirect_view_with_custom_assertion
____________________________________
main/tests/tests.py:71: in test_redirect_view_with_custom_assertion
> self.assert_not_redirected(response)
.ve/src/django-harness/django_harness/fast_dispatch.py:114: in
assert_not_redirected
> msg_prefix)
E AssertionError: 302 != 200 : unexpectedly redirected to
http://www.google.com/
____________________________________
RedirectViewTests.test_redirect_view_with_pytest_assertion
____________________________________
main/tests/tests.py:75: in test_redirect_view_with_pytest_assertion
> assert not_redirected(response)
E AssertionError: assert
not_redirected(<django.http.response.HttpResponseRedirect object at
0x493da10>)
I.e. my assertion outputs:
> AssertionError: 302 != 200 : unexpectedly redirected to
> http://www.google.com/
and the simple Python assertion outputs this:
> AssertionError: assert
> not_redirected(<django.http.response.HttpResponseRedirect object at
> 0x493da10>)
How would you generate a useful error message, bearing in mind that you
can't really access response._headers['location'][1] without first having
checked that the response is a redirect, which isn't the normal case?
> Regarding magic in Pytest - yeah - there is some =)) But if think
> deeper, inheritance and self is also magic, and Pytest just uses its own
> magic (dependency injection) that is differs from OO magic =)
Pytest's magic is (a) largely undocumented (afaik) and (b) relies on
understanding of Python bytecode which is definitely not common knowledge.
Chris Wilson
Apr 7, 2014, 5:52:12 AM4/7/14
to django-d...@googlegroups.com
Hi Andrew,
On Sun, 6 Apr 2014, Chris Wilson wrote:
> On Sun, 6 Apr 2014, Andrew Pashkin wrote:
>
>>> * It makes it much harder to write custom assertions and get meaningful
>>> display on error.
>>
>> Can you give an examples for cases with messages/breakings and for custom
>> assertions?
>
> I don't have an example of breakage to hand, I ripped out the offending code
> in disgust and replaced it with standard assertions that worked. If I find
> another example I'll be sure to let you know before I do the same again.
OK, here is one.
chris@lap-x201:~/projects/2014/webassets$ .ve/bin/tox -e py27 --
tests.test_filters:TestPyScss.test_search_path
GLOB sdist-make: /home/chris/projects/2014/webassets/setup.py
py27 inst-nodeps:
/home/chris/projects/2014/webassets/.tox/dist/webassets-0.10.dev.zip
py27 runtests: PYTHONHASHSEED='894115171'
py27 runtests: commands[0] | nosetests
tests.test_filters:TestPyScss.test_search_path
F
======================================================================
FAIL: tests.test_filters.TestPyScss.test_search_path
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/chris/projects/2014/webassets/.tox/py27/local/lib/python2.7/site-packages/nose/case.py",
line 197, in runTest
self.test(*self.arg)
File "/home/chris/projects/2014/webassets/tests/test_filters.py", line
969, in test_search_path
assert self.get('out.css') == 'h1 {\n color: #0000ff;\n}\na {\n
color: #ff8000;\n}\n'
AssertionError
----------------------------------------------------------------------
Ran 1 test in 0.055s
FAILED (failures=1)
"AssertionError". Gee, thanks. That helps a lot. So if the string wasn't
what we expected, what WAS it? How is this anywhere NEAR as useful as
self.assertEqual()?
On Sun, 6 Apr 2014, Chris Wilson wrote:
> On Sun, 6 Apr 2014, Andrew Pashkin wrote:
>
>>> * It makes it much harder to write custom assertions and get meaningful
>>> display on error.
>>
>> Can you give an examples for cases with messages/breakings and for custom
>> assertions?
>
> I don't have an example of breakage to hand, I ripped out the offending code
> in disgust and replaced it with standard assertions that worked. If I find
> another example I'll be sure to let you know before I do the same again.
chris@lap-x201:~/projects/2014/webassets$ .ve/bin/tox -e py27 --
tests.test_filters:TestPyScss.test_search_path
GLOB sdist-make: /home/chris/projects/2014/webassets/setup.py
py27 inst-nodeps:
/home/chris/projects/2014/webassets/.tox/dist/webassets-0.10.dev.zip
py27 runtests: PYTHONHASHSEED='894115171'
py27 runtests: commands[0] | nosetests
tests.test_filters:TestPyScss.test_search_path
F
======================================================================
FAIL: tests.test_filters.TestPyScss.test_search_path
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/chris/projects/2014/webassets/.tox/py27/local/lib/python2.7/site-packages/nose/case.py",
line 197, in runTest
self.test(*self.arg)
File "/home/chris/projects/2014/webassets/tests/test_filters.py", line
969, in test_search_path
assert self.get('out.css') == 'h1 {\n color: #0000ff;\n}\na {\n
color: #ff8000;\n}\n'
AssertionError
----------------------------------------------------------------------
Ran 1 test in 0.055s
FAILED (failures=1)
"AssertionError". Gee, thanks. That helps a lot. So if the string wasn't
what we expected, what WAS it? How is this anywhere NEAR as useful as
self.assertEqual()?
Andreas Pelme
Apr 7, 2014, 8:13:51 AM4/7/14
to django-d...@googlegroups.com
Hi Chris,
It looks like you invoke nosetests and not py.test, therefore you do not get the results one would expect with py.test:
> chris@lap-x201:~/projects/2014/webassets$ .ve/bin/tox -e py27 — tests.test_filters:TestPyScss.test_search_path
...
> py27 runtests: commands[0] | nose tests tests.test_filters:TestPyScss.test_search_path
...
> File "/home/chris/projects/2014/webassets/.tox/py27/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
This is what the output from a similar test with pytest would result in:
(Screenshot at http://pelme.se/~andreas/private/pytest_assertion.png to give the correct colors/formatting.)
$ py.test test_foo.py
========================================= test session starts ==========================================
platform darwin -- Python 2.7.5 -- pytest-2.5.2.dev1
plugins: xdist
collected 1 items
test_foo.py F
=============================================== FAILURES ===============================================
____________________________________________ test_something ____________________________________________
def test_something():
foo = {'out.css': 'something'}
> assert foo.get('out.css') == 'h1 {\n color: #0000ff;\n}\na {\n color: #ff8000;\n}\n'
E assert 'something' == 'h1 {\n color: #0000ff...\n color: #ff8000;\n}\n'
E - something
E + h1 {
E + color: #0000ff;
E + }
E + a {
E + color: #ff8000;
E + }
test_foo.py:4: AssertionError
======================================= 1 failed in 0.01 seconds =======================================
Cheers,
Andreas
It looks like you invoke nosetests and not py.test, therefore you do not get the results one would expect with py.test:
> chris@lap-x201:~/projects/2014/webassets$ .ve/bin/tox -e py27 — tests.test_filters:TestPyScss.test_search_path
...
> py27 runtests: commands[0] | nose tests tests.test_filters:TestPyScss.test_search_path
...
> File "/home/chris/projects/2014/webassets/.tox/py27/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
(Screenshot at http://pelme.se/~andreas/private/pytest_assertion.png to give the correct colors/formatting.)
$ py.test test_foo.py
========================================= test session starts ==========================================
platform darwin -- Python 2.7.5 -- pytest-2.5.2.dev1
plugins: xdist
collected 1 items
test_foo.py F
=============================================== FAILURES ===============================================
____________________________________________ test_something ____________________________________________
def test_something():
foo = {'out.css': 'something'}
> assert foo.get('out.css') == 'h1 {\n color: #0000ff;\n}\na {\n color: #ff8000;\n}\n'
E assert 'something' == 'h1 {\n color: #0000ff...\n color: #ff8000;\n}\n'
E - something
E + h1 {
E + color: #0000ff;
E + }
E + a {
E + color: #ff8000;
E + }
test_foo.py:4: AssertionError
======================================= 1 failed in 0.01 seconds =======================================
Cheers,
Andreas
Aymeric Augustin
Apr 7, 2014, 8:34:25 AM4/7/14
to django-d...@googlegroups.com
Before this thread goes too far, I'd like to express my doubts regarding the goals you stated.
Need for Improvement/ Current Problems
- Running of Test Cases is slow.
I don't know what the threshold for "slow" is, but faster is obviously better ;-) However, could you clarify how changing the test runner will make the tests run faster?
I gained a little bit of experience in that area when I helped divide by three the run time of the test suite last year. I don't believe unittest even registers on the radar of things that make the tests slow. If you haven't heard the "Database is Hot Lava" meme, google it.
- Selection of what tests we want to run, and what all we don’t want to run is difficult
That problem can be solved by a custom test runner extending unittest, without changing — and maybe breaking — the semantics of the tests.
I've been working on this codebase for some time and I have a hard time to believe someone will actually audit all of the 5000 test cases. Some are really tricky — like, with threads, database transactions, hacks for in-memory SQLite, and some selenium.
- We have better options available
It's hard to take such a blanket statement at face value. You need to argue why py.test is better *for Django*. For starters, using standard tools from the standard library helps lowering the barrier to contributing. Every change must come with a test, and if that involves learning a new test runner and a complex categorisation system, that's an additional barrier.
Don't get me wrong -- I know that py.test is a good tool. I'm just saying that you shouldn't start rewriting chunks of Django's test suite without convincing the core team that you understand the problem you're trying to solve.
--
Aymeric.
Carl Meyer
Apr 14, 2014, 1:43:23 PM4/14/14
to django-d...@googlegroups.com
Hi Chris,
Just a couple notes to set the record straight:
Just a couple notes to set the record straight:
This has not been true since pytest 2.1.0, released July 2011. Since then, pytest rewrites assertions in the AST via an import hook instead of re-evaluating assertion arguments on failure, so there are no such problems with side effects in one side of an assertion. See http://pytest.org/latest/assert.html for details.
* It makes it much harder to write custom assertions and get meaningful
display on error.
> It is also possible to group tests, by placing them into class without
> subclassing something.
Grouping tests with decorators allows tests to be members of multiple
suites, it's true. However this is not something that I've ever needed.
And losing the power of inheritance to add methods to my test classes, by
not putting tests into classes, is enough to kill this style completely
for me. I write a LOT of my own assertions:
https://github.com/aptivate/django-harness/tree/master/django_harness
> 4) In Pytest it is possible to define setup functions for modules, classes and specific tests.
I can do this by adding mixins to test classes. Works fine for me, and
less magical.
> 5) Pytest also has fixture mechanism, which allows to use additional
> resources in tests by specifying fixture names as an arguments (like
> client) - it is good alternative to using setUp's and subclassing
> something.
We have named fixtures in Django test classes already, so I don't see that
as much of an advantage. We can't easily add fixtures in mixins at the
moment. I've not needed that so far.
Pytest fixtures are a much more general and powerful concept than Django fixtures; they allow you to write modular code to setup and teardown any type of resource that a test may need.
Django's fixtures are limited to data dumps; IMO data-dump fixtures cause test maintainability problems and should usually be avoided (in favor of object factories).
Andrew Pashkin
May 9, 2014, 3:50:54 AM5/9/14
to django-d...@googlegroups.com
Another advantage of Pytest is its ability to launch tests written in
all styles - PyUnit, Nose, and Pytest own style.
Recently I understood that it is a good practice to launch tests not
only for project, but for all packages it uses before deployment - to
ensure that nothing is broken (like missing system library that causes
failure in functionality of image manipulation package, etc).
all styles - PyUnit, Nose, and Pytest own style.
Recently I understood that it is a good practice to launch tests not
only for project, but for all packages it uses before deployment - to
ensure that nothing is broken (like missing system library that causes
failure in functionality of image manipulation package, etc).
Reply all
Reply to author
Forward
0 new messages