Please consider adding context manager versions of setUp/tearDown to unittest.TestCase

[Neil Girdhar]

This question describes an example of the problem: https://stackoverflow.com/questions/8416208/in-python-is-there-a-good-idiom-for-using-context-managers-in-setup-teardown.  You want to invoke a context manager in your setup/tearing-down, but the easiest way to do that is to override run, which seems ugly.

Why not add two methods to unittest.TestCase whose default implementations are given below:

class TestCase:

    @contextmanager

    def method_context(self):

        self.setUp()

        try:

            yield

        finally:

            self.tearDown()

    @contextmanager

    def class_context(self):

        self.setUpClass()

        try:

            yield

        finally:

            self.tearDown()

Then, if for example someone wants to use a context manager in setUp, they can do so:

class SomeTest(TestCase):

    @contextmanager

    def method_context(self):

        with np.errstate(all='raise'):

            with super().method_context():

                yield

Best,

Neil

[Ned Batchelder]

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

I've achieved a similar effect with this:

def setup_with_context_manager(testcase, cm):
    """Use a contextmanager to setUp a test case.

    If you have a context manager you like::

        with ctxmgr(a, b, c) as v:
            # do something with v

    and you want to have that effect for a test case, call this function from
    your setUp, and it will start the context manager for your test, and end it
    when the test is done::

        def setUp(self):
            self.v = setup_with_context_manager(self, ctxmgr(a, b, c))

        def test_foo(self):
            # do something with self.v

    """
    val = cm.__enter__()
    testcase.addCleanup(cm.__exit__, None, None, None)
    return val

I think the use is easier than yours, which needs too much super and @contextmanager boilerplate.

--Ned.

[Neil Girdhar]

Nice!

The setup_with_context_manager solution is identical as long as your context manager doesn't do any exception transformation or processing.  For example, you might have a context manager that transforms certain exceptions in testing into more verbose ones so that tests are easier to debug.

You're right that the two solutions are a matter of preference.  I personally find @contextmanager as natural as builtins like @staticmethod.

Still, it's good to know that you can use your solution in code right now.  I'm going to add your solution to stackoverflow question, since I think it's much better than overriding run.

Thanks

Neil

 

--Ned.

--

---
You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/cF_4IlJq698/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

--

---
You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/cF_4IlJq698/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gregory P. Smith]

Neil, you might also bring this up on the http://lists.idyll.org/listinfo/testing-in-python list as I suspect people there have opinions on this topic.

-gps

[Nick Coghlan]

Folks, this has come up before, but: please don't post through Google
Groups, as it breaks everyone else's ability to easily reply to the
entire mailing list.

---------- Forwarded message ----------
From: Nick Coghlan <ncog...@gmail.com>
Date: 22 August 2017 at 15:32
Subject: Re: [Python-ideas] Please consider adding context manager
versions of setUp/tearDown to unittest.TestCase
To: Neil Girdhar <miste...@gmail.com>
Cc: python-ideas <python...@googlegroups.com>

On 21 August 2017 at 11:32, Neil Girdhar <miste...@gmail.com> wrote:
> This question describes an example of the problem:
> https://stackoverflow.com/questions/8416208/in-python-is-there-a-good-idiom-for-using-context-managers-in-setup-teardown.
> You want to invoke a context manager in your setup/tearing-down, but the
> easiest way to do that is to override run, which seems ugly.

Using context managers when you can't use a with statement is one of
the main use cases for contextlib.ExitStack():

def setUp(self):
self._resource_stack = stack = contextlib.ExitStack()
self._resource = stack.enter_context(MyResource())

def tearDown(self):
self._resource_stack.close()

I posted that as an additional answer to the question:
https://stackoverflow.com/questions/8416208/in-python-is-there-a-good-idiom-for-using-context-managers-in-setup-teardown/45809502#45809502

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia


--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Nick Coghlan]

On 22 August 2017 at 15:34, Nick Coghlan <ncog...@gmail.com> wrote:
> On 21 August 2017 at 11:32, Neil Girdhar <miste...@gmail.com> wrote:
>> This question describes an example of the problem:
>> https://stackoverflow.com/questions/8416208/in-python-is-there-a-good-idiom-for-using-context-managers-in-setup-teardown.
>> You want to invoke a context manager in your setup/tearing-down, but the
>> easiest way to do that is to override run, which seems ugly.
>
> Using context managers when you can't use a with statement is one of
> the main use cases for contextlib.ExitStack():
>
> def setUp(self):
> self._resource_stack = stack = contextlib.ExitStack()
> self._resource = stack.enter_context(MyResource())
>
> def tearDown(self):
> self._resource_stack.close()
>
> I posted that as an additional answer to the question:
> https://stackoverflow.com/questions/8416208/in-python-is-there-a-good-idiom-for-using-context-managers-in-setup-teardown/45809502#45809502

Sjoerd pointed out off-list that this doesn't cover the case where
you're acquiring multiple resources and one of the later acquisitions
fails, so I added the ExitStack idiom that covers that case (using
stack.pop_all() as the last operation in a with statement):

def setUp(self):
with contextlib.ExitStack() as stack:
self._resource1 = stack.enter_context(GetResource())
self._resource2 = stack.enter_context(GetOtherResource())
# Failures before here -> immediate cleanup
self.addCleanup(stack.pop_all().close)
# Now cleanup won't happen until the cleanup functions run

I also remember that using addCleanup lets you avoid defining tearDown entirely.

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Nick Timkovich]

On Tue, Aug 22, 2017 at 12:34 AM, Nick Coghlan <ncog...@gmail.com> wrote:

Folks, this has come up before, but: please don't post through Google
Groups, as it breaks everyone else's ability to easily reply to the
entire mailing list.

Mentioning this is probably going to do nothing, especially for new, future users. Can you block python...@googlegroups.com (or if it's CC'd or whatever) from posting if you just want the Groups page to be a read-only thing?

[Chris Barker]

** Caution: cranky curmudgeonly opinionated comment ahead: **

unitest is such an ugly Java-esque static mess of an API that there's really no point in trying to clean it up and make it more pythonic -- go off and use pytest and be happier.

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[rym...@gmail.com]

TBH you're completely right. Every time I see someone using unittest andItsHorriblyUnpythonicNames, I want to kill a camel.

Sometimes, though, I feel like part of the struggle is the alternative. If you dislike unittest, but pytest is too "magical" for you, what do you use? Many Python testing tools like nose are just test *runners*, so you still need something else. In the end, many just end up back at unittest, maybe with nose on top.

As much as I hate JavaScript, their testing libraries are leagues above what Python has.

--
Ryan (ライアン)
Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else
http://refi64.com

[Nick Timkovich]

Knowing nothing about the JavaScript ecosystem (other than that leftpad is apparently not a joke and everything needs more jQuery), what are the leagues-above testing libraries?

[Steven D'Aprano]

On Tue, Aug 22, 2017 at 06:20:50PM -0400, rym...@gmail.com wrote:

> TBH you're completely right. Every time I see someone using unittest
> andItsHorriblyUnpythonicNames, I want to kill a camel.

If your only complaint about unittest is that
you_miss_writing_underscores_between_all_the_words, then unittest must
be pretty good.



--
Steve

[Chris Barker]

On Tue, Aug 22, 2017 at 5:19 PM, Chris Barker <chris....@noaa.gov> wrote:

anyway, that's enough ranting.....

Got carried away with the ranting, and didn't flesh out my point.

My point is that unittest is a very static, not very pythonic framework -- if you are productive with it, great, but I don't think it's worth trying to add more pythonic niceties to. Chances are pytest (Or nose2?) may already have them, or, if not, the simpler structure of pytest tests make them easier to write yourself.

-CHB

[Chris Barker]

Getting kind of OT, but:

...  pytest is too "magical" for you,

I do get confused a bit sometimes, but for the most part, I simple don't use the magic -- pytest does a great job of making the simple things simple.

what do you use? Many Python testing tools like nose are just test *runners*, so you still need something else.

nose did provide a number of utilities to make testing friendly, but it is apparently dead, and AFAICT, nose2, is mostly a test runner for unittest2 :-(

I converted to pytest a while back mostly inspired by it's wonderful reporting of the details of test failures.

If your only complaint about unittest is that
you_miss_writing_underscores_between_all_the_words, then unittest must
be pretty good.

For my part, I kinda liked StudlyCaps before I drank the pep8 kool-aid.

What I dislike about unitest is that it is a pile of almost completely worthless boilerplate that you have to write.

what the heck are all those assertThis methods for? I always thought they were ridiculous, but then I went in to write a new one (for math.isclose(), which was rejected, and one of these days I may add it to assertAlmostEqual ... and assertNotAlmostEqual ! ) -- low and behold, the entire purpose of the assert methods is to create a nice message when the test fails. really!

This in a dynamic language with wonderful introspection capabilities.

So that's most of the code in unitest -- completely worthless boilerplate that just makes you have to type more.

Then there is the fixture stuff -- not too bad, but still a lot klunkier than pytest fixtures.

And no parameterized testing -- that's a killer feature (that nose provided as well)

anyway, that's enough ranting.....

-CHB

[Neil Girdhar]

Like you, I used nose and then switched to pytest.  The reason I proposed this for unittest is because pytest and nose and (I think) most of the other testing frameworks inherit from unittest, so improving unittest has downstream benefits.  I may nevertheless propose this to the pytest people if this doesn't make it into unittest.

--

---
You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/cF_4IlJq698/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

--

[Nick Coghlan]

On 23 August 2017 at 08:20, rym...@gmail.com <rym...@gmail.com> wrote:
> TBH you're completely right. Every time I see someone using unittest
> andItsHorriblyUnpythonicNames, I want to kill a camel.
>
> Sometimes, though, I feel like part of the struggle is the alternative. If
> you dislike unittest, but pytest is too "magical" for you, what do you use?
> Many Python testing tools like nose are just test *runners*, so you still
> need something else. In the end, many just end up back at unittest, maybe
> with nose on top.

A snake_case helper API for unittest that I personally like is
hamcrest, since that also separates out the definition of testing
assertions from being part of a test case:
https://pypi.python.org/pypi/PyHamcrest

Introducing such a split natively into unittest is definitely
attractive, but would currently be difficult due to the way that some
features like self.maxDiff and self.subTest work.

However, PEP 550's execution contexts may provide a way to track the
test state reliably that's independent of being a method on a test
case instance, in which case it would become feasible to offer a more
procedural interface in addition to the current visibly
object-oriented one.

[Neil Girdhar]

On Wed, Aug 23, 2017 at 3:31 AM Nick Coghlan <ncog...@gmail.com> wrote:

On 23 August 2017 at 08:20, rym...@gmail.com <rym...@gmail.com> wrote:
> TBH you're completely right. Every time I see someone using unittest
> andItsHorriblyUnpythonicNames, I want to kill a camel.
>
> Sometimes, though, I feel like part of the struggle is the alternative. If
> you dislike unittest, but pytest is too "magical" for you, what do you use?
> Many Python testing tools like nose are just test *runners*, so you still
> need something else. In the end, many just end up back at unittest, maybe
> with nose on top.

A snake_case helper API for unittest that I personally like is
hamcrest, since that also separates out the definition of testing
assertions from being part of a test case:
https://pypi.python.org/pypi/PyHamcrest

Introducing such a split natively into unittest is definitely
attractive, but would currently be difficult due to the way that some
features like self.maxDiff and self.subTest work.

However, PEP 550's execution contexts may provide a way to track the
test state reliably that's independent of being a method on a test
case instance, in which case it would become feasible to offer a more
procedural interface in addition to the current visibly
object-oriented one.

If you have time, could you expand on that a little bit?

 

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Chris Barker]

On Tue, Aug 22, 2017 at 7:05 PM, Neil Girdhar <miste...@gmail.com> wrote:

Like you, I used nose and then switched to pytest.  The reason I proposed this for unittest is because pytest and nose and (I think) most of the other testing frameworks inherit from unittest,

not really -- they extend unittest -- in the sense that their test runners can be used with unittest TestCases -- but they don't depend on unitest.

so improving unittest has downstream benefits. 

only to those using unittest -- a lot of folks do use pytest or nose primarily as a test runner, so those folks would benefit.

I may nevertheless propose this to the pytest people if this doesn't make it into unittest.

Anyway, I'm just being a curmudgeon -- if folks think it would be useful and not disruptive, then why not?

-CHB

 

On Tue, Aug 22, 2017 at 8:26 PM Chris Barker <chris....@noaa.gov> wrote:

On Tue, Aug 22, 2017 at 5:19 PM, Chris Barker <chris....@noaa.gov> wrote:

anyway, that's enough ranting.....

Got carried away with the ranting, and didn't flesh out my point.

My point is that unittest is a very static, not very pythonic framework -- if you are productive with it, great, but I don't think it's worth trying to add more pythonic niceties to. Chances are pytest (Or nose2?) may already have them, or, if not, the simpler structure of pytest tests make them easier to write yourself.

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

--

---
You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/cF_4IlJq698/unsubscribe.

To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

--

---
You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/cF_4IlJq698/unsubscribe.

To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

--

[Nick Coghlan]

On 24 August 2017 at 08:20, Neil Girdhar <miste...@gmail.com> wrote:
> On Wed, Aug 23, 2017 at 3:31 AM Nick Coghlan <ncog...@gmail.com> wrote:
>> However, PEP 550's execution contexts may provide a way to track the
>> test state reliably that's independent of being a method on a test
>> case instance, in which case it would become feasible to offer a more
>> procedural interface in addition to the current visibly
>> object-oriented one.
>
> If you have time, could you expand on that a little bit?

unittest.TestCase provides a few different "config setting" type
attributes that affect how failures are reported:

- self.maxDiff (length limit for rich diffs)
- self.failureException (exception used to report errors)
- self.longMessage (whether custom messages replace or supplement the
default ones)

There are also introspection methods about the currently running test:

- self.id() (currently running test ID)
- self.shortDescription() (test description)

And some stateful utility functions:

- self.addSubTest() (tracks subtest results)
- self.addCleanup() (tracks resource cleanup requests)

At the moment, these are all passed in to test methods as a piece of
explicit context (the "self" attribute), and that's what makes it hard
to refactor unittest to support standalone top-level test functions
and standalone assertion functions: there's currently no way to
implicitly make those settings and operations available implicitly
instead.

That all changes if there's a robust way for the unittest module to
track the "active test case" that owns the currently running test
method without passing the test case reference around explicitly:

- existing assertion & helper methods can be wrapped with
independently importable snake_case functions that look for the
currently active test case and call the relevant methods on it
- new assertion functions can be added to separate modules rather than
adding yet more methods to TestCase (see
https://bugs.python.org/issue18054 for some discussion of that)
- given the above enhancements, the default test loader could usefully
gain support for top level function definitions (by wrapping them in
autogenerated test case instances)

[Neil Girdhar]

Makes sense. Thanks! 

[Robert Collins]

So (wearing my maintainer hat for unittest) - very happy to consider
proposals and patches; I'd very much like to fix some structural APIs
in unittest, but I don't have the bandwidth to do so myself at this
point. And what you're asking about is largely a structural issue
because of the interactions with test reporting, with class/module
setup.

As Ned says though, the specific asked question is best solved by
using the contextmanager protocol and manually entering and exiting:
addCleanup is ideal (literally designed for this) for managing that.
The fixtures library uses this to make use of fixtures (which are
merely enhanced context managers) trivial. We should add an adapter
there I think. If I get time I'll put this on stackexchange but:

```
import unittest

import fixtures


class ContextFixture(fixtures.Fixture):

def __init__(self, cm):
super().init()
self._cm = cm

def _setUp(self):
self.addCleanup(self._cm.__exit__, None, None, None)
self._cm.__enter__()


class Example(fixtures.TestWithFixtures):
def setUp(self):
super().setUp()
self._cm_reference_if_I_need_it =
self.useFixture(ContextFixture(MyContextManager()))

def test_fred(self):
1/0
```

should (I haven't tested it :P) do the right thing

I've written about maintainability in unittest previously [1] [2], and
those experiments have worked very well. Your post has reminded me of
the stalled work in this space. In particular avoiding inheritance for
code reuse has much better maintenance properties.

I think we learnt enough to sensibly propose it as an evolution for
core unittest though some discussion is needed: for instance, the MIME
attachment aspect weirds some folk out, though its very very very
useful in the cases where it matters, and pretty ignorable in the
cases where it doesn't.

Another related thing is getting testresources awkward bits fixed so
that it becomes a joy to use - its a much better approach than
classsetup and modulesetup if for no other reason than it is
concurrency friendly [partition and execute] whereas what was put into
unittest isn't unless you also isolate the modules and class
instances, which effectively requires processes.

Lastly the broad overarching refactor I'd like to do is twofold:
- completely separate the different users for testcase: the test
executor, report, and the test author are in the same namespace today
and its super awkward. Adding a new executor only interface at e.g.
case._executor would allow test authors to have much more freedom
about what they override and don't without worrying about interactions
with the test running framework. Moving all the reporting back up to
the executor as a thunk would decouple the reporting logic from the
internals of the test case allowing for the elimination of placeholder
objects for glue between different test systems.
- tweaking the existing pseudo-streaming contracts for the executor
to be more purely forward-flow only, aware of concurrency, and more
detailed - e.g. provide a means for tests to emit metrics like
'setting up this database took 10 seconds' and have that
discarded-or-captured-if-the-reporter-supports it would be very useful
in larger test systems. Right now everyone that does this does it in a
bespoke fashion.

re: hamcrest - love it. Thats what testtools.matchers were inspired
by. But we go a bit further I think, in useful ways.

Lastly, pytest - its beautiful, great community, some bits that I will
never see eye to eye on :). Use it and enjoy, or not - whatever works
for you :)

-Rob

1: https://rbtcollins.wordpress.com/2010/05/10/maintainable-pyunit-test-suites/
2: https://rbtcollins.wordpress.com/2010/09/18/maintainable-pyunit-test-suites-fixtures/