Changing settings per test
David Cramer
object for certain test cases. The other case is that in many cases we
were (are?) relying on settings being configured a certain way for the
Django tests to even work. I brought this up in #django-dev a while
back, but wanted to open it up to further discussion.
Let me start with an example test:
def test_with_awesome_setting(self):
_orig = getattr(settings, 'AWESOME', None)
settings.AWESOME = True
# do my test
...
settings.AWESOME = _orig
So the obvious problem for me here is that I'm repeating this same
code flow in a lot of situations. Ignoring the fact that it's ugly,
it's just not fun mangling with settings like this (at least, not fun
having to reset the values).
My proposal is to include some kind of utility within the test suite
which would make this easier. There's a couple ways I could see this
working:
1. The settings object could be copied and reset on each case.
2. The settings object could be replaced with a Proxy which stores a
copy of any value changed since reset, and returns that value if
present. It could then simply just be reset (clear the proxy's dict)
on each setUp.
Anyways, I'd love to hear how others have dealt with this and any
other possible solutions.
Alex Gaynor
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>
Russ and I discussed having a decorator/context manager for monkey
patching settings:
@swap_settings(TEMPLATE_DIRS=[fuzzy], DEBUG=True)
def test_who_cares(self):
self.assertTrue(settings.DEBUG)
or
with swap_settings(DEBUG=True):
self.assertTrue(settings.DEBUG)
Alex
--
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me
Dan Fairs
> Let me start with an example test:
>
> def test_with_awesome_setting(self):
> _orig = getattr(settings, 'AWESOME', None)
> settings.AWESOME = True
>
> # do my test
> ...
>
> settings.AWESOME = _orig
>
Pedant: there's a small bug above which has bitten me before doing a similar thing - settings.AWESOME ends up set to None after the test has run if it didn't exist before.
> Anyways, I'd love to hear how others have dealt with this and any
> other possible solutions.
I've used Michael Foord's Mock library to patch a setting for the duration of a test case. Chris Withers' testfixtures library also has some sugar to provide a context manager approach, though I haven't used that in a little while.
Cheers,
Dan
--
Dan Fairs | dan....@gmail.com | www.fezconsulting.com
David Cramer
(This is dry code)
def with_settings(**overrides):
"""Allows you to define settings that are required for this
function to work"""
NotDefined = object()
def wrapped(func):
@wraps(func)
def _with_settings(*args, **kwargs):
_orig = {}
for k, v in overrides.iteritems():
_orig[k] = getattr(settings, k, NotDefined)
try:
func(*args, **kwargs)
finally:
for k, v in _orig.iteritems():
if v is NotDefined:
delattr(settings, k)
else:
setattr(settings, k, v)
return _with_settings
return wrapped
I'm not familiar with the context managers, but I imagine those would
solve things like adjusting CONTEXT_PROCESSORS.
Łukasz Rekucki
my application's tests suite :).
Currently, I'm using a pair of save/restore functions: save() monkey
patches the settings module and returns a dictionary of old values,
restore() puts back the old values based on the dictionary. I usually
put this in setUp/tearDown so I don't have to repeat in every test. I
was about to propose that
Django's TestCase should do something similar by default.
Both the decorator and context processor are very useful, but having
something to set values for the whole test case instead of a single
test or a block of code would be great too. I was thinking about
something in line of:
class EmailTestCase(TestCase):
settings = dict(DEFAULT_FROM_EMAIL="webm...@example.com")
--
Łukasz Rekucki
Alex Gaynor
Well, there's no reason the decorator couldn't be used as a class
decorator (on 2.6 and above). I'll admit that the settings attribute
on TestCase is more consistant with how we've handled other things
(urls, fixtures), however, for whatever reason I'm not a fan, as it
forces you to split up tests that should logically be grouped on a
single class.
David Cramer
views rather than class attributes. I'm not sure of the performance
implications of many classes vs decorating functions on a large class
instead, but it just seems to make more sense in some cases.
Here's a working (we're now using it) version of the previous decorator:
def with_settings(**overrides):
"""Allows you to define settings that are required for this
function to work"""
NotDefined = object()
def wrapped(func):
@wraps(func)
def _with_settings(*args, **kwargs):
_orig = {}
for k, v in overrides.iteritems():
_orig[k] = getattr(settings, k, NotDefined)
setattr(settings, k, v)
try:
func(*args, **kwargs)
finally:
for k, v in _orig.iteritems():
if v is NotDefined:
delattr(settings, k)
else:
setattr(settings, k, v)
return _with_settings
return wrapped
--
David Cramer
http://www.davidcramer.net
Sean Brant
On Nov 4, 4:26 pm, Alex Gaynor <alex.gay...@gmail.com> wrote:
> 2010/11/4 Łukasz Rekucki <lreku...@gmail.com>:
>
>
>
>
> > Funny, I had exactly the same problem today at work while refactoring
> > my application's tests suite :).
>
> > Currently, I'm using a pair of save/restore functions: save() monkey
> > patches the settings module and returns a dictionary of old values,
> > restore() puts back the old values based on the dictionary. I usually
> > put this in setUp/tearDown so I don't have to repeat in every test. I
> > was about to propose that
> > Django's TestCase should do something similar by default.
>
> > Both the decorator and context processor are very useful, but having
> > something to set values for the whole test case instead of a single
> > test or a block of code would be great too. I was thinking about
> > something in line of:
>
> > class EmailTestCase(TestCase):
>
> >>>> Let me start with an example test:
>
> >>>> def test_with_awesome_setting(self):
> >>>> _orig = getattr(settings, 'AWESOME', None)
> >>>> settings.AWESOME = True
>
> >>>> # do my test
> >>>> ...
>
> >>>> settings.AWESOME = _orig
>
> >>> Pedant: there's a small bug above which has bitten me before doing a similar thing - settings.AWESOME ends up set to None after the test has run if it didn't exist before.
>
> >>>> Anyways, I'd love to hear how others have dealt with this and any
> >>>> other possible solutions.
>
> >>> I've used Michael Foord's Mock library to patch a setting for the duration of a test case. Chris Withers' testfixtures library also has some sugar to provide a context manager approach, though I haven't used that in a little while.
>
> >>> Cheers,
> >>> Dan
>
> >>> --
> >>> --
> >>> You received this message because you are subscribed to the Google Groups "Django developers" group.
> >>> To post to this group, send email to django-d...@googlegroups.com.
> >>> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> >> --
> >> You received this message because you are subscribed to the Google Groups "Django developers" group.
> >> To post to this group, send email to django-d...@googlegroups.com.
> >> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> > --
> > Łukasz Rekucki
>
> > --
> > You received this message because you are subscribed to the Google Groups "Django developers" group.
> > To post to this group, send email to django-d...@googlegroups.com.
> > To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> Well, there's no reason the decorator couldn't be used as a class
> decorator (on 2.6 and above). I'll admit that the settings attribute
> on TestCase is more consistant with how we've handled other things
> (urls, fixtures), however, for whatever reason I'm not a fan, as it
> forces you to split up tests that should logically be grouped on a
> single class.
>
> Alex
>
> --
> "I disapprove of what you say, but I will defend to the death your
> right to say it." -- Voltaire
> "The people's good is the highest law." -- Cicero
> "Code can always be simpler than you think, but never as simple as you
> want" -- Me
current settings with ability to overwrite values in your tests then
self.settings could be reset on each test run.
Mikhail Korobov
some of them are cached and the change of the value doesn't change
django behaviour (e.g. TEMPLATE_CONTEXT_PROCESSORS option behaves like
this).
On 5 ноя, 02:23, Łukasz Rekucki <lreku...@gmail.com> wrote:
> Funny, I had exactly the same problem today at work while refactoring
> my application's tests suite :).
>
> Currently, I'm using a pair of save/restore functions: save() monkey
> patches the settings module and returns a dictionary of old values,
> restore() puts back the old values based on the dictionary. I usually
> put this in setUp/tearDown so I don't have to repeat in every test. I
> was about to propose that
> Django's TestCase should do something similar by default.
>
> Both the decorator and context processor are very useful, but having
> something to set values for the whole test case instead of a single
> test or a block of code would be great too. I was thinking about
> something in line of:
>
> class EmailTestCase(TestCase):
>
> >>> Let me start with an example test:
>
> >>> def test_with_awesome_setting(self):
> >>> _orig = getattr(settings, 'AWESOME', None)
> >>> settings.AWESOME = True
>
> >>> # do my test
> >>> ...
>
> >>> settings.AWESOME = _orig
>
> >> Pedant: there's a small bug above which has bitten me before doing a similar thing - settings.AWESOME ends up set to None after the test has run if it didn't exist before.
>
> >>> Anyways, I'd love to hear how others have dealt with this and any
> >>> other possible solutions.
>
> >> I've used Michael Foord's Mock library to patch a setting for the duration of a test case. Chris Withers' testfixtures library also has some sugar to provide a context manager approach, though I haven't used that in a little while.
>
> >> Cheers,
> >> Dan
>
> >> --
> >> --
> >> You received this message because you are subscribed to the Google Groups "Django developers" group.
> >> To post to this group, send email to django-d...@googlegroups.com.
> >> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> > --
> > You received this message because you are subscribed to the Google Groups "Django developers" group.
> > To post to this group, send email to django-d...@googlegroups.com.
> > To unsubscribe from this group, send email to django-develop...@googlegroups.com.
Łukasz Rekucki
> Agree with Alex. We're considering moving more towards decorating
> views rather than class attributes. I'm not sure of the performance
> implications of many classes vs decorating functions on a large class
> instead, but it just seems to make more sense in some cases.
What exactly do you mean by "decorating views" in this context ?
> On Thu, Nov 4, 2010 at 2:26 PM, Alex Gaynor <alex....@gmail.com> wrote:
>>
>> Well, there's no reason the decorator couldn't be used as a class
>> decorator (on 2.6 and above). I'll admit that the settings attribute
>> on TestCase is more consistant with how we've handled other things
>> (urls, fixtures), however, for whatever reason I'm not a fan, as it
>> forces you to split up tests that should logically be grouped on a
>> single class.
You can alter the setting both with a class attribute/decorator and
then alter it some more for a particular test. For example (excuse the
horrible naming):
@alter_class_settings(USE_L10N=True, OTHER_SETTING=False):
class MyTest(TestCase):
# 20 tests that rely on L10N
@alter_test_settings(USE_L10N=False)
def test_no_localization(self): pass
I meant the class-level decorator/attribute as a shortcut for a common
case. I agree that splitting tests just because of settings is bad,
but repeating the same decorator all over the test case is bad too,
imho.
Here[1] is an implementation of a context manager and a class
decorator based on the function decorator that David provided. Does it
look reasonable ?
[1]: https://gist.github.com/663367
--
Łukasz Rekucki
Russell Keith-Magee
Love the idea in general. Django's own test suite is full of this
pattern (or buggy partial implementations of it), which is a prime
indication that we should providing something at the framework level
to make this easy to do.
The devil is in the detail. If you look at the ways Django uses this
pattern, there are lots of edge cases that need to be handled. For
example:
* Some settings are lists, and the test case needs to
append/prepend/insert into that existing list, rather than overwriting
a list. An example of this is adding an extra context processor for
test purposes. Yes, this could be handled by manually specifying the
full list, but it's a fairly common pattern, so it would be nice to
have a quick way to represent the pattern.
* Settings that are internally cached. For example, anything that
modifies INSTALLED_APPS.
* Settings that need to make call to reset state affected by loading
new new group of settings. For example, if you change TZ, you need to
call "time.tzset()" on teardown to reset the timezone. Similarly for
deactivating translations if you change USE_I18N.
* Settings that need to be removed, rather that set to None. Again,
TZ is an example here -- there is a difference between "TZ exists and
is None" and "TZ doesn't exist".
I've probably missed a couple of other edge cases; it would be worth
doing an audit of Django's test suite to see all the places that we've
used this pattern, and the workarounds we've had to use to clean up
after it.
There's also the matter of making this system easy to use in the
practical sense. The context manager approach that have been given so
far in this thread is a nice syntactic fits, but miss one big use case
-- modifying settings for *every* test in a TestCase. This is the way
that settings changes are used right now in Django's test suite. The
provided decorator has the same limitation, but it shouldn't be too
hard to modify it to be a class-or-function decorator.
Lastly, a persistent source of bugs in Django's own test suite is
having complete settings isolation. If you're doing true unit tests,
it's not enough to just replace one or two settings -- you have to
clear the decks to make sure that the user's settings file doesn't
provide an environment that breaks your test. This manifests itself as
tests for contrib apps that pass fine in Django's own suite, but fail
when an end user deployes the app -- for example, you deploy
contrib.auth in an environment that doesn't have the login URLs
deployed, and your tests fail; or you deploy contrib.flatpages but
don't deploy contrib.sites, and the tests fail.
Over time, we've cleaned up these issues as we've found them, but the
real fix is to make sure a test runs in a completely clean settings
environment. That is, we reset settings to a baseline
(global_settings.py would be a an obvious candidate) rather than just
tinkering with the one or two settings that we think are important.
The "Clear the decks" approach is a different use case to "just change
these three settings", but it's closely related, and worth tackling at
the same time, IMHO.
So - tl;dr. love the idea. However, we need to handle the edge cases
if it's going to be added to Django trunk, and replacing Django's own
usage of the ad-hoc pattern is as good a test as any that we've
tackled the edge cases.
Yours,
Russ Magee %-)
Alex Gaynor
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>
I think it's a little cavalier to see what we do is ad-hoc. Sure
there are a bunch of tests with:
def setUp(self):
self.old_SETTING = getattr(settings, "SETING", _missing)
def tearDown(self):
if self.old_SETTING is _missing:
del settings.SETTING"
else:
settings.SETTING = self.old_SETTING
but how else would you write that? That's the whole point of setUp
and tearDown, and I can't think of a more succinct formulation of that
that covers all of the cases (assign/reset attribute, append/insert,
set env variable, etc.).
Russell Keith-Magee
> On Thu, Nov 4, 2010 at 8:14 PM, Russell Keith-Magee
> <rus...@keith-magee.com> wrote:
>> So - tl;dr. love the idea. However, we need to handle the edge cases
>> if it's going to be added to Django trunk, and replacing Django's own
>> usage of the ad-hoc pattern is as good a test as any that we've
>> tackled the edge cases.
>
> there are a bunch of tests with:
>
> def setUp(self):
> self.old_SETTING = getattr(settings, "SETING", _missing)
>
> def tearDown(self):
> if self.old_SETTING is _missing:
> del settings.SETTING"
> else:
> settings.SETTING = self.old_SETTING
>
> but how else would you write that? That's the whole point of setUp
> and tearDown, and I can't think of a more succinct formulation of that
> that covers all of the cases (assign/reset attribute, append/insert,
> set env variable, etc.).
Ad hoc is from the Latin "For this". Ad-hoc doens't mean the pattern
is bad or wrong, it just means it isn't generalized. We manually
reproduce variations of the pattern you describe every time we need
it, rather than having a generalized framework level tool for handling
settings. I think "ad hoc" is a pretty apt description of what
Django's test suite does.
Yours,
Russ Magee %-)
akaariai
> def setUp(self):
> self.old_SETTING = getattr(settings, "SETING", _missing)
>
> def tearDown(self):
> if self.old_SETTING is _missing:
> del settings.SETTING"
> else:
> settings.SETTING = self.old_SETTING
change_setting(name, new_value)
which returns old setting or a marker when there is nothing configured
for that value. This function would clear the caches if the setting is
cached somewhere, and also handle the env updates for timezone or
other settings needing that. Clearing caches is a problem that is not
handled well at all in the Django test suite.
You could then revert the setting using the same function. This would
again handle clearing the caches & env handling. If passed in
new_value was the marker for nothing, then the setting would be
deleted.
Using those functions setting changes would be done like this:
def setUp(self):
self.old_SETTING = settings.change_setting("SETTING", new_val)
# or if you want just to store the setting and change it later in
the actual tests
# self.old_SETTING = settings.change_setting("SETTING")
# will just fetch the old setting, or the marker for missing
setting
def tearDown(self):
settings.change_setting("SETTING", self.old_SETTING)
And you would not need to care if the setting was cached somewhere,
the change_setting function will take care of that. I don't know if it
would be good to have also settings.load_defaults (returns a dict
containing all the old settings, loads global_settings). This would
need a reverse function, I can't think of a good name for it,
settings.revert_load_defaults(old_settings_dict) or something...
The append case could have still another function,
append_setting(name, value), returning old list (or marker if nothing)
and inserting the new value in the list. Reverting would be just
change_setting(name, append_setting_ret_val).
Handling what needs to be done when changing a setting could be signal
based (register_setting_change_listener), this would allow using the
same mechanism for settings used by apps not in core.
Of course, there could also be decorators which would use these
functions...
- Anssi
Santiago Perez
* Settings that are internally cached. For example, anything that
modifies INSTALLED_APPS.
* Settings that need to make call to reset state affected by loading
new new group of settings. For example, if you change TZ, you need to
call "time.tzset()" on teardown to reset the timezone. Similarly for
deactivating translations if you change USE_I18N.
* Settings that need to be removed, rather that set to None. Again,
TZ is an example here -- there is a difference between "TZ exists and
is None" and "TZ doesn't exist".
David Cramer
ideal candidate to solve that problem.