Yak shaving the test framework on the way to pluggable user models (#3011)
166 views
Skip to first unread message
Russell Keith-Magee
Aug 25, 2012, 4:15:07 AM8/25/12
to Django Developers
Hi all,
So, I've been working on a Django branch [1] to implement the approach
to pluggable user models that was decided upon earlier this year [2]
[1] https://github.com/freakboy3742/django/tree/t3011
[2] https://code.djangoproject.com/wiki/ContribAuthImprovements
The user-swapping code itself is coming together well. However, I've
hit a snag during the process of writing tests that may require a
little yak shaving.
The problem is this: With pluggable auth models, you lose the
certainty over exactly which User model is present, which makes it
much harder to write some tests, especially ones that will continue to
work in an unpredictable end-user testing environment.
With regards to pluggable Users, there are 5 types of tests that can be run:
1) Contrib.auth tests that validate that the internals of
contrib.auth work as expected
2) Contrib.auth tests that validate that the internals of
contrib.auth work with a custom User model
3) Contrib.auth tests that validate that the currently specified user
model meets the requirements of the User contract
The problem is that because of the way syncdb works during testing
some of these tests are effectively mutually exclusive. The test
framework runs syncdb at the start of the test run, which sets up the
models that will be available for the duration of testing -- which
then constrains the tests that can actually be run.
This doesn't affect the Django core tests so much; Django's tests will
synchronise auth.User by default, which allows tests of type 1 to run.
It can also provide a custom User model, and use @override_settings to
swap in that model as required for tests of type 2. Tests of type 3
are effectively integration tests which will pass with *any* interface
compliant User model.
However, if I have my own project that includes contrib.auth in
INSTALLED_APPS, ./manage.py test will attempt to run *all* the tests
from contrib.auth. If I have a custom User model in play, that means
that the tests of type 1 *can't* pass, because auth.User won't be
synchronised to the database. I can't even use @override_settings to
force auth.User into use -- the opportunity for syncdb to pick up
auth.User has passed.
We *could* just mark the affected tests that require auth.User as
"skipUnless(user model == auth.User)", but that would mean some
projects would run the tests, and some wouldn't. That seems like an
odd inconsistency to me -- the tests either should be run, or they
shouldn't.
In thinking about this problem, it occurred to me that what is needed
is for us to finally solve an old problem with Django's testing -- the
fact that there is a difference between different types of tests.
There are tests in auth.User that Django's core team needs to run
before we cut a release, and there are integration tests that validate
that when contrib.auth is deployed in your own project, that it will
operate as designed. The internal tests need to run against a clean,
known environment; integration tests must run against your project's
native environment.
Thinking more broadly, there may be other categories -- "smoke tests"
for quick sanity check that a system is working; "Interaction tests"
that run live browser tests; and so on.
Python's unittest library contains the concept of Suites, which seems
to me like a reasonable analog of what I'm talking about here. What is
missing is a syntax for executing those suites, and maybe some helpers
to make it easier to build those suites in the first place.
I don't have a concrete proposal at this point (beyond the high level
idea that suites seem like the way to handle this). This is an attempt
to feel out community opinion about the problem as a whole. I know
there are efforts underway to modify Django's test discovery mechanism
(#17365), and there might be some overlap here. There are also a range
of tickets relating to controlling the test execution process (#9156,
#11593), so there has been plenty of thought put into this general
problem in the past. If anyone has any opinions or alternate
proposals, I'd like to hear them.
Yours,
Russ Magee %-)
So, I've been working on a Django branch [1] to implement the approach
to pluggable user models that was decided upon earlier this year [2]
[1] https://github.com/freakboy3742/django/tree/t3011
[2] https://code.djangoproject.com/wiki/ContribAuthImprovements
The user-swapping code itself is coming together well. However, I've
hit a snag during the process of writing tests that may require a
little yak shaving.
The problem is this: With pluggable auth models, you lose the
certainty over exactly which User model is present, which makes it
much harder to write some tests, especially ones that will continue to
work in an unpredictable end-user testing environment.
With regards to pluggable Users, there are 5 types of tests that can be run:
1) Contrib.auth tests that validate that the internals of
contrib.auth work as expected
2) Contrib.auth tests that validate that the internals of
contrib.auth work with a custom User model
3) Contrib.auth tests that validate that the currently specified user
model meets the requirements of the User contract
The problem is that because of the way syncdb works during testing
some of these tests are effectively mutually exclusive. The test
framework runs syncdb at the start of the test run, which sets up the
models that will be available for the duration of testing -- which
then constrains the tests that can actually be run.
This doesn't affect the Django core tests so much; Django's tests will
synchronise auth.User by default, which allows tests of type 1 to run.
It can also provide a custom User model, and use @override_settings to
swap in that model as required for tests of type 2. Tests of type 3
are effectively integration tests which will pass with *any* interface
compliant User model.
However, if I have my own project that includes contrib.auth in
INSTALLED_APPS, ./manage.py test will attempt to run *all* the tests
from contrib.auth. If I have a custom User model in play, that means
that the tests of type 1 *can't* pass, because auth.User won't be
synchronised to the database. I can't even use @override_settings to
force auth.User into use -- the opportunity for syncdb to pick up
auth.User has passed.
We *could* just mark the affected tests that require auth.User as
"skipUnless(user model == auth.User)", but that would mean some
projects would run the tests, and some wouldn't. That seems like an
odd inconsistency to me -- the tests either should be run, or they
shouldn't.
In thinking about this problem, it occurred to me that what is needed
is for us to finally solve an old problem with Django's testing -- the
fact that there is a difference between different types of tests.
There are tests in auth.User that Django's core team needs to run
before we cut a release, and there are integration tests that validate
that when contrib.auth is deployed in your own project, that it will
operate as designed. The internal tests need to run against a clean,
known environment; integration tests must run against your project's
native environment.
Thinking more broadly, there may be other categories -- "smoke tests"
for quick sanity check that a system is working; "Interaction tests"
that run live browser tests; and so on.
Python's unittest library contains the concept of Suites, which seems
to me like a reasonable analog of what I'm talking about here. What is
missing is a syntax for executing those suites, and maybe some helpers
to make it easier to build those suites in the first place.
I don't have a concrete proposal at this point (beyond the high level
idea that suites seem like the way to handle this). This is an attempt
to feel out community opinion about the problem as a whole. I know
there are efforts underway to modify Django's test discovery mechanism
(#17365), and there might be some overlap here. There are also a range
of tickets relating to controlling the test execution process (#9156,
#11593), so there has been plenty of thought put into this general
problem in the past. If anyone has any opinions or alternate
proposals, I'd like to hear them.
Yours,
Russ Magee %-)
Aymeric Augustin
Aug 25, 2012, 4:24:25 AM8/25/12
to django-d...@googlegroups.com
On 25 août 2012, at 10:15, Russell Keith-Magee wrote:
> We *could* just mark the affected tests that require auth.User as
> "skipUnless(user model == auth.User)", but that would mean some
> projects would run the tests, and some wouldn't. That seems like an
> odd inconsistency to me -- the tests either should be run, or they
> shouldn't.
FWIW it doesn't seem odd to me. If a project doesn't use Django's
> We *could* just mark the affected tests that require auth.User as
> "skipUnless(user model == auth.User)", but that would mean some
> projects would run the tests, and some wouldn't. That seems like an
> odd inconsistency to me -- the tests either should be run, or they
> shouldn't.
built-in User model then you don't need to test it in that project's
test suite.
Indeed, running django.contrib.* tests within a project can fail in
a variety of interesting ways. Trying to isolate them sufficiently
is an endless quest. Generally speaking, I like the idea to identify
various types of test. The only downside I can imagine is the risk
that some categories lose meaning over time, leading to situations
similar to modeltests vs. regressiontests today.
Best regards,
--
Aymeric.
Russell Keith-Magee
Aug 25, 2012, 8:32:08 PM8/25/12
to django-d...@googlegroups.com
On Sat, Aug 25, 2012 at 4:24 PM, Aymeric Augustin
<aymeric....@polytechnique.org> wrote:
> On 25 août 2012, at 10:15, Russell Keith-Magee wrote:
>
>> We *could* just mark the affected tests that require auth.User as
>> "skipUnless(user model == auth.User)", but that would mean some
>> projects would run the tests, and some wouldn't. That seems like an
>> odd inconsistency to me -- the tests either should be run, or they
>> shouldn't.
>
> FWIW it doesn't seem odd to me. If a project doesn't use Django's
> built-in User model then you don't need to test it in that project's
> test suite.
A possible miscommunication here -- I don't think it's odd that the
<aymeric....@polytechnique.org> wrote:
> On 25 août 2012, at 10:15, Russell Keith-Magee wrote:
>
>> We *could* just mark the affected tests that require auth.User as
>> "skipUnless(user model == auth.User)", but that would mean some
>> projects would run the tests, and some wouldn't. That seems like an
>> odd inconsistency to me -- the tests either should be run, or they
>> shouldn't.
>
> FWIW it doesn't seem odd to me. If a project doesn't use Django's
> built-in User model then you don't need to test it in that project's
> test suite.
tests wouldn't be run; I only think it would be odd to have those
tests report as "skipped". It feels to me like they should be either
run or not run, not reported as skipped.
> Indeed, running django.contrib.* tests within a project can fail in
> a variety of interesting ways. Trying to isolate them sufficiently
> is an endless quest. Generally speaking, I like the idea to identify
> various types of test. The only downside I can imagine is the risk
> that some categories lose meaning over time, leading to situations
> similar to modeltests vs. regressiontests today.
open and closed) that follow the pattern of "this test can't pass if X
is/isn't in settings".
I'm not so concerned about the proliferation/confusion of categories,
however. To my mind, there's only 2 types of tests that really matter.
* Internal checks of system functionality. These need to run against
a clean set of known settings. There might be multiple tests for
different configurations (e.g., run this test when a cache backend is
defined; now run it when a cache backend is *not* defined); but
ultimately this is just "run with known settings", for different
values of "known".
* Integration tests. These need to run against against the currently
active project settings.
Others may want to add other categories (like the smoke and
interaction tests that I mentioned), but we can accommodate that by
providing flexibility -- we don't have to use them ourselves.
Yours,
Russ Magee %-)
Julien Phalip
Aug 25, 2012, 8:42:00 PM8/25/12
to django-d...@googlegroups.com
Perhaps some inspiration could be found in other testing frameworks. In particular, py.test has an interesting 'marking' feature [1] and I believe that Nose also has something similar. This allows to limit the execution of tests that have been marked with a particular tag, or in other terms, to dynamically specify which sub-suite to execute.
The challenge is that everyone in the industry seems to have different definitions for what constitutes a smoke, unit, functional, integration or system test. So it's probably best to follow a flexible, non-authoritative approach as the meaning for those types of tests is so subjective. Therefore my suggestion would be to:
- add a feature to allow marking tests in a similar fashion as in py.test. Any app's maintainer would be free to mark their tests with whatever tags they want, as long as they document those tags to help the user use them adequately when creating their own test suites. The Django contrib apps tests themselves could be marked with certain tags ('smoke', 'unit' or whatever) if we can agree on what makes sense in the context of Django core.
- automatically mark tests from any app with the app's name and the TestCase name. This would allow to continue to do things like: manage.py test auth gis.GEOSTest
- for convenience, allow to explicitly exclude tests marked with certain tags (either automatic or custom): e.g. manage.py test --exclude=auth --exclude=gis.GEOSTest --exclude=sessions-custom-bleh --exclude=tastypie-custom-tag-blah
Hopefully this could be a backwards-compatible, yet flexible way of creating custom test suites that make sense in the specific context of any project.
My 2 cents :)
Brett H
Aug 26, 2012, 12:36:10 AM8/26/12
to django-d...@googlegroups.com
Once #17365 (as per https://github.com/jezdez/django-discover-runner) is implemented it will be a lot cleaner, since that splits out 1 & 2 from 3. Lets call them Django, Django api compatible, and Django project api
Three things hammered home for me that Carl's approach is the correct way to do tests:
1.) Deploying to Heroku which encourages a small virtualenv size to as small as possible for deployment. Including test code in deployment is daft.
2) The usefulness of 3rd party application test suites can be limited especially when developing against trunk or varying versions. I don't want to fork just to make their test suite pass if there's nothing wrong with the code base itself - just the tests.
3.) My project provided a custom middleware and auth backend which extended the auth functionality by adding sites to users, and authenticating against that (along with email usernames). It broke the django tests in all sorts of places.
That's when I discovered Carl's talk and was an instant convert.
If I'm developing django itself or to be compatible with the django api then I need to run the django test suite and ensure it passes with my custom app, but there are any number of ways that a project or even other applications can break the test suite so project tests should never be run against the django test suite.
ptone
Aug 29, 2012, 2:01:56 AM8/29/12
to django-d...@googlegroups.com
On Saturday, August 25, 2012 5:32:08 PM UTC-7, Russell Keith-Magee wrote:
On Sat, Aug 25, 2012 at 4:24 PM, Aymeric Augustin
<aymeric....@polytechnique.org> wrote:
> On 25 août 2012, at 10:15, Russell Keith-Magee wrote:
>
>> We *could* just mark the affected tests that require auth.User as
>> "skipUnless(user model == auth.User)", but that would mean some
>> projects would run the tests, and some wouldn't. That seems like an
>> odd inconsistency to me -- the tests either should be run, or they
>> shouldn't.
>
> FWIW it doesn't seem odd to me. If a project doesn't use Django's
> built-in User model then you don't need to test it in that project's
> test suite.
A possible miscommunication here -- I don't think it's odd that the
tests wouldn't be run; I only think it would be odd to have those
tests report as "skipped". It feels to me like they should be either
run or not run, not reported as skipped.
Isn't it just semantic nuance at that point? Seems like not a heap of distinction between "not run" vs "skipped" when a condition isn't met such that the test would be applicable.
-Preston
Russell Keith-Magee
Aug 29, 2012, 3:44:54 AM8/29/12
to django-d...@googlegroups.com
there is a different. A skipped test is something that could -- or
even *should* be run -- but can't due to missing some optional
prerequisite. In this case, we're talking about tests that can't ever
be run. To my mind, it doesn't make sense to have those tests present
but "skipped".
Yours,
Russ Magee %-)
Alex Ogier
Aug 29, 2012, 4:03:29 AM8/29/12
to django-d...@googlegroups.com
On Wed, Aug 29, 2012 at 3:44 AM, Russell Keith-Magee
<rus...@keith-magee.com> wrote:
>
> I suppose you could see it as a semantic nuance. However, to my mind,
> there is a different. A skipped test is something that could -- or
> even *should* be run -- but can't due to missing some optional
> prerequisite. In this case, we're talking about tests that can't ever
> be run. To my mind, it doesn't make sense to have those tests present
> but "skipped".
I'm not sure I see the difference between a configuration that makes a
<rus...@keith-magee.com> wrote:
>
> I suppose you could see it as a semantic nuance. However, to my mind,
> there is a different. A skipped test is something that could -- or
> even *should* be run -- but can't due to missing some optional
> prerequisite. In this case, we're talking about tests that can't ever
> be run. To my mind, it doesn't make sense to have those tests present
> but "skipped".
test unnecessary and a missing optional dependency that makes a test
unnecessary. In both cases a skipped test means roughly, "A test was
found, but due to the particulars of your environment it doesn't make
sense to run it." Django users trying to validate a django deployment
can and should ignore both kinds of skipped tests. Core developers
cutting a release should investigate both kinds of skipped tests,
because it means the test coverage wasn't 100%. Since a skipped test
signals the roughly the same thing no matter the cause, there's little
reason to differentiate them.
Best,
Alex
Reply all
Reply to author
Forward
0 new messages