Testing/Fixtures/Production/TestRunner
Chris Green
I'm running into some confusion on the intended way to architect tests/
users/etc. I've been playing with the latest SVN snapshot.
I've currently got my test cases setup around:
1) Setup Custom Users, Groups,Permissions that are used for all test
cases
2) Change out my authentication middleware (I use CAS for production
but local accounts work just fine for unittesting)
3) Add testcases that aren't related to INSTALLED_APPLICATIONS
4) unittest.TestCase()s
I've been adding most data manually for testing but would like to move
towards using fixtures. Seems like a good job for django.test.TestCase
Issues I've encountered:
1) initial_data has bitten me not realizing it was synced in
production each time. I think that initial_data should only get
inserted if you have an empty new database. Perhaps things synced
each time should be called "constant_data". The fixtures
documentation in testing does say it gets synched each time but also
says its purpose is to populate the initial empty new database.
2) Setting up users/groups/users manually in the runner gets blown
away by the TestCase architecture. Works as documented. However, I'm
not sure where I should put things that should happen post-each syncdb
for testing only. Should there be a testrunner function that gets
called post each sync or should I create a single TestCase derived
from django.test.TestCase that does the "pre-setup"?
3) DateTime serialization won't work if the date is before 1901 - I
have a bug/real life data set where I get some bogus dates out of
another system. This is really a python problem.
Russell Keith-Magee
>
> 1) initial_data has bitten me not realizing it was synced in
> production each time. I think that initial_data should only get
> inserted if you have an empty new database. Perhaps things synced
> each time should be called "constant_data". The fixtures
> documentation in testing does say it gets synched each time but also
> says its purpose is to populate the initial empty new database.
There is an argument to be made that initial_data for a model should
only be loaded once - when the model is added to the database. I can't
say I've been bitten by this problem myself, but I'm happy to
entertain discussion/patches that address the issue.
> 2) Setting up users/groups/users manually in the runner gets blown
> away by the TestCase architecture. Works as documented. However, I'm
> not sure where I should put things that should happen post-each syncdb
> for testing only. Should there be a testrunner function that gets
> called post each sync or should I create a single TestCase derived
> from django.test.TestCase that does the "pre-setup"?
I'm not sure I follow. TestCase goes out of its way to avoid
interfering with setUp/tearDown methods, and in a TestCase, there is a
complete flush after each test, so any setup would need to be rerun on
a per-test basis, not a per-sync basis. Can you provide an example of
the problem?
> 3) DateTime serialization won't work if the date is before 1901 - I
> have a bug/real life data set where I get some bogus dates out of
> another system. This is really a python problem.
Again, if you can provide an example, it would help. However, if the
problem is with Python itself, there's not a great deal we can do.
Yours,
Russ Magee %-)
Chris Green
wrote:
> On 8/9/07, Chris Green <gree...@gmail.com> wrote:
> > 1) initial_data has bitten me not realizing it was synced in
> > production each time.
> There is an argument to be made that initial_data for a model should
> only be loaded once - when the model is added to the database. I can't
> say I've been bitten by this problem myself, but I'm happy to
> entertain discussion/patches that address the issue.
I'll look into creating such a patch. The test case was having some
"basic templates" to choose from but then allowing users to edit those
templates to meet their own prefs. The syncdb to add additional apps
killed the modified templates.
> > 2) Setting up users/groups/users manually in the runner gets blown
> > away by the TestCase architecture. Works as documented. However, I'm
> > not sure where I should put things that should happen post-each syncdb
> > for testing only. Should there be a testrunner function that gets
> > called post each sync or should I create a single TestCase derived
> > from django.test.TestCase that does the "pre-setup"?
>
> I'm not sure I follow. TestCase goes out of its way to avoid
> interfering with setUp/tearDown methods, and in a TestCase, there is a
> complete flush after each test, so any setup would need to be rerun on
> a per-test basis, not a per-sync basis. Can you provide an example of
> the problem?
Sure: My runner simplified
The suggestion is I need to move setup_auth() to my test methods.
That does seem to be the right way to do it so I can have different
perms across each item.
def permission_finder(s):
""" Given a string in the format app_label.codename, return the
exact permission """
parts = s.split('.')
if len(parts) != 2:
raise UnknownPermission, "Unable to parse permission string
%s"% s
m,c = parts
model_perms =
Permission.objects.filter(content_type__app_label__exact=m)
if not model_perms:
raise UnknownPermission, "No such model permissions: %s" % m
perms = model_perms.filter(codename__exact=c)
if not perms:
raise UnknownPermission, "No permissions found for %s.%s" %
(m,c)
return perms
def setup_auth():
# Helpdesk Perms
helpdesk_perms = permission_finder('helpdesk.view_logs')
# Level 2 Perms
l2_perms = (helpdesk_perms,
permission_finder('helpdesk.reset_passwords'))
l1group = Group(name='Level1 Support')
l1group.save()
l2group = Group(name='Level2 Support')
l2group.save()
for each in l2_perms:
assert(each)
for p in each:
print "adding %s to %s" % (p,g1)
g1.permissions.add(p)
u1 = User.objects.create_user('level1user', 'l...@example.com',
'password')
u1.save()
u1.groups.add(g1)
def run_tests(...):
setup_middleware() # get rid of cas for Unit Testing
setup_test_environment()
[ Build the test suite ]
old_name = settings.DATABASE_NAME
create_test_db(verbosity,autoclobber=True)
setup_auth()
unittest.TextTestRunner(verbosity=verbosity).run(suite)
destroy_test_db(old_name, verbosity)
teardown_test_environment()
>
> > 3) DateTime serialization won't work if the date is before 1901 - I
> > have a bug/real life data set where I get some bogus dates out of
> > another system. This is really a python problem.
>
> Again, if you can provide an example, it would help. However, if the
> problem is with Python itself, there's not a great deal we can do.
>
class DateModel(models.Model):
time = models.DateTime()
dm = DateModel(time=datetime.datetime(1,1,1))
Since strptime() breaks with dates before 1901, it gets serialized out
with dumpdata but fails on a loaddata()
Russell Keith-Magee
>
> On Aug 9, 7:55 am, "Russell Keith-Magee" <freakboy3...@gmail.com>
> wrote:
> > On 8/9/07, Chris Green <gree...@gmail.com> wrote:
> > > 2) Setting up users/groups/users manually in the runner gets blown
> > > away by the TestCase architecture. Works as documented. However, I'm
> > > not sure where I should put things that should happen post-each syncdb
> > > for testing only. Should there be a testrunner function that gets
> > > called post each sync or should I create a single TestCase derived
> > > from django.test.TestCase that does the "pre-setup"?
> >
> > I'm not sure I follow. TestCase goes out of its way to avoid
> > interfering with setUp/tearDown methods, and in a TestCase, there is a
> > complete flush after each test, so any setup would need to be rerun on
> > a per-test basis, not a per-sync basis. Can you provide an example of
> > the problem?
>
> Sure: My runner simplified
Ah - I didn't catch on that you were using a customized test runner.
Understood now.
As for how to fix this problem, I have two suggestions.
1) Are you sure you can't just fix this with a static fixture? I can
see that you are using python code to generate the custom permissions,
but are the resulting permissions actually dynamic? Couldn't you just
build them one, capture them, and add them as an 'test_auth' fixture
that you specify each time?
2) Alternatively, have you considered using a syncdb 'post_sync'
trigger? When Django loads an application, it looks for management.py,
and loads it if it is available. This provides an entry point to
listen for the 'post_sync' signal.
This is how Django installs its permissions to begin with. In
management.py, you install a listener on the post_sync message that is
emitted as part of the synchronization process. This signal is
re-emitted as part of a flush (which is what happens at the start of
each test). Look at django.contrib.auth.management for an example of
how to set this up.
> Since strptime() breaks with dates before 1901, it gets serialized out
> with dumpdata but fails on a loaddata()
Ok - this looks to be a python problem (although I'm seeing the issue
with strftime, not strptime). It's worth logging as a ticket so that
it isn't forgotten.
Although this is a python problem, it is a python problem born of a
desire to solve Y2K issues by supporting '04' as 1904. Since we are
only using dates on the backend and in serialization, we can impose
more rigorous datetime formatting constraints, so we might be able to
solve this problem by using an alternate strftime/strptime
implementation that requires 4 digit years. If you want to try your
hand at this, feel free.
Yours,
Russ Magee %-)
Chris Green
On Aug 11, 11:15 pm, "Russell Keith-Magee" <freakboy3...@gmail.com>
wrote:
> As for how to fix this problem, I have two suggestions.
>
> 1) Are you sure you can't just fix this with a static fixture?
Yes, that should work just fine. The reason I added them dynamically
is when I was rebuilding the models and playing around a lot, it
seemed easier to add them by name.
> 2) Alternatively, have you considered using a syncdb 'post_sync'
> trigger? When Django loads an application, it looks for management.py,
> and loads it if it is available. This provides an entry point to
> listen for the 'post_sync' signal.
That's cute. I just learned of that framework the other day in the
django master slides
> > Since strptime() breaks with dates before 1901, it gets serialized out
> > with dumpdata but fails on a loaddata()
>
> Ok - this looks to be a python problem (although I'm seeing the issue
> with strftime, not strptime). It's worth logging as a ticket so that
> it isn't forgotten.
>
> Although this is a python problem, it is a python problem born of a
> desire to solve Y2K issues by supporting '04' as 1904. Since we are
> only using dates on the backend and in serialization, we can impose
> more rigorous datetime formatting constraints, so we might be able to
> solve this problem by using an alternate strftime/strptime
> implementation that requires 4 digit years. If you want to try your
> hand at this, feel free.
I'll enter a ticket for that one; I can probably pull something
together as it bites me often enough on a system I have to
interoperate with.