Testing django-admin and manage.py
Russell Keith-Magee
I've been looking at #6017 and #5943 of late. These two tickets have
been around for a while; one of the biggest reasons that they have
taken so long to address is that they cover an area of Django where
the testing isn't automated - the behaviour of django-admin.py and
manage.py. As a result, it's very hard to validate that when a change
is made that the change doesn't break some obscure use case for these
scripts.
I've just uploaded a patch to #6017 that will hopefully address this
issue. The patch 'django-admin-tests.diff' contains a prototype test
framework for django-admin and manage.py scripts. It works by spawning
a child process using popen3, setting up the test environment using a
bash script and checking the contents of stdout an stderr as a test
condition.
At this point, I'd appreciate some feedback. In particular, I need a
sanity check that the test scripts run on other unix-based systems.
I've run this test under OS X Leopard, and all the tests pass for me.
However, I don't have particularly ready access to a Linux etc
development box, so I can't confirm if there is any platform-specific
wierdness that I need to account for.
I would also appreciate it if anyone could confirm that the same test
approach will work for Windows. I have almost no access to Windows
boxes for development purposes, so I'm completely in the dark here.
However, as far as I am aware, popen3 should be available and operate
in much the same fashion as it does under Unix, so the only
modifications required should be writing a .BAT script to set up the
environment and modifying the utility routine that starts the script.
If I'm wrong, any suggestions for an alternate approach are welcome.
Yours,
Russ Magee %-)
Malcolm Tredinnick
On Mon, 2008-07-07 at 22:00 +0800, Russell Keith-Magee wrote:
[...]
> I would also appreciate it if anyone could confirm that the same test
> approach will work for Windows. I have almost no access to Windows
> boxes for development purposes, so I'm completely in the dark here.
> However, as far as I am aware, popen3 should be available and operate
> in much the same fashion as it does under Unix, so the only
> modifications required should be writing a .BAT script to set up the
> environment and modifying the utility routine that starts the script.
> If I'm wrong, any suggestions for an alternate approach are welcome.
It'd also be great if somebody with knowledge of eggs could decode the
comment about egg problems at the end of #5943. Even having read the
docs and setuptools source a bit, I cannot figure out what "anonymous"
is doing and a problem that can't be repeated cannot be fixed.
This is related to what Russell's doing, since the patch in #5943 is
fairly reasonable, but it seems to have an edge-case that fails.
Regards,
Malcolm
Jacob Kaplan-Moss
<freakb...@gmail.com> wrote:
> I've just uploaded a patch to #6017 that will hopefully address this
> issue. The patch 'django-admin-tests.diff' contains a prototype test
> framework for django-admin and manage.py scripts. It works by spawning
> a child process using popen3, setting up the test environment using a
> bash script and checking the contents of stdout an stderr as a test
> condition.
This looks like a great idea!
> At this point, I'd appreciate some feedback. In particular, I need a
> sanity check that the test scripts run on other unix-based systems.
> I've run this test under OS X Leopard, and all the tests pass for me.
> However, I don't have particularly ready access to a Linux etc
> development box, so I can't confirm if there is any platform-specific
> wierdness that I need to account for.
One thing to watch for: I'm nearly 100% sure this won't work under the
buildbot -- it looks like since the settings file is written in the
same directory as the code you'll get conflicts between simultaneously
running tests. I think you'll need to use a temp dir.
That said, I don't really know anything amount about how the buildbot
sets up its environment, so it might actually work... And frankly the
buildbot isn't in great health anyway, so I wouldn't be outraged if
you broke it -- it might prompt me or someone to actually start
maintaining it again.
> I would also appreciate it if anyone could confirm that the same test
> approach will work for Windows. I have almost no access to Windows
> boxes for development purposes, so I'm completely in the dark here.
One thing this brings up: we really need a way to skip certain tests
under certain setups: if for some reason this can't be made to work on
windows, the tests are still very valuable run elsewhere. Also think
of stuff like the serializer tests and MySQL. Don't mean to derail,
but tests like this are awesome even if they can't be properly run
everywhere.
Jacob
Simon Willison
wrote:
> issue. The patch 'django-admin-tests.diff' contains a prototype test
> framework for django-admin and manage.py scripts. It works by spawning
> a child process using popen3, setting up the test environment using a
> bash script and checking the contents of stdout an stderr as a test
> condition.
variables. Is there any reason you can't just directly assign to
os.environ from within Python code to achieve this? Eliminating the
bash script would make it much easier to achieve Windows compatibility.
Karen Tracey
Hi all,
I've been looking at #6017 and #5943 of late. These two tickets have
been around for a while; one of the biggest reasons that they have
taken so long to address is that they cover an area of Django where
the testing isn't automated - the behaviour of django-admin.py and
manage.py. As a result, it's very hard to validate that when a change
is made that the change doesn't break some obscure use case for these
scripts.
I've just uploaded a patch to #6017 that will hopefully address this
issue. The patch 'django-admin-tests.diff' contains a prototype test
framework for django-admin and manage.py scripts. It works by spawning
a child process using popen3, setting up the test environment using a
bash script and checking the contents of stdout an stderr as a test
condition.
At this point, I'd appreciate some feedback. In particular, I need a
sanity check that the test scripts run on other unix-based systems.
I've run this test under OS X Leopard, and all the tests pass for me.
However, I don't have particularly ready access to a Linux etc
development box, so I can't confirm if there is any platform-specific
wierdness that I need to account for.
I get 17 failures running on Ubuntu (gutsy), but maybe I'm missing something else needed for the tests to run properly? I just added the django-admin-tests.diff to a pristine trunk checkout. I had to create an __init__.py file in the admin-scripts directory and set the execute bit on the .sh files. Here's the output:
http://dpaste.com/61132/
Of 77 tests that run only 17 report failures so some things are working but it isn't immediately obvious to me what's going wrong with the rest.
I would also appreciate it if anyone could confirm that the same test
approach will work for Windows. I have almost no access to Windows
boxes for development purposes, so I'm completely in the dark here.
However, as far as I am aware, popen3 should be available and operate
in much the same fashion as it does under Unix, so the only
modifications required should be writing a .BAT script to set up the
environment and modifying the utility routine that starts the script.
If I'm wrong, any suggestions for an alternate approach are welcome.
Simon's idea of eliminating the bash scripts entirely sounds good to me, assuming it can be done. I do have Windows boxes and could probably cobble up some .bat files if really necessary, though I haven't done .bat programming since DOS days (and not much then). So, if it's not possible to eliminate the platform-dependent scripts I can try to replicate what you've got in the shell scripts on Windows, but I'd prefer the approach of eliminating them. In any event I can do testing on Windows.
Karen
Leo Soto M.
+1
That also would help to be compatible with other Python VMs, which
don't recognize the PYTHONPATH environment variable (IronPython uses
IRONPYTHONPATH; Jython uses a Java VM property)
--
Leo Soto M.
http://blog.leosoto.com
Russell Keith-Magee
>
> One thing to watch for: I'm nearly 100% sure this won't work under the
> buildbot -- it looks like since the settings file is written in the
> same directory as the code you'll get conflicts between simultaneously
> running tests. I think you'll need to use a temp dir.
Hrm. If this is the case, it would be a right pain. Can anyone with
more buildbot experience confirm this?
> One thing this brings up: we really need a way to skip certain tests
> under certain setups:
We already do this in a few places; for example, the fixture modeltest
disables some tests if you are using MySQL. This uses
settings.DATABASE_ENGINE, but it's not big drama to use os.name
instead.
Russ %-)
Russell Keith-Magee
I originally avoided this because of a CLASSPATH mangling hangover
resulting from my dim dark days as a Java programmer.
However, I've just tested your suggestion, and it appears to work just
fine. I've updated the patch on #6017 to use a scriptless approach; I
suppose this means that the tests should now work under Windows
(although, again, I haven't got the facilities to validate this).
Russ %-)
Russell Keith-Magee
> On Mon, Jul 7, 2008 at 10:00 AM, Russell Keith-Magee
> else needed for the tests to run properly? I just added the
> django-admin-tests.diff to a pristine trunk checkout. I had to create an
> __init__.py file in the admin-scripts directory and set the execute bit on
> the .sh files. Here's the output:
>
> http://dpaste.com/61132/
>
> Of 77 tests that run only 17 report failures so some things are working but
> it isn't immediately obvious to me what's going wrong with the rest.
To the best of my knowledge, there shouldn't be anything else required
to run the tests, other than python in your system path.
One possible cause is the __init__.py files; there are 3 required in
total - admin_scripts, admin_scripts/management, and
admin_scripts/management/commands. The patch contains entries for
those files, but they're empty, so there are no patch instructions.
Either way, I've made some modifications to the tests so that they are
a little bit more meaningful on error - in particular, I've added
validation that no errors are received if none are expected. If the
problem isn't the __init__.py files, the extra assertions should make
it easier to track down the problem.
Could I impose upon you to take the new patch for a spin?
> Simon's idea of eliminating the bash scripts entirely sounds good to me,
> assuming it can be done.
Can be, and has. Updated patch available on #6107. I believe this
means that the new scripts should run under Windows; If you could take
this version for a spin under Windows too, I'd be much obliged.
Yours,
Russ Magee %-)
Karen Tracey
To the best of my knowledge, there shouldn't be anything else required
to run the tests, other than python in your system path.
One possible cause is the __init__.py files; there are 3 required in
total - admin_scripts, admin_scripts/management, and
admin_scripts/management/commands. The patch contains entries for
those files, but they're empty, so there are no patch instructions.
That was it. Missing the first one generated an error that screamed "missing __init__.py file" but the others not so much. So I only added the first one, not noticing the others were needed as well. With all of them all 77 tests pass on Ubuntu (both new and old patch).
> Simon's idea of eliminating the bash scripts entirely sounds good to me,
> assuming it can be done.Can be, and has. Updated patch available on #6107. I believe this
means that the new scripts should run under Windows; If you could take
this version for a spin under Windows too, I'd be much obliged.
Yes, the tests run on Windows now as well. Only problem is two tests with file paths in their output, the forward slashes are backslashes on Windows so the output doesn't match what the tests are looking for:
http://dpaste.com/61364/
Not sure if/how you ordinarily get around that in testing?
Cheers,
Karen
mrts
> file paths in their output, the forward slashes are backslashes on Windows
> so the output doesn't match what the tests are looking for:
>
> http://dpaste.com/61364/
>
> Not sure if/how you ordinarily get around that in testing?
'models.pyc'), ...)