GSoC 2009: Testing Upgrades/Awesomeness
Kevin Kubasik
ideas, I want them in Django, so I have teh following proposal, lemme
know what you think. I would love feedback of all flavors, both on the
application itself and what its proposing.
It's below in restructuredText:
Upgrade the Awesomness Quotient of the Django Test Utils and
Regression Suite
=============================================================================
Abstract
********
To fix and test, so cover the Admin site/
with Windmill use, avoiding the great fright.
At moment be, we cover not, popular/
is the admin but tested is not.
Great addition tests would be/
the key in promoting greater Django harmony.
The Problem
***********
Django has a fantastic set of regression tests which cover much of the
codebase, but the famous Admin interface isn't covered by any sort of
automated tests. In addition, the tools for testing a site under the
Django framework are weak, even if the API's available are quite
powerful.
Proposal
--------
Overall Django has some great testing tools, but the addition of a few
key integrations and tools could make it a much more comprehensive and
easy to use testsuite. Namely, nose runners and extensions, Windmill
integration, and some expansions to Eric's django-test-utils package.
Namely, I want to accomplish the following:
1. Work with Windmill and Nose to make sure running django tests
with Nose and running Windmill tests through django is easy and
painless. This will mostly involve work on the external projects, and
probably won't touch Django proper. But in coordinating with these
projects, I also hope to provide any missing or needed extension
points in Django to make this integration possible.
2. Expand the Django test runner to provide Code Coverage
statistics. There are already some efforts towards this goal underway,
and I don't plan on reinventing the wheel. My hope is to integrate
their work, and potentially expand it based on Django's community
requests. Most likely the base of this work will be figleaf, but I
plan on researching other options. Lastly, I want to evaluate the
coverage of Django's current regression suite and provide a wiki page/
writeup detailing any major deficiencies so that they can be
addressed.
3. Utilize the new Windmill test support to provide coverage of the
Admin interface. This is an extremely large task overall, but given
how easy Windmill has made the creation of tests, I hope that I will
be able to provide coverage of a majority of the admin's
functionality. This will not include support for the
django.contrib.gis namespace, but I will try and test all other
namespaces that have Admin integration or functionality (namely auth,
admin and comments).
4. The last major component of this project will be the extension
and expansion of the django-test-utils package to add the following
components:
1. Update the TestGen Middleware to serialize all testing
data (probably just via cPickle initially) to disk.
2. Implement a pluggable 'Processor' framework to generate
the runnable tests from the serialized request data.(Aka twill/django
unittest/unittest)
3. A basic web interface to start/stop the TestingMiddleware
and display its status/results.
4. Utilizing the new Plugin framework allow for test-time
analysis such as unused Context variables and dead links.
5. Integrate something like Werkzeug_ on test fails from web
interface runner.
Timeline
--------
Weekly I should be able to put in between 20-30 hours a week, but that
will be focused on the weekends. During the week I will have classes,
but will be available during many of them for discussion, but will
avoid coding. (I have 2 lab classes that require me to be in a room
for 4 hours, but most of the time is watching something settle or spin
or do nothing at all). A breakdown of my projected timing:
Week 1: Gain familiarity with Windmills capabilities and limitations,
start integration of existing work on Nose and Windmill support.
Week 2-3: Get code coverage working for both Django Regression Suite
and under a manage command.
Week 4: Write first Windmill test for admin. Determine the
'environment' or set of Models we will use to test the Admin, confirm
with community that I am testing *every* major feature available in
the Admin.
Week 5-10: Test mania. Windmill test writing fury. Breakdown goes:
- Model list page and single object add (simple forms)
- Changelist (filters/drilldown/search etc.)(no actions yet)
- single object delete + complex forms (datetime, image, slug etc.)
- inline objects
- actions and history
- misc features and remaining niches
Week 10-12: Discuss missing tests or other cases to check. Possibly
provide branch of tests for new admin work (if that project is
accepted for GSOC). Add cool fun features and integration to django-
test-utils.
About you
---------
I'm a long-time open source junkie in my last few quarters at Neumont
University. I've been writing Python the past 2 years or so, and have
been a Django user for the past year.
.. _Werkzeug: http://code.google.com/p/django-command-extensions/wiki/RunServerPlus
Russell Keith-Magee
Hi Kevin!
> The Problem
> ***********
>
> Django has a fantastic set of regression tests which cover much of the
> codebase, but the famous Admin interface isn't covered by any sort of
> automated tests.
Not entirely true - the admin does have _some_ tests, in the
regression tests module. However, I agree that these tests could be
substantially improved.
> In addition, the tools for testing a site under the
> Django framework are weak, even if the API's available are quite
> powerful.
Again - depends what you mean by 'testing a site' - if you mean
functional testing at the UI level, then I will concur wholeheartedly.
> Proposal
> --------
>
> Overall Django has some great testing tools, but the addition of a few
> key integrations and tools could make it a much more comprehensive and
> easy to use testsuite. Namely, nose runners and extensions, Windmill
> integration, and some expansions to Eric's django-test-utils package.
> Namely, I want to accomplish the following:
>
> 1. Work with Windmill and Nose to make sure running django tests
> with Nose and running Windmill tests through django is easy and
> painless. This will mostly involve work on the external projects, and
> probably won't touch Django proper. But in coordinating with these
> projects, I also hope to provide any missing or needed extension
> points in Django to make this integration possible.
I like the sentiment and the goal - my only concern is the extent to
which this is in scope for a Django GSoC project. If making this
integration requires changes on the Django side, you're fine - but if
you need to make changes on the Nose/Windmill side, then you have a
problem, as we don't have any control over those projects. Also - I
was under the impression that both Nose and Windmill already had
Django interfaces of some description. I could be mistaken - I haven't
really used either of them extensively myself.
> 2. Expand the Django test runner to provide Code Coverage
> statistics. There are already some efforts towards this goal underway,
> and I don't plan on reinventing the wheel. My hope is to integrate
> their work, and potentially expand it based on Django's community
> requests. Most likely the base of this work will be figleaf, but I
> plan on researching other options. Lastly, I want to evaluate the
> coverage of Django's current regression suite and provide a wiki page/
> writeup detailing any major deficiencies so that they can be
> addressed.
+1. There are a few coverage options out there, and there is already
an accepted ticket for this feature.
> 3. Utilize the new Windmill test support to provide coverage of the
> Admin interface. This is an extremely large task overall, but given
> how easy Windmill has made the creation of tests, I hope that I will
> be able to provide coverage of a majority of the admin's
> functionality. This will not include support for the
> django.contrib.gis namespace, but I will try and test all other
> namespaces that have Admin integration or functionality (namely auth,
> admin and comments).
Requiring Windmill here makes me a little nervous. I have no
experience with Windmill to know if it is a good choice for this task.
I don't want to just add Windmill tests because we can - we need to be
adding tests that actually add value for regression purposes. It's
very easy to write functional tests that don't actually validate
functionality - they just make it difficult to modify code.
We also need to ensure that the test suite continues to run for those
that haven't got Windmill installed (albeit with a warning). There is
an old ticket that proposed to allow skipping tests that are known
failures, reporting them as 'known failures/skipped tests' rather than
outright failures. This might be something worth including in your
proposal.
> 4. The last major component of this project will be the extension
> and expansion of the django-test-utils package to add the following
> components:
> 1. Update the TestGen Middleware to serialize all testing
> data (probably just via cPickle initially) to disk.
> 2. Implement a pluggable 'Processor' framework to generate
> the runnable tests from the serialized request data.(Aka twill/django
> unittest/unittest)
> 3. A basic web interface to start/stop the TestingMiddleware
> and display its status/results.
> 4. Utilizing the new Plugin framework allow for test-time
> analysis such as unused Context variables and dead links.
> 5. Integrate something like Werkzeug_ on test fails from web
> interface runner.
django-test-utils is a great set of tools, but working on those tools
is (IMHO) out of scope for the SoC, since it isn't part of the Django
project itself.
If you're looking for something to pad out your proposal, there are
plenty of testing tickets on the Django Trac that have been accepted,
and would be fantastic enhancements to Django. Take your pick :-)
> Timeline
> --------
>
> Weekly I should be able to put in between 20-30 hours a week, but that
> will be focused on the weekends.
I was under the impression that GSoC was intended to be a full-time
activity - however, I couldn't find any reference to this in the FAQ.
You may want to confirm that a 'part time' application is allowed.
Yours,
Russ Magee %-)
Jacob Kaplan-Moss
<freakb...@gmail.com> wrote:
> I like the sentiment and the goal - my only concern is the extent to
> which this is in scope for a Django GSoC project. If making this
> integration requires changes on the Django side, you're fine - but if
> you need to make changes on the Nose/Windmill side, then you have a
> problem, as we don't have any control over those projects. Also - I
> was under the impression that both Nose and Windmill already had
> Django interfaces of some description. I could be mistaken - I haven't
> really used either of them extensively myself.
I'm not sure about Nose either, but Windmill has Django support, and
actually grew a management comment as part of the PyCon sprints.
You'll add 'windmill' to INSTALLED_APPS, and then run `manage.py
test_windmill` (or similar); see
http://trac.getwindmill.com/changeset/1172.
So we don't have to make a single change to Django to make that
support work, which makes me very happy :)
> Requiring Windmill here makes me a little nervous. I have no
> experience with Windmill to know if it is a good choice for this task.
AFAIK the two choices for doing browser tests are Selenium and
Windmill. Windmill has (as of today) direct Django integration, and
the developers are keen to help us get whatever we need out of
Windmill. Selenium doesn't, and I don't know what the developer's
priorities are. I don't see that it's that hard a choice given the
alternatives.
> I don't want to just add Windmill tests because we can - we need to be
> adding tests that actually add value for regression purposes. It's
> very easy to write functional tests that don't actually validate
> functionality - they just make it difficult to modify code.
We *really* need coverage of the admin UI. Working on the admin
actions, I've twice checked in JavaScript that doesn't work on IE. We
wouldn't tolerate committing code broken on MySQL; why do we tolerate
code broken on IE? I see adding functional UI tests as a necessity if
we want to keep adding new admin features.
> We also need to ensure that the test suite continues to run for those
> that haven't got Windmill installed (albeit with a warning).
Indeed. Anything else is broken.
> There is
> an old ticket that proposed to allow skipping tests that are known
> failures, reporting them as 'known failures/skipped tests' rather than
> outright failures. This might be something worth including in your
> proposal.
Yeah, it's http://code.djangoproject.com/ticket/4788; that'd make an
obvious addition to the project. Kevin, you should take a look at the
skip-test feature added to Python (trunk) recently; might be worth
ripping off... err... adapting for our purposes.
> django-test-utils is a great set of tools, but working on those tools
> is (IMHO) out of scope for the SoC, since it isn't part of the Django
> project itself.
Agreed. Find Django core stuff to work on, not third-party projects.
> I was under the impression that GSoC was intended to be a full-time
> activity - however, I couldn't find any reference to this in the FAQ.
> You may want to confirm that a 'part time' application is allowed.
Google doesn't specify
(http://socghop.appspot.com/document/show/program/google/gsoc2009/faqs#student_time);some
projects require full-time status, but I'm OK with not as long as the
student's up front about how much time he'll be able to spend. Given
that none of *us* work full-time on Django I don't see that we should
absolutely require our students to do so.
Jacob
Kevin Kubasik
Kevin Kubasik
http://kubasik.net/blog
Kevin Kubasik
Almad
> On Tue, Mar 31, 2009 at 8:52 PM, Russell Keith-Magee
> Windmill. Windmill has (as of today) direct Django integration, and
> the developers are keen to help us get whatever we need out of
> Windmill. Selenium doesn't, and I don't know what the developer's
> priorities are. I don't see that it's that hard a choice given the
> alternatives.
that can integrate selenium into tests.
I'd be glad if they can be added/integrated/used/whatever; see them at
http://devel.almad.net/trac/django-sane-testing/browser/djangosanetesting/noseplugins.py
Your consert in Selenium Plugin.
BTW, main problem with both Windmill and mine integration is kind of
hacky way to put live server up (and, they're also conflicting with
each other). Any solution to that (like, integrating into django core)
would be just great (althrough I don't know if accepted, see
http://code.djangoproject.com/ticket/2879 ).
> Jacob
Almad
Russell Keith-Magee
> 3. Utilize the new Windmill test support to provide coverage of the Admin
> interface. This is an extremely large task overall, but given how easy
> Windmill has made the creation of tests, I hope that I will be able to
> provide coverage of a majority of the admin's functionality. This will not
> include support for the django.contrib.gis namespace, but I will try and
> test all other namespaces that have Admin integration or functionality
> (namely auth, admin and comments).
This proposal doesn't really capture the dependency between this task
and fixing/implementing #4788. This isn't a big problem, but it's
worth mentioning.
> 4. **New** Cleanup/Expand the current testing API to include a few new
> super-awesome features/functions. Namely, I want to implement/integrate the
> following tickets (listed in order of priority):
> a. `Skipping Tests <http://code.djangoproject.com/ticket/4788>`_
> b. `assertNumQueries <http://code.djangoproject.com/ticket/5416>`_
> c. `assertContext <http://code.djangoproject.com/ticket/5333>`_
I haven't checked for certain, this one may not be required any more
following the changes to the test response.context introduced in v1.1.
> d. `assertNoBrokenLinks <http://code.djangoproject.com/ticket/5418>`_
> e. `Mock Request Creation Outside of urls
> <http://code.djangoproject.com/ticket/9002>`_
> f. `fixtures with doctest <http://code.djangoproject.com/ticket/5624>`_
> h. `TestOnly Models <http://code.djangoproject.com/ticket/7835>`_
#7835 is a big one for me - this would allow for much better testing
of the admin, as well as admin tests that were actually associated
with the admin contrib app. It would also significantly simplify the
process of testing schema evolution type apps, or any other
application that operates on models.
> i. `reStructured Text Checker <http://code.djangoproject.com/ticket/7474>`_
> j. `fuzz or datagen testing <http://code.djangoproject.com/ticket/5419>`_
On the subject of these tickets - it would be worth mentioning in your
proposal that many of these tickets already have patches in various
forms. At least part of the task here is to review and update any
existing patches. This goes some way to explaining why closing 10
non-trivial feature tickets is only scheduled as 5 weeks part-time
work.
Other than that - this is looking quite attractive as a proposal. The
only additional suggestion I would make would be to beef out the
"About me" section. There are two parts required for your project to
be selected:
* Your proposal needs to be sound and of interest to Django
* We need to have confidence that you are someone that is capable of
bringing the proposal to completion.
You've pretty much got the first bit sorted now, but you haven't
really got much to convince us of the second point.
Yours,
Russ Magee %-)
Kevin Kubasik
Mikeal Rogers
I'm one of the co-creators of the Windmill project and I thought I
should clarify the different levels of support Windmill has for django
and nose.
Windmill tests have a fairly integrated test framework of their own.
Many months back we added a feature that allowed django projects to
define a unittest class that bootstrapped the windmill framework to
run tests inside django's testing framework. This solution was
somewhat hacky as it meant that a single pass or fail was returned to
django's testing framework. This support also overrides some django
TestCase internals to do what (http://code.djangoproject.com/ticket/
2879) adds.
During the PyCon sprints I wrote the management command, which starts
up the django test server and starts windmill directly. This meant
that the tests showed pass/fail for every test and showed tracebacks
for failed tests and returned sys.exit(1) on any failures and all that
good stuff. This solution discovers your windmill tests in your django
applications and doesn't override any of the django TestCase internals
and is the most elegant solution for running django windmill tests
thus far.
http://trac.getwindmill.com/wiki/WindmillAndDjango#Usingmanage.pytest_windmill
If you are moving to nose you should also know that windmill has a
great nose plugin. In fact the nose plugin doesn't have to start up
windmill's testing framework at all as nose's format for setup/
teardown of modules is compatible with windmill's.
http://trac.getwindmill.com/wiki/BookChapter-5-RunningTests#RunningTestsfromNose
If you have any questions or concerns we are very attentive to our dev
list (http://groups.google.com/group/windmill-dev) and we are also
very active on IRC (#windmill or irc.freenode.net)
-Mikeal