Request for feedback on GSoC proposal "Test framework cleanup"

[Varun Sharma]

Hi All,

I've been writing this proposal from past few days. After going through the pycon talks of Carl Meyer and Julien Phalip on Django's testing framework and working on the test framework from last 2 months, I have drafted some improvements but I'm not sure about any roadblocks or feasibility issues that may arise during the implementation. In order to minimise that, I request you all to have a look at the proposal and help me out on the feasibility issues. I'd really appreciate any kind of feedback and will do the required changes accordingly.

Proposal: https://gist.github.com/varunsharma/597058c5da51d86698e3

Thanks

Varun

[Russell Keith-Magee]

Hi Varun,

I've just taken a look at this proposal; I've got some concerns.

 * You say you can get test coverage from 81% to 90% - that's an admirable goal, but is there any reason to believe that it is achievable? In many cases, the coverage is low because a feature/branch is so difficult to test that the cost in developing test harnesses vastly exceeds the likelihood of breakage, or because the branches that aren't covered aren't *ever* going to be executed (i.e "you shouldn't be here" branches). 

Don't get me wrong - more coverage would be better - but you're going to need to provide evidence of what you're going to focus on to gain 9% in test coverage. It's not just a matter of saying "I'm gonna get 9% more".

 * I am in no way convinced that shuffling deck chairs to have a unit/integration/regression separation will be in any way helpful. Django once had a separate regression tests directory - we *removed* that separation specifically because it was becoming burdensome to identify whether a test was a "core function" test or a "regression" test.

 * I'm not convinced that more options to ./runtests.py is an area where we should be spending effort - or, to the extent that it is, we should only be seeking to expose functionality provided by the base unittest runner. If this means that changes are needed in unittest itself, then that's a project to direct towards Python core - which unfortunately puts it out of scope for Django.

 * I'm not convinced that a runner to help run all database backends at once is ultimately that helpful - I wouldn't have considered the burden of running three different test runs (or relying on the CI server, which does this for you) was sufficiently onerous to warrant spending time on the problem.

 * Regarding schedule - I've said this in other threads, and I appreciate that this is a first cut proposal -- but I'm *deeply* sceptical of any proposal that contains estimates of granularity > 2 weeks. The bulk of your timeline is in a 5 week block and a 4 week block - but you haven't provided any evidence that 5 weeks is an appropriate amount of time for the work provided, rather than just a number that adds up nicely to 12 weeks.

Yours,

Russ Magee %-)

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAJ7sbhEcj4uCk9U0%3DwbMY8wV1J0wQRjcma703W_%2BMe%2Br0Gmj%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Carl Meyer]

Hi Varun,

I think there are some interesting pieces in here. Specific comments:

- I'm more optimistic than Russ that we can/should be able to get close
to 100% code coverage in Django's test suite (when coverage is summed
over all the configurations that CI runs, which I don't think the
current django-coverage job does -- finding a way to fix that would be a
useful task in itself, I think), and "analysing and improving test
coverage" is the piece of your current proposal that I find the most
interesting. In addition to improving test coverage, I would expect that
project to also uncover some bugs and some dead code paths. It would be
nice to see a bit more analysis of the current state of coverage in your
proposal, where the gaps are, and what it would take to fill them. While
I appreciate that you've broken down that part of the work into
week-long chunks, it looks a bit like you've just assumed that you can
do two top-level modules per week, which doesn't indicate that you've
put a great deal of thought into what coverage is missing and how many /
what kinds of tests you'd need to write to fill it. I would guess that
there might be several-order-of-magnitude differences in the quantity of
work needed between various top-level modules with missing coverage.

- Like Russ, I do not think that splitting up the test suite into
"unit", "integration", and "regression" categories will result in enough
benefit to justify the work.

- I am not excited about adding more options to runtests.py. I'm not
sure that Django should be gradually inventing a more and more complex
in-house test runner, when the Python community already has several
battle-tested runners that already include many of the features we want
(and have flexible plugin architectures that would allow adding any
missing features we need). IMHO the best option currently is py.test.
I'd be more interested in a proposal to explore converting the Django
test suite to run with py.test than in a proposal to add more features
to runtests.py. But that's just my opinion; there may be others on the
core team who would be opposed to such a move.

- Regarding the specific options you propose to add to runtests.py:

-- My initial thought is that running multiple databases consecutively
doesn't seem that useful; it's not hard to run two commands (you can
even put them on the same shell command-line if you want). But then I
thought that the one place this feature could come in handy is making it
easier to measure coverage across runs with all database backends.

-- Specifying specific test apps to run is something we already have;
you can just list them on the command line.

-- Specifying specific test apps to skip is something I don't think I've
ever wanted; do you have a use case for it?

-- Like I said above, I don't think the unit/regression/integration
split is worth the work, and I'm not sure what the actual use case would
be for running just one "type" and not the others.

Hope this feedback is useful to you in improving your proposal.
Carl