[GSOC] Improving the Test-Suite

[Akshay Jaggi]

Hello!

I was planning to work on improving the Test-Suite for GSOC 2014.

I would need some help in formalising and improving my proposal, so that it meets the requirements.

One of the improvements I see is classification of test cases. Classifying them into categories (read multiple-categories), would make it easier for users/developers/maintainers to run them. Basis of classification,etc is what I am still thinking on. But surely classification will help in deciding which all test cases to run. For example - just running third-party app test cases, or just run my test cases, or those which check part ABC of my project, or just those with priority set to important.

How to run tests? Here we have a few choices.

-> Allowing the ability to decide and run test cases from the admin interface. (Will people like it? Choosing what tests to run will become easy for sure. This will require the server to be up though. Will this be problematic?)

-> Sticking with and improving the current way. Specify what all tests to be run. (Do I want to add to this, every time I put new apps/add test cases? Or delete from this?)

Specify default settings per app? App developer can decide if he wants the tests to be included or excluded by default. Helpful, if say, app (or certain components of the app) not dependent on other things, say DB,etc. and it has been tested many times before, then there is no need to run those tests by default.

-> Having both of the above. ( Coherence between both of these? )

I'm still brain-storming on other possible improvements we can do. And I am going through tickets to see current problems with testing.

I'm willing to hear your opinions and comments. 

Thanks

Akshay Jaggi

[Russell Keith-Magee]

Hi Akshay,

On Fri, Feb 14, 2014 at 9:12 PM, Akshay Jaggi <akshay1...@gmail.com> wrote:

Hello!

I was planning to work on improving the Test-Suite for GSOC 2014.

I would need some help in formalising and improving my proposal, so that it meets the requirements.

Great to hear! Here's some feedback:

 

One of the improvements I see is classification of test cases. Classifying them into categories (read multiple-categories), would make it easier for users/developers/maintainers to run them. Basis of classification,etc is what I am still thinking on. But surely classification will help in deciding which all test cases to run. For example - just running third-party app test cases, or just run my test cases, or those which check part ABC of my project, or just those with priority set to important.

How to run tests? Here we have a few choices.

-> Allowing the ability to decide and run test cases from the admin interface. (Will people like it? Choosing what tests to run will become easy for sure. This will require the server to be up though. Will this be problematic?)

This doesn't sound like a very viable idea to me. This is something that needs to be persisted in code; a web server interface to manage this doesn't strike me as a good idea. Unless you've got a particularly inspired way for handling this, I don't think this will work.

 

-> Sticking with and improving the current way. Specify what all tests to be run. (Do I want to add to this, every time I put new apps/add test cases? Or delete from this?)

Specify default settings per app? App developer can decide if he wants the tests to be included or excluded by default. Helpful, if say, app (or certain components of the app) not dependent on other things, say DB,etc. and it has been tested many times before, then there is no need to run those tests by default.

-> Having both of the above. ( Coherence between both of these? )

I would envisage that this would be a declarative process - in code, marking a specific test as a "system" test or an "integration" test (or whatever other categories we develop). 

 

I'm still brain-storming on other possible improvements we can do. And I am going through tickets to see current problems with testing.

I'm willing to hear your opinions and comments. 

My other comment: have a look into how other systems handle this. In particular, look at other test running tools, like nose and py.test. Where at all possible, I would prefer to avoid reinventing the wheel; if we can leverage the tools of an existing testing framework, then we should do that in preference to building our own.

Yours,

Russ Magee %-)

 

[Chris Wilson]

Hi all,

On Sat, 15 Feb 2014, Russell Keith-Magee wrote:

> One of the improvements I see is classification of test cases.
> Classifying them into categories (read multiple-categories), would make
> it easier for users/developers/maintainers to run them. Basis of
> classification,etc is what I am still thinking on. But surely
> classification will help in deciding which all test cases to run. For
> example - just running third-party app test cases, or just run my test
> cases, or those which check part ABC of my project, or just those with
> priority set to important.

[...]

> I would envisage that this would be a declarative process - in code,
> marking a specific test as a "system" test or an "integration" test (or
> whatever other categories we develop).

It just occurred to me that most classification systems are completely
arbitrary and therefore not very useful. What's a "system" test and how
would I know whether I need to run it?

But some ideas that I can think of that might be useful are:

* Automatically building test coverage maps for each test, and reversing
them, so we can see which tests touch the line(s) of code that we just
modified, and rerun them easily. A good smoke test to run while modifying
part of Django.

* Categorising by imports: run all tests that import django.db or
django.core.http for example. Not perfect, some tests may touch facilities
without needing to actually import them, but it would be quick and cheap.

* Profile and speed up the test suite, so that we can run all tests more
quickly, especially with databases like postgres where it takes an hour to
run them all.

Cheers, Chris.
--
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Citylife House, Sturton Street, Cambridge, CB1 2QF, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

[Josh Smeaton]

Just a few observations that I've had when running the test suite that may be relevant.

- There are lots and lots of different test modules that may be relevant to a particular change, and some may not seem relevant until you run the entire suite

- bug* modules are hard to classify without reading the tests or the ticket

- *_regress modules seem too complex, and should be folded back in to the main test module

- Creating a new test module is not as easy as it could be. Basically, copy another test module, and search/replace the fixtures (if they exist)

- Setup/Teardown can be quite expensive across a large number of tests. Perhaps individual tests could be longer (which is bad practise, but practical with respect to time)

- There are 13 individual admin test modules. Perhaps it'd be nicer to have them all under a single admin module, with separate TestCases within. This applies to other systems like the ORM.

- It'd be nice if test modules could be parallelised, to improve total run time of the test suite

I think that restructuring the modules could take the place of tagging or classification in a naive kind of way, but does not allow marking two systems as relevant. For example, the checks framework has tests relating to the admin, which should be run alongside any admin tests, but wouldn't necessarily live in the admin module.

Would pytest help with any of the issues observed with the current test suite?

- Josh

[Akshay Jaggi]

I'm so sorry. Was busy with some assignments. I will look into all the mentioned points and reply back as soon as possible. :)

[Akshay Jaggi]

On Sunday, 16 February 2014 06:27:01 UTC+5:30, Josh Smeaton wrote:

Just a few observations that I've had when running the test suite that may be relevant.

- There are lots and lots of different test modules that may be relevant to a particular change, and some may not seem relevant until you run the entire suite

- bug* modules are hard to classify without reading the tests or the ticket

- *_regress modules seem too complex, and should be folded back in to the main test module

- Creating a new test module is not as easy as it could be. Basically, copy another test module, and search/replace the fixtures (if they exist)

- Setup/Teardown can be quite expensive across a large number of tests. Perhaps individual tests could be longer (which is bad practise, but practical with respect to time)

- There are 13 individual admin test modules. Perhaps it'd be nicer to have them all under a single admin module, with separate TestCases within. This applies to other systems like the ORM.

- It'd be nice if test modules could be parallelised, to improve total run time of the test suite

I think that restructuring the modules could take the place of tagging or classification in a naive kind of way, but does not allow marking two systems as relevant. For example, the checks framework has tests relating to the admin, which should be run alongside any admin tests, but wouldn't necessarily live in the admin module.

 

This is exactly what I was saying. Grouping them into corresponding folders sure is a naive way of classification, but there could be tests which may belong to multiple categories. Even if it becomes a pain to classify existing tests, giving the option to do so, in future, will surely lead to more structured and logically separated tests.

 

Would pytest help with any of the issues observed with the current test suite?

It will surely help. With features such as parallelising tests, to distributing them to multiple machines and testing on multiple platforms at once, pytest seems to me like a good choice.

Thanks for your comments. :)

--

Akshay Jaggi

[Akshay Jaggi]

On Saturday, 15 February 2014 20:47:37 UTC+5:30, Chris Wilson wrote:

Hi all,

It just occurred to me that most classification systems are completely
arbitrary and therefore not very useful. What's a "system" test and how
would I know whether I need to run it?

For development purposes we can stick to certain pre-defined and fixed categories. 

Users may define their own categories according to their own needs and wishes.

 

But some ideas that I can think of that might be useful are:

* Automatically building test coverage maps for each test, and reversing
them, so we can see which tests touch the line(s) of code that we just
modified, and rerun them easily. A good smoke test to run while modifying
part of Django.

py.test has a plugin for coverage reporting -> https://bitbucket.org/memedough/pytest-cov/overview.

Might turn out to be helpful.

 

* Categorising by imports: run all tests that import django.db or
django.core.http for example. Not perfect, some tests may touch facilities
without needing to actually import them, but it would be quick and cheap.

This looks like another possible way we can go.

 

* Profile and speed up the test suite, so that we can run all tests more
quickly, especially with databases like postgres where it takes an hour to
run them all.

Py.test -> we can parallelise and distribute test loads.

--

Akshay Jaggi

[Akshay Jaggi]

Hi,

On Saturday, 15 February 2014 06:42:49 UTC+5:30, Russell Keith-Magee wrote:

Hi Akshay,

Great to hear! Here's some feedback:

 

One of the improvements I see is classification of test cases. Classifying them into categories (read multiple-categories), would make it easier for users/developers/maintainers to run them. Basis of classification,etc is what I am still thinking on. But surely classification will help in deciding which all test cases to run. For example - just running third-party app test cases, or just run my test cases, or those which check part ABC of my project, or just those with priority set to important.

How to run tests? Here we have a few choices.

-> Allowing the ability to decide and run test cases from the admin interface. (Will people like it? Choosing what tests to run will become easy for sure. This will require the server to be up though. Will this be problematic?)

This doesn't sound like a very viable idea to me. This is something that needs to be persisted in code; a web server interface to manage this doesn't strike me as a good idea. Unless you've got a particularly inspired way for handling this, I don't think this will work.  

-> Sticking with and improving the current way. Specify what all tests to be run. (Do I want to add to this, every time I put new apps/add test cases? Or delete from this?)

Specify default settings per app? App developer can decide if he wants the tests to be included or excluded by default. Helpful, if say, app (or certain components of the app) not dependent on other things, say DB,etc. and it has been tested many times before, then there is no need to run those tests by default.

-> Having both of the above. ( Coherence between both of these? )

I would envisage that this would be a declarative process - in code, marking a specific test as a "system" test or an "integration" test (or whatever other categories we develop).  

This is exactly what I had in mind.

 

I'm still brain-storming on other possible improvements we can do. And I am going through tickets to see current problems with testing.

I'm willing to hear your opinions and comments. 

My other comment: have a look into how other systems handle this. In particular, look at other test running tools, like nose and py.test. Where at all possible, I would prefer to avoid reinventing the wheel; if we can leverage the tools of an existing testing framework, then we should do that in preference to building our own.

Py.test looks good to me. It is being used in a lot of places, and can also help with speeding up the testing process, reporting line-coverage, etc.

There has been work on it -> http://pytest-django.readthedocs.org/en/latest/

Summarising, I have three things on my mind (sorted according to decreasing priority) ->

1) Py.test

2) Classification

3) Refactoring existing test-cases

 

Would be glad to hear your comments.

Thanks. :)

--

Akshay Jaggi

[Akshay Jaggi]

I just finished reading the unittest documentation. I'll start with the Django testing code ( the definitions for SimpleTestCase, TransactionTestCase, TestCase, etc.) tomorrow. Any feedback at this time, would be quite encouraging. Even a -0 or a -1 would tell me if I need suitable course correction. I seriously am quite interested, and want to present a very strong proposal on this project.

Thanks

Akshay Jaggi

[Andrew Pashkin]

BTW - I recent;y trid to run Django test suite with Py.test and pytest-django plugin and achived some success - main problem, that some tests are failing and reason of it is not clear to me yet.
This solution require pytest, django-pytest packages and lightweight configuration.
See patch in attachment. I was tested it with Django 1.4.10. Command that I used:
PYTHONPATH=..:$PYTHONPATH py.test

[Akshay Jaggi]

On Thursday, 20 February 2014 00:35:43 UTC+5:30, Andrew Pashkin wrote:

BTW - I recent;y trid to run Django test suite with Py.test and pytest-django plugin and achived some success - main problem, that some tests are failing and reason of it is not clear to me yet.
This solution require pytest, django-pytest packages and lightweight configuration.
See patch in attachment. I was tested it with Django 1.4.10. Command that I used:
PYTHONPATH=..:$PYTHONPATH py.test

I checked. Some of them were failing because 'raise unittest.SkipTest()' was used to skip over them, which is not supported by Py.Test. I'm looking into the others. :) 

[Akshay Jaggi]

On Thursday, 20 February 2014 00:35:43 UTC+5:30, Andrew Pashkin wrote:

BTW - I recent;y trid to run Django test suite with Py.test and pytest-django plugin and achived some success - main problem, that some tests are failing and reason of it is not clear to me yet.
This solution require pytest, django-pytest packages and lightweight configuration.
See patch in attachment. I was tested it with Django 1.4.10. Command that I used:
PYTHONPATH=..:$PYTHONPATH py.test

 

Even other errors raised are usually those which are in some way incompatible with Py.test. 

(I am just sampling some random tests from the 177+57 failing tests, and looking at the reason) 

For example -> 

Look at this code slice from  test_utils/tests.py:196

def test_failure(self):

        with self.assertRaises(AssertionError) as exc_info:

            with self.assertNumQueries(2):

                Person.objects.count()

        self.assertIn("1 queries executed, 2 expected", str(exc_info.exception))

        self.assertIn("Captured queries were", str(exc_info.exception))

Now, to catch the assertion in Py.Test we ought to use pytest.raises() instead of unittest.assertRaises().

Since we are not using that, it fails at the uncaught exception assertNumQueries(2).