[RFC] Test methods filtering on tests run

[Journeyman]

Hello everyone,

my name is Antonio, I'm using Django 1.9 and first of all I'd like to
thank you all contributors for the amazing framework I can work with.

I've just joined the django-dev mailing list to ask your opinion on a
really trivial patch I thought implementing.

Our backend server has with several apps: each app has its own test
package; each test package has several modules; each module has several
classes that has several methods.

In short, when we need to run tests we rarely run:

$ python manage.py test mydjangoapp.app_one.tests

More likely we run:

$ python manage.py test \
mydjangoapp.app_one.tests.test_module.TestClass.test_method_1

This happens especially when we need to investigate a regression, we
need to do a lot of writing because single tests are failing and we just
need to run that very test.

So I thought that it could be very useful a shortcut to get to the
"test_method_1" without writing the full CamelCased namespace all the
way down there.

I hacked together as proof a patch that adds an optional parameter to
the Django Command test.py:

$ python manage.py test --method-filter test_method_1

this can be a comma separated list of methods:

$ python manage.py test --method-filter test_method_1,test_method_2,...

Basically this patch rewrites the TestSuite object created in
DiscoverRunner.run_test(), filtering out method names that do not match
that list. Of course in case of name clashes two test methods from
different apps will run, but I don't find it an issue as we give
meaningful names to test methods. Also, there are many ways that such a
check could be implemented to give it flexibility (albeit simple).

So far I find it quite useful, but maybe because this patch solves a
very narrow use case (mine :-)) but I would like to know if 1) I missed
an already implemented way for such thing and if not 2) would you be
interested in a pr with such a patch.

Thanks for your comments, thoughts, suggestions, etc.
Sorry for the long post.

Regards,
Antonio

[Josh Smeaton]

Hi Antonio

I have the same problem when running tests in Django's test suite. When there's a failure, I have to copy the test path, paste that, then copy the failing test. The entire path to the test isn't printed in the failures. I'd be a big fan of *some* kind of implementation that allows me to filter test methods. I think I'd prefer some kind of glob syntax though, so I could do:

./manage.py test app_package**test_method

That would allow you to be as targeted as you needed to be.

Would you mind posting your patch somewhere? Reviewing code along with the description and seeing potential for changes would be cool.

Josh

[ludovic coues]

I might be saying something stupid, but rather than filtering test,
would it not be better to have a flag to rerun all the test that
failed in the previous run ?

So the command would always be the same for testA, testB or even both,
and might be more user-friendly if there is a bunch of test failing.

> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/0f9fc55e-9568-41d1-92f9-79effabd8e4b%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.



--

Cordialement, Coues Ludovic
+336 148 743 42

[Steve Jalim]

On Tuesday, May 17, 2016 at 11:06:43 AM UTC+1, ludovic coues wrote:

I might be saying something stupid, but rather than filtering test,
would it not be better to have a flag to rerun all the test that
failed in the previous run ?

That's why we extende DiscoverRunner to make https://pypi.python.org/pypi/django-juno-testrunner 

It's not the best code, but it works well for us. Maybe something in this pattern (ie, a flag to generate and consume a rerun log) might be another way?

Steve

[Journeyman]

On 17/05/2016 02:32, Josh Smeaton wrote:

> I have the same problem when running tests in Django's test suite. When
> there's a failure, I have to copy the test path, paste that, then copy
> the failing test.

Now that you mention it, another improvement could be to compose a
ready-to-copy-and-paste "url" for the output of failing tests. Instead of:

====================
ERROR: test_method_1 (mydjangoapp.app_one.tests.test_module.TestClass)
Test docstring
====================

we could tweak printErrorList() into something along the line of:

====================
ERROR: test_method_1 (mydjangoapp.app_one.tests.test_module.TestClass)
Test docstring
url: mydjangoapp.app_one.tests.test_module.TestClass.test_method_1
====================

> Would you mind posting your patch somewhere? Reviewing code along with
> the description and seeing potential for changes would be cool.

Sure, no problem.

http://paste.debian.net/686941/

It's somehow rough, I'm simply adding a new optional parameter to the
Django Command that runs the tests; this parameter will force an
iteration through the test suite and discard those methods whose name is
not specified. I could not directly access the TestSuite objects to
remove an item (which is funny since internally unittest.TestSuite uses
a list).

It could be refactored somewhere else or improved like you suggested,
provided some examples of what kind of globbing/regex you think it would
be useful f.e. glob test methods only? Packages only? A dot separated
syntax to specify both? Etc.

Concerning the suggestion of Ludovic of gathering the failing tests and
re-run them, I agree that a dedicated tool like the juno-testrunner is
better suited for the job.

Thanks for your comments.

Antonio

[Journeyman]

I have created a ticket on TRAC:
https://code.djangoproject.com/ticket/26659

If the ticket is accepted I'll push it as a pr.

Comments are welcome.

Regards,
Antonio