test suite

[Honza Král]

Hey everybody,

I'd like to propose a change to our test suite. The way we are
currently running the tests is very cumbersome and fragile. I'd like
to unify the test suite into one and introduce some tool to help with
that - either nose or py.test.

I imagine the different backend tests will be separated into their own
modules (as they are now) and then that module's setup() method will
check for the search engine running (optionally spawn it's own ?) ans
skip the tests if it doesn't find it.

How does this sound?

Thanks!


Honza Král
E-Mail: honza...@gmail.com
Phone: +420 606 678585

[Chris Adams]

This sounds great to me – the status quo is definitely too much overhead. I'm not opposed to nose/py.text but I think even just using the stdlib unittest2 would yield major improvements: most of the current test structure predates the existence of features like setupModule or skipIf which would be pretty big wins at minimal expense. Actually, on that note I'm going to go mark the content extraction tests to be skipped if pysolr <= 3.1.0… 

One other thought: as part of this I think it would make sense to review the backend-specific tests and see which ones could be split so we could maintain more of a separation between the frontend SearchQuerySet/SearchQuery interfaces and the lower-level backend tests. I've fixed a few cases which I encountered for other reasons but in connection with #925 and your more ambitious proposal it might be good to plow through and see how many cases we have where something could be tested with a mock for all backends (this would probably inform that refactor, too).

Chris

[Jannis Leidel]

On 26.01.2014, at 22:37, Honza Král <honza...@gmail.com> wrote:

> Hey everybody,
>
> I'd like to propose a change to our test suite. The way we are
> currently running the tests is very cumbersome and fragile. I'd like
> to unify the test suite into one and introduce some tool to help with
> that - either nose or py.test.
>
> I imagine the different backend tests will be separated into their own
> modules (as they are now) and then that module's setup() method will
> check for the search engine running (optionally spawn it's own ?) ans
> skip the tests if it doesn't find it.
>
> How does this sound?

+1

I also tried this a while ago but never pushed it to master since I tried to do too much (move to py.test) and eventually gave up. But if you can pull it off this would also be my preference. OTOH using the new discover test runner in Django gives you a bit more useful test discover already (and can be made backwards compatible with django-discover-runner).

Jannis

[Honza Král]

Hi Guys,

I managed to get it working in
https://github.com/HonzaKral/django-haystack/tree/test_refactor

I moved all the tests to test_haystack dir. You can run them using
your favorite test runner (i just run nosetests in that directory)
that can deal with setup/teardown in packages to control setting up
environment and skipping tests when no drivers and/or running server
is running (no running solr means skipping solr, multiindex and
spatial tests for example).


I have some trouble with solr tests where they I cannot get them to
work on python 2 and there is still some dependency between the tests
so while all the tests work for me in isolation when I run them all at
once a bunch of solr tests fail. I would greatly appreciate help with
those.

outstanding issues:
* remove the tests directory after moving the utility stuff to new
location (run solr, requirements, ...)
* provide a convenient script to run the tests + python setup.py test
* document how to run the tests
* hook up CI
* move stuff around and remove as many dependencies as possible (move
multipleindex to work on whoosh & simple backend for example)


Please let me know what you think and what works for you or doesn't

Thanks!

Honza Král
E-Mail: honza...@gmail.com
Phone: +420 606 678585

[Honza Král]

updated:
* old tests are gone
* docs updated
* script added

any feedback would be greatly appreciated.

Honza Král
E-Mail: honza...@gmail.com
Phone: +420 606 678585

[Honza Král]

Many thanks to Chris for helping review and test the code a bit, for
your consideration, a PR:

https://github.com/toastdriven/django-haystack/pull/951

Honza Král
E-Mail: honza...@gmail.com
Phone: +420 606 678585