Sphinxtest

[The...@gmx.de]

Hi all,

I've uploaded a patch with a WIP-version of the sphinx extension I'm
hacking on. It's about doctests with fixtures similar to unittests. So
far it generates html output and allows you to run your documentation
as a normal unittest testsuite. I've written some documentation and
added it to the sphinx documentation.

What do you think about this?

Regards,
Ontje

[SHIBUKAWA Yoshiki]

Hi Ontje,

I'm thinking same thing and just started implementing.

Behavior Driven Development(BDD) tries to get closer to documentation
rather than TDD.
So I think BDD tools is more friendly to Sphinx. Techinically, BDD and
TDD are very close.
These are just different in method's names.

And more, I'm interested in cucumber's test. It is implemented on Ruby/Rspec.

Thanks,
SHIBUKAWA Yoshiki

2009/11/2 The...@gmx.de <The...@gmx.de>:

[Chris Withers]

The...@gmx.de wrote:
> I've uploaded a patch with a WIP-version

What is a WIP-version?

> hacking on. It's about doctests with fixtures similar to unittests.

Have you looked at Manuel?

http://packages.python.org/manuel/

> So
> far it generates html output

Why are you looking for html output?

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

[Ontje Lünsdorf]

Hi Chris,

Am Samstag, 7. November 2009 11:40:11 schrieb Chris Withers:
> What is a WIP-version?
A Work in progress version ;) That version is far from perfect and I just
wanted to get general feedback on the idea.

> Have you looked at Manuel?

No, not yet. Thanks for the hint.

From a first glance there is some functionality I was looking for (running the
documentation as testsuites to embed them in continuous testing utilities like
bitten).

The syntax I realized is more testsuite centric. For example a testsuite in
your documentation will look like this:

-----------8<--------------------------------
.. fixture:: simple_fixture

>>> a = 5 # Assign a variable.

>>> if a != 10: raise ValueError('w00t? I can\'t multiply!')

.. test:: testcase_with_fixture
:fixtures: simple_fixture

>>> a *= 2

This test doesn't fail because ``a`` has been assigned in the setup code.

Also take a look at the teardown code. As that is a doctest too, failures
will be reported as they would in a normal unittest.
------------------------>8--------------------

Note that the above two paragraphs are inlined documentation for the testcase.

> Why are you looking for html output?

This will result in html output as in the attached screenshot. I hope that the
integration of testsuite in your documentation will lead to uncryptic and
understandable testcode as I've seen it in my own and other projects ;-)

Also note that each testcase is run in its own environment to prevent
aftereffects. I know that this can be done with doctest blocks as well, but
they don't integrate well with testsuites as far as I know.

So, do you think that there's the need for something like this?

Regards,
Ontje

[Georg Brandl]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ontje Lünsdorf schrieb:

Are you aware of the existing doctest extension (if not, see
http://sphinx.pocoo.org/ext/doctest.html)? It already does something similar;
but not as sophisticated. It is more oriented towards making sure that the
examples you give in the documentation don't stop working.

Georg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.13 (GNU/Linux)

iEYEARECAAYFAkr6iC8ACgkQN9GcIYhpnLAtvgCffScI88nec/MYC6E95N/Cpsym
u44An2V07zeqGfx6sp1JVAYY4Yllk6xw
=JCR3
-----END PGP SIGNATURE-----

[Ontje Lünsdorf]

Hi Georg,

Am Mittwoch 11 November 2009 10:47:27 schrieb Georg Brandl:
> Are you aware of the existing doctest extension (if not, see
> http://sphinx.pocoo.org/ext/doctest.html)? It already does something
> similar; but not as sophisticated. It is more oriented towards making
> sure that the examples you give in the documentation don't stop working.

Yes, in fact I've used it extensively before. It's very nice but there were at
least two drawbacks which I tried to circumvent with sphinxtest:

Doctests in a documentation file are run as a singular test. But you often want
to test your code in different use-cases which you don't want to scatter over
multiple files. If all tests are run in a singular environment it may get messy
to find the root cause of a failure.

The sphinx doctest builder is difficult to integrate into other utilities like
continuous testing frameworks (these usually expect unittests).

I will experiment a bit more with doctest blocks and see if I can get
comfortable with those :)

Regards,
Ontje

[Georg Brandl]

I'm not saying you have to use the doctest extension :) As I said,
it's not a
serious unittesting tool, but to make sure that snippets continue to
work.
(In docs, you often use objects created earlier in later snippets, so
the shared
environment makes sense.) I just wanted to make sure you know that it
exists.

What you're developing looks very interesting, because it could at one
point
combine documentation and tests -- kind of "literate testing" :)

Georg

[Chris Withers]

Ontje Lünsdorf wrote:
> Doctests in a documentation file are run as a singular test. But you often want
> to test your code in different use-cases which you don't want to scatter over
> multiple files. If all tests are run in a singular environment it may get messy
> to find the root cause of a failure.

http://packages.python.org/manuel/#test-isolation

> The sphinx doctest builder is difficult to integrate into other utilities like
> continuous testing frameworks (these usually expect unittests).

http://packages.python.org/manuel/#using-unittest