Tests for tg2devtools

[Christoph Zwerschke]

In devtools.test there is a test_pastetemplate which is broken in 2.3
because devtools.pastetemplate does not exist any more. Are there plans
to add tests to devtools again before 2.3 is out?

-- Christoph

[Alessandro Molina]

I never noticed tg.devtools had a testsuite, as it is does it actually make any sense? 

As It runs a subset of the tests available in the testsuite of the generated package, can't we just run the testsuite of the generated app?

On Mon, Jun 17, 2013 at 6:32 PM, Christoph Zwerschke <ci...@online.de> wrote:

In devtools.test there is a test_pastetemplate which is broken in 2.3 because devtools.pastetemplate does not exist any more. Are there plans to add tests to devtools again before 2.3 is out?

-- Christoph

--
You received this message because you are subscribed to the Google Groups "TurboGears Trunk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to turbogears-trunk+unsubscribe@googlegroups.com.
To post to this group, send email to turbogears-trunk@googlegroups.com.
Visit this group at http://groups.google.com/group/turbogears-trunk.
For more options, visit https://groups.google.com/groups/opt_out.

[Jorge Vargas]

So yea, coming back from the death here :) 

That code actually does two things.

a) It automates starting a project and makes sure that is possible.

b) It then runs a generate and then runs the tests of the generated project to make sure it's tests are properly run.

So it's a test of both the generator and the generated project. To validate that changes to the templates don't break quickstart which will be very bad.

To unsubscribe from this group and stop receiving emails from it, send an email to turbogears-tru...@googlegroups.com.
To post to this group, send email to turbogea...@googlegroups.com.

[Alessandro Molina]

Great to hear you again Jorge ^^

Ok, I understand that, but as it is, it's not much more than "gearbox quickstart PROJX && cd PROJX; nosetests -v", I think that it might make sense to have it around only if we modify it to actually try all the available quickstart options to make sure that every combination of template+backend still passes the tests and is not broken.

Otherwise it's not much more than a quickstart+nosetests shell script replacement written in python.

[Alessandro Molina]

On Tue, Jun 18, 2013 at 12:18 AM, Alessandro Molina <alessand...@gmail.com> wrote:

Ok, I understand that, but as it is, it's not much more than "gearbox quickstart PROJX && cd PROJX; nosetests -v", I think that it might make sense to have it around only if we modify it to actually try all the available quickstart options to make sure that every combination of template+backend still passes the tests and is not broken.

I tried to implement a more flexible test suite for quickstart, it now runs the quickstart with different options trying to run the test suite for each one of them.

Maybe someone more proficient in nose knows a better way to run the tests of the generated application from inside the testsuite, right now it loads them with nose.run which makes the whole testsuite result as a single test.

[Christoph Zwerschke]

Am 20.06.2013 00:06, schrieb Alessandro Molina:
> I tried to implement a more flexible test suite for quickstart, it now
> runs the quickstart with different options trying to run the test suite
> for each one of them.

You beat me to it, I wanted to do in a similar way.

> Maybe someone more proficient in nose knows a better way to run the
> tests of the generated application from inside the testsuite, right now
> it loads them with nose.run which makes the whole testsuite result as a
> single test.

Yes, that's not so meaningful. The test should check how many tests were
run (should be different depending on quickstart options) and whether at
least one test was created and run at all.

We could give nose2 a try, it doesn't have the run() method but
something like discover(exit=False) should do. I'll look into that
tomorrow unless you beat me to it again.

-- Christoph

[Christoph Zwerschke]

Am 20.06.2013 00:06, schrieb Alessandro Molina:

> Maybe someone more proficient in nose knows a better way to run the
> tests of the generated application from inside the testsuite, right now
> it loads them with nose.run which makes the whole testsuite result as a
> single test.

I found that nose.run() did not run properly. There were several
problems: argv[0] must be the command name, the egg_info must be
generated first, and it had also issues with output redirection.

So I solved it differently, by running "setup.py test" for the
quickstarted project in a subprocess and checking the output. This has
the advantage that TG can switch to any other test runner like nose2 and
nothing has to be adapted in this test.

The improved test_quickstart found a failing test with Mako on Windows
due to different linefeeds that would otherwise have gone unnoticed.

We should now also add a Travis service hook for tg2devtools.

-- Christoph

[Alessandro Molina]

On Thu, Jun 20, 2013 at 11:29 PM, Christoph Zwerschke <ci...@online.de> wrote:

Am 20.06.2013 00:06, schrieb Alessandro Molina:

Maybe someone more proficient in nose knows a better way to run the

tests of the generated application from inside the testsuite, right now
it loads them with nose.run which makes the whole testsuite result as a
single test.

We should now also add a Travis service hook for tg2devtools.

Yes, that I something I wanted to tackle but there were two issues I wanted to solve before:

1) How to manage dependencies required by the tests of the quickstarted app (and this should have been solved by your change to use "setup.py test" that should install the dependencies)

2) What to do on Py3, the current travis env for TG2.3 tests on Py26, 27, 32, 33. But some of the quickstarts don't work on Py3 due to missing libraries (ming, sprox, tgext.admin mostly). What should we test? Should "gearbox quickstart" create the project anyway or should it raise an error like "--ming not supported on py3"? Or should we just SKIP the tests in tg2devtools on py3?

[Christoph Zwerschke]

Am 21.06.2013 15:12, schrieb Alessandro Molina:
> 2) What to do on Py3, the current travis env for TG2.3 tests on Py26,
> 27, 32, 33. But some of the quickstarts don't work on Py3 due to missing
> libraries (ming, sprox, tgext.admin mostly). What should we test? Should
> "gearbox quickstart" create the project anyway or should it raise an
> error like "--ming not supported on py3"? Or should we just SKIP the
> tests in tg2devtools on py3?

I would just skip them, since as long the quickstarted project doesn't
work anyway, we don't need to care.

-- Christoph

[Alessandro Molina]

Ok,

tests are now passing on both Py2 and Py3.

I provided a travis configuration file and enabled travis, but all the tests are failing due to missing dependencies,

it seems I was wrong on the "setup.py test" subprocess assumption that it would install the subtest dependencies, we currently have two issues:

1) Some tests run outside the subprocess, so we would need to install the dependencies anyway

2) The subprocess tests are failing anyway even though they would be installing dependencies.

The most simple solution would be to specify all the dependencies in tests_require of devtools, but I don't think this is the right solution, the tests should probably install the dependencies depending on the quickstarted app.

-- Christoph

[Christoph Zwerschke]

Am 23.06.2013 14:51, schrieb Alessandro Molina:
> The most simple solution would be to specify all the dependencies in
> tests_require of devtools, but I don't think this is the right solution,
> the tests should probably install the dependencies depending on the
> quickstarted app.

But then BaseTestQuickStart must also create and destroy a separate
virtualenv in its setUp and tearDown methods.

By the way, I think it's better to use setUpClass and tearDownClass in
this case, creating the environments and quickstarting the project for
every test seems overkill to me.

-- Christoph

[Alessandro Molina]

On Sun, Jun 23, 2013 at 3:27 PM, Christoph Zwerschke <ci...@online.de> wrote:

But then BaseTestQuickStart must also create and destroy a separate virtualenv in its setUp and tearDown methods.

I tried to tackle the problem in https://github.com/TurboGears/tg2devtools/commit/e44447b2ba16fd7ab7a068530346cfc1d0ec71eb

but is more complex than I expected with the root virtualenv colliding with the sub virtualenvironments and issues with sys.real_prefix.

It seems that the dependencies are correctly installed within the newly created environment, but there are some problems with the successive enter_virtualenv calls before the tests which fail due to sys.real_prefix odd status.

 

[Alessandro Molina]

On Mon, Jun 24, 2013 at 3:50 AM, Alessandro Molina <alessand...@gmail.com> wrote:

On Sun, Jun 23, 2013 at 3:27 PM, Christoph Zwerschke <ci...@online.de> wrote:

But then BaseTestQuickStart must also create and destroy a separate virtualenv in its setUp and tearDown methods.

It seems that the dependencies are correctly installed within the newly created environment, but there are some problems with the successive enter_virtualenv calls before the tests which fail due to sys.real_prefix odd status.

I made some advancements on the testsuite and now is able to pass on Py2.7 on travis: https://travis-ci.org/TurboGears/tg2devtools/builds/8390939

On Py3.2 has issues probably due to Jinja2 not supporting Py3.2

On Py2.6 and Py3.3 some tests are failing and I need to investigate why.

[Christoph Zwerschke]

Am 24.06.2013 18:18, schrieb Alessandro Molina:
> I made some advancements on the testsuite and now is able to pass on
> Py2.7 on travis: https://travis-ci.org/TurboGears/tg2devtools/builds/8390939

Great. I tried it under Windows and fixed a small issue. Another problem
is that some packages like Genshi are not available as binaries or only
available as 32bit version and will not install on Windows unless you
have installed a suitable Visual Studio version. And of course, running
the test suite is now very slow since all this stuff is now loaded from
the Internet every time, which is awful if you have a slow connection.
There should be an option to run the tests without virtualenvs or with
--system-site-packages so nothing needs to be installed when everything
is already available on the system.

-- Christoph

[Alessandro Molina]

On Mon, Jun 24, 2013 at 8:01 PM, Christoph Zwerschke <ci...@online.de> wrote:

Am 24.06.2013 18:18, schrieb Alessandro Molina:

I made some advancements on the testsuite and now is able to pass on
Py2.7 on travis: https://travis-ci.org/TurboGears/tg2devtools/builds/8390939

Great. I tried it under Windows and fixed a small issue. 

 https://travis-ci.org/TurboGears/tg2devtools/builds/8401092

It's now working on all systems apart py2.6, I'll setup a 2.6 environment with pythonz/brew and I'll try to see what goes wrong. 

Travis logs don't help much to understand the problem, we should probably print out subtests output for such cases

[Alessandro Molina]

On Mon, Jun 24, 2013 at 11:35 PM, Alessandro Molina <alessand...@gmail.com> wrote:

It's now working on all systems apart py2.6, I'll setup a 2.6 environment with pythonz/brew and I'll try to see what goes wrong. 

Ok, finally found the reason.

While on more recent versions the output for subtests is:

  tgtest00.tests.functional.test_authentication.TestAuthentication.test_forced_login

  Anonymous users are forced to login ... ok

On Python2.6 the class path is missing:

  Anonymous users are forced to login ... ok

So the script is unable to detect which tests are passing and which are not.

As this is happening on same version of nose, until a work-around is found, I'm just going to disable Py2.6 as a testing environment on travis.

[Christoph Zwerschke]

Am 25.06.2013 00:29, schrieb Alessandro Molina:
> Ok, finally found the reason.
>
> While on more recent versions the output for subtests is:
>
> tgtest00.tests.functional.test_authentication.TestAuthentication.test_forced_login
> Anonymous users are forced to login ... ok
>
> On Python2.6 the class path is missing:
>
> Anonymous users are forced to login ... ok

Right, I remember. Unittest had the annoying feature of only showing the
docstring before this was fixed in Py 2.7 with unittest2.

It is actually possible to change the output by adding a method
shortDescription() to the test class. But this only works if the test
class derives from unittest.TestCase.

-- Christoph