order of execution of tests
scastro
I've read somewhere the fact that nose doesn't run the test fixtures
in the order they appear in the class. Is this going to be fixed in
the next release?
The other thing I've noticed is that if I add three classes from three
different files to a list of testcases and give it to nose to run, it
will not keep the order of the classes I give either. An example:
file1.py:
class first(unittest.TestCase):
def test1(self)
pass
file2.py:
class between(unittest.TestCase):
def test2(self)
pass
file3.py:
class last(unittest.TestCase):
def test3(self)
pass
In another file, I create a list of the tests:
# get classes from file1.py, file2.py and file3.py, so that testlist
contains all testMethods from
all three files classes.
testlist = []
for c in classes:
for attr, value in c.__dict__.iteritems():
testlist.append(c(attr))
# I give this list to nose:
nose.run(argv=[sys.argv[0],"-d","-s","--with-xunit","--verbosity=2","--
xunit-file="+xmlfile], suite=testlist)
If I want to make sure that file2.py runs between file1.py and
file3.py, this is not possible, as nose will not run them in the order
they were added in testlist. Is there any way to do this?
I appreciate any help.
Sandra
Mike Mazur
On Tue, Jul 20, 2010 at 17:40, scastro <dri...@gmail.com> wrote:
> I've read somewhere the fact that nose doesn't run the test fixtures
> in the order they appear in the class. Is this going to be fixed in
> the next release?
It's good unit testing practice to have all tests independent of each
other. If this is true, the order in which the tests run is
irrelevant.
If you have tests whose result depend on the order they're run, you
have an Erratic Test[1] code smell.
I hope nose never supports specifying the order of unit tests :)
Hope that helps,
Mike
[1] Of particular interest are the Interacting Tests, Interacting Test
Suites and Lonely Test sections:
http://xunitpatterns.com/Erratic%20Test.html
scastro
Thanks for your reply. I understand that. My tests are all independent
of each other. But in the following example:
nosetests -d -s --verbosity=2 test_report.py test_lsof.py
test_flagdata.py test_lsof.py
I want test_lsof.py to run right after each of the other two tests. In
this particular case, test_lsof.py contains a system call to "lsof" to
monitor how many open files are left by the unit tests. That's why the
order is important here. The above is a valid example of when the
order is important. If the user explicitly specify this order, it
should be kept.
Sandra
Mike Mazur
On Tue, Jul 20, 2010 at 19:59, scastro <dri...@gmail.com> wrote:
> Thanks for your reply. I understand that. My tests are all independent
> of each other. But in the following example:
>
> nosetests -d -s --verbosity=2 test_report.py test_lsof.py
> test_flagdata.py test_lsof.py
>
> I want test_lsof.py to run right after each of the other two tests. In
> this particular case, test_lsof.py contains a system call to "lsof" to
> monitor how many open files are left by the unit tests. That's why the
> order is important here.
If there's a need to run lsof after a test, why not call a function
which performs the desired task at the end of the test?
It feels like you're overloading tests for more than one purpose.
There's probably a better way to get what you need. That's usually
what code smells indicate :)
Mike
scastro
hundreds of files and I don't want to modify each one of them to
include the system call.
jminne
what you need. Sounds like you just need to define afterTest.
http://somethingaboutorange.com/mrl/projects/nose/0.11.3/plugins/interface.html#nose-plugin-api
.:. brainsik
> It's good unit testing practice to have all tests independent of each
> other. If this is true, the order in which the tests run is
> irrelevant.
execute before more complex tests. This lets me easily see how
fundamental a class of error may be. When tests are mixed up, it takes
longer to see that there is a base case failing that is causing other
cases to fail as well. I've gotten in the habit of prefixing super
simple tests that I want to run first with something like "test_000_".
That said, I'm open to any suggestions or comments regarding this
practice.
Thanks!
.:. jeremy
Garrett Cooper
Having determinism is paramount to determining faults in code
(otherwise, what's the purpose of testing without traceability?). So
I'd argue that this non-determinism will cause you more grief than it
would help in making things more efficient.
-Garrett
scastro
beauty of the plugins and wrote one which calls lsof after each test,
doing exactly what I need.
Cheers,
Sandra
On Jul 22, 3:03 am, Garrett Cooper <yaneg...@gmail.com> wrote:
> On Wed, Jul 21, 2010 at 5:57 PM, .:. brainsik
>
Mark Roddy
> You received this message because you are subscribed to the Google Groups "nose-users" group.
> To post to this group, send email to nose-...@googlegroups.com.
> To unsubscribe from this group, send email to nose-users+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nose-users?hl=en.
>
>
The way I deal with this (at least when using nose as a test runner),
is to manage tests across multiple directories. The fast tests used
for quick feedback are in the unit test directory and anything needing
to perform out of process IO (files, network requests, etc) and any
test known to take more than a certain amount of time go in the
integration tests directory. I've toyed with breaking them down
further, but that hasn't been necessary at this point. Then there is
a script that runs them all, in order of their complexity, and stops
on the first failure. That way you get the immediate feedback.
-Mark
.:. brainsik
executing first. We run a lot of tests in parallel (using --
processes=NN). The problem we found is a few of our tests take as long
to run as all the other tests combined. If we kick those long-running
tests off first and have the others running in parallel with them,
they tend to all finish around the same time. If those long-running
tests get kicked off later, then our build time gets long. We have a
continuous integration system and are moving to continuous deployment,
so having builds go as fast as possible is important.
We ended up making a special directory that would sort before the
others and put our long-running tests in there. They kick off first,
which is great, but we've had to separate them from the rest of test
directory structure, which is confusing.
Any suggestions for doing this kind of thing?
Thanks!
.:. jeremy
.:. brainsik
> Having determinism is paramount to determining faults in code
> (otherwise, what's the purpose of testing without traceability?). So
> I'd argue that this non-determinism will cause you more grief than it
> would help in making things more efficient.
doesn't change the outcome in terms of success or failure, just the
ordering of the failures displayed. When I'm scanning the failures, I
like to start at the top and work my way down. If I see a simple test
fail, then I know it's going to be followed by a slew of more complex
tests failing, but I can ignore those and get to fixing the issue that
affected the simple test. When all the results are jumbled together it
takes longer to see if there is a root problem common to many
failures.
.:. jeremy
Holger Krekel
that mirrors my experiences why test ordering can be helpful.
If i want to ensure that tests are isolated i rather like to
explicitely use some randomization technique.
best,
holger
Garrett Cooper
Well, yeah... if you group things together appropriately, they can
focus as one atomic block that doesn't affect the other set of tests,
and can parse the results, I suppose it's not an issue.
-Garrett
Kumar McMillan
<brainsi...@saucelabs.com> wrote:
> ... I like to see simpler tests
> execute before more complex tests. This lets me easily see how
> fundamental a class of error may be. When tests are mixed up, it takes
> longer to see that there is a base case failing that is causing other
> cases to fail as well. I've gotten in the habit of prefixing super
> simple tests that I want to run first with something like "test_000_".
>
> That said, I'm open to any suggestions or comments regarding this
> practice.
In addition to the suggestion of keeping your "fast" tests and "slow"
tests in their own directories you could also use attributes:
http://somethingaboutorange.com/mrl/projects/nose/0.11.2/plugins/attrib.html
Your first run of tests could be:
$ nosetests -a fast
then if that passes,
$ nosetests -a slow
>
> Thanks!
.:. brainsik
> Your first run of tests could be:
> $ nosetests -a fast
>
> then if that passes,
> $ nosetests -a slow
to run as a sanity check). However, the problem for our build is we
need the slow tests and the fast tests to run at the same time (via --
processes=NN) with the slow tests started first. Unfortunately, the
order attributes are specified doesn't matter:
$ nosetests -v -a slow -a fast
tests.test_slow1 ... ok
tests.test_fast1 ... ok
tests.test_fast2 ... ok
tests.test_fast3 ... ok
tests.test_slow2 ... ok
tests.test_slow3 ... ok
.:. jeremy
Kumar McMillan
<brainsi...@saucelabs.com> wrote:
> On Jul 22, 8:58 pm, Kumar McMillan <kumar.mcmil...@gmail.com> wrote:
>> Your first run of tests could be:
>> $ nosetests -a fast
>>
>> then if that passes,
>> $ nosetests -a slow
>
> I've been thinking of doing that for a smoketest (quick set of tests
> to run as a sanity check). However, the problem for our build is we
> need the slow tests and the fast tests to run at the same time (via --
> processes=NN) with the slow tests started first. Unfortunately, the
> order attributes are specified doesn't matter:
As mentioned earlier in the thread, it sounds like instead you
probably need to perform this as setup either in the top level package
or in a custom plugin. That way you can guarantee that the code your
tests depend on will run at the right time. Depending on the order of
test execution is a slippery slope and will start to bite you hard
once you scale up your test suite.
>
> $ nosetests -v -a slow -a fast
> tests.test_slow1 ... ok
> tests.test_fast1 ... ok
> tests.test_fast2 ... ok
> tests.test_fast3 ... ok
> tests.test_slow2 ... ok
> tests.test_slow3 ... ok
>
> .:. jeremy
>
.:. brainsik
getting repeated:
On Jul 23, 9:09 am, Kumar McMillan <kumar.mcmil...@gmail.com> wrote:
> That way you can guarantee that the code your tests depend on will run at the right time. Depending on the order of test execution is a slippery slope and will start to bite you hard once you scale up your test suite.
before fast tests because it significantly lowers the time it takes to
run our test suite since we use --processes.
Also, in the example I gave before about wanting to see simpler tests
fail before complex ones, all the tests are independent.
For some reason, there seems to be a belief that if you want a
particular order of execution it's because of dependency problems. In
both my examples, this is not the case. There are valid reasons for
wanting independent tests to happen in a particular order.
.:. jeremy
Kumar McMillan
<brainsi...@saucelabs.com> wrote:
> I don't want to keep beating this horse, but this misconception keeps
> getting repeated:
I was responding to this quote in your last message:
"...However, the problem for our build is we
need the slow tests and the fast tests to run at the same time (via --
processes=NN) with the slow tests started first..."
If you need the slow tests to start first then your tests are not
independent. If that is my misunderstanding then maybe you can
explain it in a different way.
> Our tests are completely independent. We want to start slow tests
> before fast tests because it significantly lowers the time it takes to
> run our test suite since we use --processes.
if it's just for a speedup then again I'd suggest something like a
plugin that defines begin() so that the plugin can perform the needed
preparation code that will give you the speedup.
If instead you still want to control ordering there's probably not an
easy way to do that in Nose, especially when using multiple processes.
-Kumar
>
> Also, in the example I gave before about wanting to see simpler tests
> fail before complex ones, all the tests are independent.
>
> For some reason, there seems to be a belief that if you want a
> particular order of execution it's because of dependency problems. In
> both my examples, this is not the case. There are valid reasons for
> wanting independent tests to happen in a particular order.
>
> .:. jeremy
>
jonozzz
those who need this kind of functionality and I was able to implement
it using python-graph in our current testing framework.
For instance a test suite can take up to 4 hours and it 99% of it
depends on some critical tests that have to run at the beginning. If
one of those critical tests fails, everything after that may pass/fail
but the results are meaningless and the test runner is just wasting 4
hours and needed resources. By using graphs I was able to create
complex tree hierarchies in which if root fails then all its dependent
children are skipped.
I hoped that I'd find this functionality in nose or that someone wrote
a plugin that could do that. Apparently it's not that trivial but at
least it's possible :)
J
> On Mon, Jul 26, 2010 at 11:46 AM, .:. brainsik
>
> > I don't want to keep beating this horse, but this misconception keeps
> > getting repeated:
>
> I was responding to this quote in your last message:
>
> "...However, the problem for our build is we
> need the slow tests and the fast tests to run at the same time (via --
> processes=NN) with the slow tests started first..."
>
> If you need the slow tests to start first then your tests are not
> independent. If that is my misunderstanding then maybe you can
> explain it in a different way.
>
> > Our tests are completely independent. We want to start slow tests
> > before fast tests because it significantly lowers the time it takes to
> > run our test suite since we use --processes.
>
> if it's just for a speedup then again I'd suggest something like a
> plugin that defines begin() so that the plugin can perform the needed
> preparation code that will give you the speedup.
>
> If instead you still want to control ordering there's probably not an
> easy way to do that in Nose, especially when using multiple processes.
>
> -Kumar
>
>
>
> > Also, in the example I gave before about wanting to see simpler tests
> > fail before complex ones, all the tests are independent.
>
> > For some reason, there seems to be a belief that if you want a
Holger Krekel
> I'm definitely in favor of test dependencies/ordering. I'm one of
> those who need this kind of functionality and I was able to implement
> it using python-graph in our current testing framework.
>
> For instance a test suite can take up to 4 hours and it 99% of it
> depends on some critical tests that have to run at the beginning. If
> one of those critical tests fails, everything after that may pass/fail
> but the results are meaningless and the test runner is just wasting 4
> hours and needed resources. By using graphs I was able to create
> complex tree hierarchies in which if root fails then all its dependent
> children are skipped.
>
> I hoped that I'd find this functionality in nose or that someone wrote
> a plugin that could do that. Apparently it's not that trivial but at
> least it's possible :)
Do you have ideas or examples in mind how you'd like to concretely
define ordering or test dependencies?
best,
holger
