combined hard and soft dependencies in single @Test?
Alison Winters
Is there any way to specify both hard AND soft dependencies for a
single @Test? For our case this is important so that we can shut down
a service that was started up, regardless of whether the other tests
dependent on that service succeeded. For example:
@Test
public void startUpService() {
}
@Test(dependsOnMethods = { "startUpService" })
public void doServiceTest1() {
}
@Test(dependsOnMethods = { "startUpService" })
public void doServiceTest2() {
}
@Test(???)
public void shutDownService() {
}
If we replace ??? with dependsOnMethods = { "startUpService" }, then
we may shut down the service before it has finished being tested. On
the other hand, if we add a group "serviceTests" and then do
dependsOnGroups = { "serviceTests" }, then the service will never get
shut down if any of the tests in that group fail. It would be ideal to
have a hard dependency on "startUpService" and a soft dependency on
"serviceTests".
Is there any other way of constructing a test class so that it will
have matching pairs of "start up" and "tear down" that are treated
independently to the rest of the tests? I guess we could use
@BeforeXXX and @AfterXXX, but that'd require splitting out a lot of
these tests into different classes or groups, plus we'd lose the
results testing the actual start up and shutdown procedure.
Alison
Jeff
Alison
--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
--
Alison Winters
the setup fails. I'd love to see results like this:
start up: success
test 1: success
test 2: fail
shut down: success
or
start up: fail
test 1: skip
test 2: skip
shut down: skip
I have been experimenting with the "priority" argument, but I don't
think it works the way I expect it to.
Alison
On Dec 1, 3:53 pm, Jeff <predato...@gmail.com> wrote:
> Use a group but put @Test(alwaysRun=true) on the shutDownService...maybe?
> See my LinkedIn profile at:http://www.linkedin.com/in/rjeffreyvincent
> I ♥ DropBox <http://db.tt/9O6LfBX> !!
Jason McSwain
- i'd recommend that you use the @AfterXXX and @BeforeXXX annotations for startup and teardown.
- you can define these method in a parent Test Class and have other test classes exend from that class.
- failures in before and after methods are still reported, so you will know if there is a problem.
- Also, consider that if you are running this code (start svc / stop svc), then there is an "implied test" there, and no need to test is by itself.
Another couple thoughts, if you really want a separate test for start and stop services...
- start & stop service aren't really tests, they are actions.
- create a service util class that implements 3 methods, stop(), start(), isRunning().
- system-style tests can use the @BeforeXXX / @AfterXXX methods to call start() / stop()
- create a new test class with a single test method:
- confirm service is not running
- service start()
- confirm service is running
- service stop()
- confirm service is not running
Good Luck,
-Jason-
Alison Winters
stopping a service ;-) I was trying to frame my problem in a generic
way.
In this instance I am actually converting a bunch of old Selenium
tests from JUnit 3 to TestNG, primarily so that they integrate better
with Maven and so that we can more easily group related tests. In a
lot of cases we do stuff like log into the system, add an entity of
some sort, perform some actions with that entity, then delete that
entity. What I'd like to be able to do is always attempt to delete the
entity that was added, whether or not the actions performed on it
succeeded. The idea is to make a best-case attempt at returning the
database to a pristine state to prepare it for the next test class or
test group.
Most of the online stuff I can find regarding Selenium+TestNG suggests
creating a testng.xml and hard-coding the test order from there, which
would be fine, but that's essentially going back to the old JUnit way
of doing things where you have to edit both the test class and
build.xml any time you add a new test. It would be much more
convenient if we could just add a new test in one place (the test
class) and everything else would work automatically. As it is we have
already lost Eclipse automation because TestNG runs the test methods
randomly (vs JUnit which "probably" runs them in the order they were
declared in the source files).
Alison
On Dec 1, 4:41 pm, Jason McSwain <jasonwmcsw...@gmail.com> wrote:
> Alison,
>
> - i'd recommend that you use the @AfterXXX and @BeforeXXX annotations
> for startup and teardown.
> - you can define these method in a parent Test Class and have other test
> classes exend from that class.
> - failures in before and after methods are still reported, so you will
> know if there is a problem.
> - Also, consider that if you are running this code (start svc / stop
> svc), then there is an "implied test" there, and no need to test is by
> itself.
>
> There are multiple routes you can go here, but this is how i do this type
> of testing in my Test Framework.
>
> Another couple thoughts, if you really want a separate test for start and
> stop services...
>
> - start & stop service aren't really tests, they are actions.
> - create a service util class that implements 3 methods, stop(),
> start(), isRunning().
> - system-style tests can use the @BeforeXXX / @AfterXXX methods to call
> start() / stop()
> - create a new test class with a single test method:
> - confirm service is not running
> - service start()
> - confirm service is running
> - service stop()
> - confirm service is not running
>
> Good Luck,
> -Jason-
Jason McSwain
However in that case, i can't be much help to you. Our Test Framework has been designed, to be able, to execute @Tests independently of eachother. no special order required. the tests can be a little more complex that way, but in the long run it's turned out very nicely. We don't really consider a Test as being a single simple action, but more of a User Scenario where mulitple actions and verifications are included.
if you were to rid your framework of "depends-on" stuff, then you will easily get your Eclipse compatibility back, and be able to execute a single @Test whenever you like. (great for debugging)
Best of luck,
-Jason-
Jeff
--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
--
preda...@gmail.com
P. Hill
> I think Jason's suggestion using a service can be applied to any
> start(), isStarted(), shutDown() scenario, such as setup(), isSetup(),
> cleanUp().
> Set a flag ( isSetup = true ) if @BeforeXXX succeeds and then set all
> @AfterXXX methods to alwaysRun=true. In the @AfterXXX method, do the
> cleanup only if isSetup==true.
> If you are intent on the report showing the 'shut down' method as
> skipped if start/setup fails, then I think you could write a listener
> to help handle the specific scenario you want to enforce. I don't
> have much experience with them however.
>
1. The ordering you get out of Eclipse TestNG vs ant vs testng suite
running.
and
2. ability to cleanup always.
For the cleanup, I think the setup flag with afterXXX alwaysRun=true
(as suggeted by Jeff) can be generalized to flags in the test methods,
not just the setupXXX methods.
All cleanup would catch (swallow, log or throw skip exceptions ) any
errors. I think this then allows right click running of individual
tests assuming they needn't run as a group. But if there really is a
series of tests in the class that need to be in order, you have to get
used to clicking on the class name to run the sequence.
As to forcing the ordering, Is
@Test( singlethreaded=true ) class TestingFooBar ...
useful here? I don't actually know.
Wasn't there a recent change in the ordering behavior?
But as to cleanup always, I note you can have any number of @afterXXX
methods. They don't have to be in separate classes, or all bundled in
one shutdown method.
So my suggestsions together look like:
public class TestNG (singleThreaded=true) {
private boolean cleanT1 = false;
private boolean cleanT2 = false;
@BeforeTest
public void before() {
}
@Test
public void t1() {
System.out.println("t1...");
Assert.assertTrue(false);
cleanT1 = true;
}
@Test(alwaysRun=true) // but alwaysRun might not always be appropriate
public void t2() {
System.out.println("t2... passed");
cleanT2 = true;
}
@AfterTest(alwaysRun=true )
public void cleanT1() {
if ( cleanT1 ) {
System.out.println("cleanT1()");
}
}
@AfterTest(alwaysRun=true)
public void cleanT2() {
if ( cleanT2 ) {
System.out.println("cleanT2()");
}
}
}
I get
t1...
t2... passed
cleanT2()
PASSED: t2
FAILED: t1
<stack trace>
Did I miss part of the cleanup need?
-Paul
Cédric Beust ♔
- With dependsOnGroups. This is a "hard" dependency, if a test you depend on fails, you will be skipped.
- With @Priority. This is a "soft" dependency, just an ordering that you'd like your tests to follow. Failure of a method with a lower priority will not cause a skip.
- With testng.xml. Listing classes and methods in the XML files will cause these classes and methods to be run in the same order they are found in the XML file. Again, this is a soft dependency.
--
P. Hill
>
> * With @Priority. This is a "soft" dependency, just an ordering that
> you'd like your tests to follow. Failure of a method with a lower
> priority will not cause a skip.
>
One thing missing from the documentation is that if you mix tests _with_
explicit priority "@test( priority=2 )" and ones _without_ priority
"@test " then the ones _without_ priority will be treated as if they
have priority=0. Or to say it shorter, the default priority is 0.
Using that notion, I think one part of a solution for Alison would be to
put any have-to-run-first tests at negative priorities and the
have-to-run-later tests at very large priorities. To address her(his)
concern.
>It would be much more
>convenient if we could just add a new test in one place (the test
>class) and everything else would work automatically.
if someone adds a test at the end, it will by default follow the
run-1sts and before the run-lasts. If it needs a particular position in
the sequence then an explicit priority (and physical move to that
position in the class for maintainability) sounds like a great solution.
I could imagine a set of test methods grouped into a sequence of the
priority 100-199 and another set of methods with priorities in 200-299,
but I would try to keep one sequence per test class. An alternative to
two sequence of test methods in one class is a base class that knows how
to setup and cleanup everything used in both sequences, but when called
(in the same VM) it remembers if it already setup everything, then when
each subclass (one for the priority 100s sequence, one for the priority
200s sequence) calls the setups and cleanup, each is done setup/cleanup
is done once.
Meanwhile, I think I'd still keep my cleanupWhenNeeded() methods as
@afterXXX, and not @Test( priority=99999) because I probably don't want
them to be counted as tests completed when everything runs "green".
-Paul
Alison Winters
little bit so that dependsOnMethods/dependsOnGroups works for this
scenario, though I did hit a subsequent problem with Maven/Surefire
ignoring my superclass' @BeforeClass (though Eclipse runs it fine).
I'll experiment a little more before asking more questions around
that :)
Regarding the "@Priority" thing: Paul, when I use the priority=
argument for the @After/@Before annotations it doesn't work like I
would expect it to. Specifically, it doesn't explicitly set the
ordering of the methods as an additional level of granularity above
(or below) the dependsOnMethod= or dependsOnGroup= arguments. I looked
through the Google Groups archives and it seems that the priority=
argument is only intended to set the thread priority when running
multi-threaded tests, which should probably never happen for Selenium
tests anyway. I think what Cedric was referring to with @Priority was
the custom method interceptor he described in a blog post a year or
two back. This is exactly what I need, but from what I gather from the
documentation it looks like custom method interceptors won't work on
methods with existing dependsOnMethod= or dependsOnGroup= arguments,
so I'd lose the other dependencies in that case. Ah well.
Alison
P. Hill
> Thanks everyone for all your thoughts. I've restructured the tests a
> little bit so that dependsOnMethods/dependsOnGroups works for this
> scenario, though I did hit a subsequent problem with Maven/Surefire
> ignoring my superclass' @BeforeClass (though Eclipse runs it fine).
> I'll experiment a little more before asking more questions around
> that :)
>
> Regarding the "@Priority" thing: Paul, when I use the priority=
> argument for the @After/@Before annotations it doesn't work like I
> would expect it to. Specifically, it doesn't explicitly set the
> ordering of the methods as an additional level of granularity above
> (or below) the dependsOnMethod= or dependsOnGroup= arguments.
Hmm, curious combinations. I had tried with just @BeforeXXX and
@AfterXXX with @Test( priority=n )
and got what to me looked like useful scenerios, so I think it is more
functionality than just in threads. Hopefully it is not accidental
functionality. It looks useful to me.
But I see you are right, any @test(dependsOnMethod= ) overrules any
attempt at using priority for ordering in a single threaded class and
are run after any other logical group that includes the depended-on
method (in this case the list of prioritized methods all run 1st).
At this time I would say using priority (without threads) would be an
alternative to a chain of @Test(DependsOnMethod=), the difference is
that you lose the ability to do @Test(... AlwaysRun=false). At least in
the Eclipse plugin, priority effects run order, and does not make
anything depend on anything, thus all methods run. priority is
approximately like a chain of explicitly labeled
@Test(dependsOnMethod="", AlwaysRun=true). I would think it would NOT
be useful to support the combination of both and expect priority and
dependsOnMethod to interweave together. Maybe DependsOnClass could be
useful here, but I'd have to try it and understand your whole setup and
test dependency chain to see if everything would combine in useful ways.
Good luck,
-Paul
p.s. The by-design running of setups ONLY with the same groups has been
useful for me.
Alison Winters
dependsOnMethod= or dependsOnGroup=). It's good to know that it works,
though like you say it doesn't give us a whole lot more than just
explicitly depending on each previous method. I'll keep plugging
away :)
Alison