getting more info from testNG with the eclipse plugin?

[Ed]

I am using the testNG in eclipse, both the latest releases.

I would like to perform additional actions when I detect that my test
has failed in a easy friendly way.

I use the @BeforeMethod and @AfterMethod annotations to get informed
when a test stopped/failed, but I need to know if if failed.
I know you can add a ITestListener to retrieve this information and
can also register it in the Eclipse Preferences for all tests. But
that's not very friendly and to global.

I am looking for a more easy way, such that when I right click the
method and run it as a testNG test, the listener is automatically
detected and used.
I looked in the forum and doc's but can't really find something like
this.
Is there something like this?

I was thinking about something like that: annotate the test class and
specify the listener that you want to use. The eclipse plugin will
automatically pick up this annotation, and add the listener to the
test runner.

I in the same way I would love to be able to specify other jvm
arguments like the memory usage? I have quite some tests that need
more memory. In maven they run ok, as it's configured in Maven, but I
let the testNG eclipse plugin create my test configuration when first
runing it, which work very nice and friendly.

I think it would be nice to be able to add JVM arguments like the
listener and memory usage through an annotation.

Please feedback on this?
Ed

[Cédric Beust ♔]

Hi Ed,

That's an interesting problem.  A few quick thoughts.

My first reaction was that when you right click on a class/method, the plug-in could just pick the listener information from the testng.xml, but then I wondered "which one?" since there can be several.

We could consider a special file called "testng-eclipse.xml" that the plug-in would always use to configure the temporary runner.

We could also look at annotations, which could be either part of TestNG or only part of the Eclipse plug-in (my preference right now after a few minutes of thinking).  The downside of annotations is that we might end up duplicating a lot of the information that's already well described in the XML file and also that these annotations will be local to the Java file your are running.

What do you think of these ideas so far?

-- 

Cedric

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--

Cédric

[ed bras]

Hi Cedric,

What about offer them both?
- eclipse-testng.xml is ok, but maybe to global if you want to use different settings for a certain test class.
- annotation might result in many duplications, like you mention.

Offering both would result in eclipse-testng.xml containing the default settings that can be overridden by an annotation. You could also consider defining the eclipse-testng.xml files through an annoation overriding the default file.....

Problem scope: able to define test settings for the temporarily tests run with the eclipse plugin. Example: jvm and testNG arguments.

2009/12/9 Cédric Beust ♔ <cbe...@google.com>

[Ed]

Extra note of an example of injecting settings:
I use @BeforeSuite to pass in an optional Url parameter.
This parameter has to be optional such that it's called through the
eclipse plugin with a null value (without complaints)...
If the passed in parameter is null, I will retrieve the parameter from
an annotation on the concrete test class.
The @BeforeSuite is defined in a common super class.

My goal: testers can easily make common changes in a friendly way...
In this case: through an annotation....

[François Reynaud]

the eclipse-testng.xml for global param and settings when the plugin is used would be perfect for me.
Mixing annotation + xml may lead to some confusion when people move from eclipse to hudson for instance for the tests.

To support multiple scenarios, maybe it would be possible to have a check box in the testNG plugin settings of the project, something like "prompt me for eclipse-testng.xml to be used" where users could choose between eclipse-seetingsXX.xml, eclispe-settingsB.xml etc.

thanks
François

[Ed]

> Mixing annotation + xml may lead to some confusion when people move from
> eclipse to hudson for instance for the tests.

Yep, that't true, but to support desired features/complex situations
that aren't that uncommon, you have always have this risk. The easy
way to deal with this is to set common defaults that 90% of the people
use so they aren't bothered with the complex features. But for people
that want to make complex configuration, they need these features.

Only use the xml files, might be a bit too restrictive.
For example: in my case, I use Maven (triggered by Hudson) to run my
testNG tests through the testNG xml files. So the testNG xml files
contain all the necessary things.
However, some tests, and only those tests, need a testNG listener to
be informed about failures after a test method has run (in the after
method). Adding this listener to a global testNG xml file would be no
option as it's too global. However, I want to run these tests in a
lazy-user-friendly way in eclipse, by simple clicking it (otherwise it
get's confusing to the rest of the team. Another catch: this listener
must be used when running one test that needs it, but also like X
tests that need this listener. It just depends on the tests that you
need to do your work, like fixing a bug.
I think that the only "correct" way to indicate the usage of the
listener is through annotations. You can add the annotation to every
test or through a super class of all tests that use the listener...
The listener will be picked up by the testNG Eclipse plugin and by the
testNG runner that is triggered by Maven, so I don't think this causes
a lot of duplication.
Again: I think using one global testNG xml file is great, but as you
see also a bit too global and no always very useful. Especially if I
look at my Eclipse environment: having about 20 eclipse projects and
only one project needs a certain listener...

Any other idea's are more than welcome

[Ed]

BTW: is there an easy way to find out in the method marked with
@AfterMethod, to find out if a test was sucessfull or not ?

[Ed]

Cederic,

Sorry, I don't want to sound too rude, and you probably are busy
implementing tons of things that are much more important. But, just
curious, any idea when something like this will be available ?

Mabye it's nice to describe my use case :)
I run my selenium test cases with testNG (even do Selenium as a Junit
extension, I just love testNG :) )... and when a test fails, I need to
make a screenshot of the current browser screen. I now simple always
make a screenshot in the method that is marked with @AfterMethod,
which works fine, but I do like to fine tune this one time in the
future ...(again: it works fine for now).. BTW: I save the screenshots
with the same name as the test method beeing called as that is
available in the method marked with @BeforeMethod
The testers aren't "real" developers. They run their test through
Eclipse with the testNG eclipse plugin which must work very dumm-user-
friendly..

[Lidia Marchioni]

Just wanted to chime in that this would be a very usefull feature for me as well (also running Selenium from TestNG and in need to easily detect test failure at @AfterMethod level...).

Lidia

[Cédric Beust ♔]

Hi Ed,

On Tue, Dec 15, 2009 at 6:16 AM, Ed <post2...@gmail.com> wrote:

Cederic,

Sorry, I don't want to sound too rude, and you probably are busy
implementing tons of things that are much more important. But, just
curious, any idea when something like this will be available ?

Well, the good news is that you (and a few others) convinced me that it was a useful feature to add to the plug-in so I would be happy to put it at the top of my TODO list.

The bad news (for you :-)) is that my wife just gave birth to baby twins so my hacking time is going to be severely limited in the near future, so I can't really promise you anything right now...  Really sorry about this, but I'll do my best.

--

Cédric

[ed bras]

Congratulations, that's good news.
Don't worry about it, it's just a piece of software, I am fine for now :)
If I would have the time, I would be more than happy to change it and contribute it, but I am very busy setting up my new company and already have such big delays :(... (I know....: bad excuus.. sorry)...
Enjoy the time together

[Cédric Beust ♔]

Hi everyone,

I found some time to implement this feature (to my wife's dismay :-)).

There is now a new section "Use an XML template file" in the TestNG preferences, see the attached screen shot.  Check the box if you want to use that feature and pick a TestNG XML file.

Next time you launch a Debug/Run configuration that is not an XML suite (class, package, method or group), the plug-in will pick the settings found in that XML file and it will use them to launch that configuration.  The values picked are:

• All the attributes of the <suite> tag (name, parallel, thread-pool-size, etc...).
• The following children of the <suite> tag:  <listeners>, <parameters> and <method-selectors>.

Please update your plug-in to 5.11.0.27 at the usual location (http://beust.com/eclipse) and let me know how it works for you.

-- 

Cédric

2009/12/16 Cédric Beust ♔ <cbe...@google.com>

[ed bras]

Hi Cederic,

Thanks for the change, Yep notice the preference option.

The problem I am having with this change is that it's too global for me. I have many tests that I run within Eclipse (a few hunderd that I run as a package/group)... But I have only about 20 tests that I need a listener for and also about a hunderd that I need to give extra memory during there run. I don't want other tests to bother with this global settings.

I think we can solve this through Annotations like explained in my post. What do you think how I can solve this?
The other advantage of using annotation is that people don't need to change there eclipse configuration....

I hope I don't sound too negative about the change :(

Ed

[Cédric Beust ♔]

Yes, this occurred to me as well.

Would it help if that setting was available as a launch configuration option instead of (or on top of) a global preference?

-- 

Cédric

On Thu, Dec 31, 2009 at 2:11 AM, ed bras <post2...@gmail.com> wrote:

Hi Cederic,

Thanks for the change, Yep notice the preference option.

The problem I am having with this change is that it's too global for me. I have many tests that I run within Eclipse (a few hunderd that I run as a package/group)... But I have only about 20 tests that I need a listener for and also about a hunderd that I need to give extra memory during there run. I don't want other tests to bother with this global settings.

I think we can solve this through Annotations like explained in my post. What do you think how I can asolve this?

The other advantage of using annotation is that people don't need to change there eclipse configuration....

I hope I don't sound too negative about the change :(

Ed

[ed bras]

Could you give an example please?

2009/12/31 Cédric Beust ♔ <cbe...@google.com>

[Cédric Beust ♔]

You would just define several launch configurations in Eclipse and each could use a different XML template file.  In your case, it looks like you would have three:  one with no file, another with a file that defines a few reporters and a third one with the extra memory (which you probably already have since it's a JVM option?).

I'm still thinking about your annotation suggestion, can you give me an example of what you'd like to see?

-- 

Cedric

Cédric

[ed bras]

Let me think about it... I have to sport a bit now an celebrate new year ave ;)... Otherwise my girlfriend might kick my ars... :)...
You probably recognize that hihihi...

Have good night (here its 6 pm).

2009/12/31 Cédric Beust ♔ <cbe...@google.com>

[ajay mehra]

I wish a very Happy n Prosperous New Year 2010 to all TestNG community members.

[Ed]

I think the idea would be to model all or part of the testNG.xml file
through one ore more annotations.

When running a test, the testNG test runner will then scan the class
and all super classes of the test method for these annotation(s) and
will use them in preference of the testNG xml file. And the
annotations in the subclasses can override annotated value's in super
classes. This latter overriding feature I use a lot during my tests
and works nicely as you can configure global tests settings in super
classes and subclasses can override parts or all of these settings.

These settings are those that I need at this moment:
- Listener to get informed about test result.
- Setting JVM parameters.

For me, setting these things through annotations, is the best way I
came up with. Through configurations files is too global and needs an
extra configuration step in your development environment that I don't
like. I mean: a new developer can easily forget to set these
configuration settings in his testNG preference and then it doesn't
work. With annotations, the test runner will automatically search for
them and use them.. No extra configuration needed.

Examples:
@TestNG(listener="com.bla.ListenerTestNG", JvmArgs="-Xms512m -
Xmx756m")

Or maybe it's better to split them up:
@Listener("com.bla.ListenerTestNG")
@JvmArgs("-Xms512m -Xmx756m")

I think I prefer this latter one (splitted up). To my experience this
works better due to the syntact/default value nature of annotation. I
started using annotation putting everyting in one, but nowedays prefer
using more small annotations...

What do you think?

[Cédric Beust ♔]

Hi Ed,

I don't really understand how having the JVM parameters in annotations help:  by the time TestNG parses these annotations, the JVM is already running, so how would you use these parameters?

I still think that these two pieces of information are better described in the Eclipse launch configuration:  when you are about to run a TestNG, this is where you configure everything, what groups/classes/XML file to run, the parameters to the JVM, etc...

So I was thinking of adding the same setting that I added globally to launch configurations so that you can override the template XML file there, I think it would solve your problems completely.

What do you think?

-- 

Cedric

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--

Cédric

[ed bras]

> I don't really understand how having the JVM parameters in annotations help:
> by the time TestNG parses these annotations, the JVM is already running, so
> how would you use these parameters?

Interesting... I don't know exactly how the plugin works and how it
starts the tests. So the plugin does run the test directly but also
create a launch configuration?...
I hoped it would be something like: create the launch configuration
and tell Eclipse to run it. So the Annotation would just add the jvm
arguments to the launch configuration.... Is it possible through the
plugin to run tests like this: create launch config and tell Eclpse to
run it..?
What is the best way to add these jvm parameters then ?

> I still think that these two pieces of information are better described in
> the Eclipse launch configuration: when you are about to run a TestNG, this

Euuhhhh... I sometimes have to remove my testNG launch configurations
as there are about a hunderd of them in Eclipse. So as you can
imagine, configuring all these launch configurations to meet the
requirements doesn't really make me happy....

> So I was thinking of adding the same setting that I added globally to launch
> configurations so that you can override the template XML file there, I think
> it would solve your problems completely.

Yes, but that requires an extra step to run a test (in required
flexible way): to configure the launch configuration which I don't
want.
++++ Let go back to the original problem: launching a test in a user
friendly way that uses "common" settings.
---- By user friendly I mean: right clicking a method and running it,
and the common settings that are required to run the test correctly
are automatically used without performing an extra step.
What I understand from your post: the tester that runs the tests, has
to perform an extra step: modifying the launch configuration to meet
the required settings such that the test is run correctly. If he
launches 2 tests, or the failed tests, or the all tests in one
package, or all tests in one class, he has to modify all these
automatically created launch configuration. In our projects, that can
lead to duplicate modifications of maybe hundred launch configurations
:(
Even if the tester has knowledge of doing this: you can understand
that this is not an option.
Secondly, a tester and developer, that use both Eclipse have different
development knowledge and it's not acceptable to assume that a tester
knows anything about Eclipse launch configurations. They know who
configure the scripts, right click the method/class/package and run
the test....That's about it.. And I think is also the desired
situation. Why bother them with unnecessary things.

--- By common settings I mean: define commons settings per class at
least. This is exactly what you can do with an annotation and it keeps
it all together. These global settings are too common to use them for
all tests..

My point: how to define common settings for a "group" of test methods
such that the are automatically picked up when running a test through
the context menu (without the extra step)...

What could be an idea is adding an extra group and that this group
contains additional settings. But I am not sure if a group is suitable
for that.

I donnoooo...

How I run a test now, that needs extra jvm param's? Through the
context menu: run the test and stop it directly. Then go to the Run
configuration and change the jvm params of the just automatically
created test. Ofcourse I only do this once, but when I upgrade my
eclipse, or remove all my testNG launch configurations, I have to
start alllll over again... And believe me: that doesn't really make me
happy...
This doesn't sound like a healthy way of running your tests (method,
class, failed tests, package, etc...)
How can I improve this?

To keep it flexible: maybe it might be an idea to define an
interceptor through an annotation that will receive the test runner as
argument and in this way is offered to do what ever he wants with it.
For example: adding a listener such that you are informed if a test
fails or not... If the interceptor annotation is encountered, it will
use it, otherwise not of course...

Ed

>> testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

>> .
>> For more options, visit this group at
>> http://groups.google.com/group/testng-users?hl=en.
>>
>>
>>
>
>
> --

> ***Cédric
> *

[Ed]

Hi Cediric,

Just curious if you got time to look at my last post ?

[Cédric Beust ♔]

Hi Ed,

Not yet.  I'm on paternity leave right now with two newborn twins and a laptop that just died on me so it might be a while...

-- 

Cedric

On Thu, Jan 7, 2010 at 10:10 AM, Ed <post2...@gmail.com> wrote:

Hi Cediric,

Just curious if you got time to look at my last post ?

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--

Cédric

[Adrian Mitev]

Wish you to be a happy father and to raise a healthy kids.

2010/1/7 Cédric Beust ♔ <cbe...@google.com>:

--
Although nobody can come back and make a new start, anyone can start
now and make a new end

[ed bras]

Ok, good luck :)...

Don't worry, we have time, was just afraid you might had missed the post...

[ed bras]

Oeppsss same here btw: congratulationsssss.... :)

2010/1/7 Cédric Beust ♔ <cbe...@google.com>