IntelliJ IDEA Test NG support

[Robert Krüger]

Hi,

could any of the IntelliJ users share their experience with the Test
NG support in that IDE? We are evaluating moving from Eclipse to
IntelliJ for other reasons as well but my growing frustration with the
problems with the Test NG Eclipse plugin and the resulting
inefficiencies in our development cycle may become another factor (the
other alternative to fix that is go back to JUnit as their plugin
AFAIR had fewer problems but that is not really what I want as I think
that Test NG is the better framework of the two but basically I can do
anything I am doing right now with Test NG also with JUnit 4).

Any feedback appreciated.

Robert

[Mark Derricutt]

I'll say that I just LOVE the TestNG support in IntelliJ IDEA. But then
I'm biased as I wrote the original plugin which Jet brains have since
taken over.

I thought I'd covered a lot of bases with the original plugin but things
seem to keep getting evolved and improved.

One thing I added early on which I found a life saver at times was
auto-completion of dataProvider="foo..." to declared data provider
methods, and also an intention/inspection that would find invalid
dataProviders and tell you about them.

There's good support for overriding testng.xml properties from the
run-configuration, rerunning failed tests etc.

The ONE downside which was a contentious issue awhile ago was that due
to the way the plugin was originally developed, the version of TestNG
shipped with the plugin was what was used when running from within
IntelliJ IDEA, rather than whatever version you had on the classpath for
your project ( this caused some headaches when Cedric changed the
JVM->JVM String communication protocol and Jetbrains weren't in the
position to upgrade the version of TestNG they used.

Those problems seem to have all gone away now. I'd suggest downloading
the Community edition of IDEA and just giving it a shot...

Mark

[Robert Krüger]

Thanks a lot for that information!

Do you know if they support rerunning individual tests that got
created by using a data provider? This is currently a major
productivity problem with the eclipse-based setup we have. We have
tests that do stuff like video conversions and then use data providers
to do that with a huge number of permutations of settings and input
files and rerunning individual tests during development is not
possible currently at least not without painful workarounds and I
guess you can imagine that running 100+ video conversions to check if
your code change fixed the problem, is not really an efficient
development cycle.

> --
> You received this message because you are subscribed to the Google Groups
> "testng-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to testng-users...@googlegroups.com.
> To post to this group, send email to testng...@googlegroups.com.
> Visit this group at http://groups.google.com/group/testng-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Cédric Beust ♔]

Hi Robert,

On Wed, May 29, 2013 at 10:53 PM, Robert Krüger <kru...@lesspain.de> wrote:

Thanks a lot for that information!

Do you know if they support rerunning individual tests that got
created by using a data provider? This is currently a major
productivity problem with the eclipse-based setup we have.

You might already know this, but FYI, this feature is supported in TestNG (you can specify in your testng.xml to run only the indices "3 and 5" of your data provider) but not in the Eclipse plug-in.

-- 

Cédric

-- 

Cédric

[Robert Krüger]

Hi Cedric

yes, I know. That's the kind of workaround I mean. The development
cycle I am aiming for is:

1) Developer runs a run configuration for a test suite in Eclipse
(long-running) and can track the progress of that in the UI
2) If individual tests fail, Dev looks at the failing test, changes
code and then rerun's that individual test
3) If test fails again, repeat 2) until it is green
4) Run entire suite again
5) If any tests fail, go back to 1)

Currently the workaround for 2) would be to create a temporary run
configuration with the index (dev would count in the UI and select,
say, test number 47 of 200) and run that then. It's doable and
probably what I will do for as long as I haven't either switched the
IDE, the test framework (I have to say that I have not used this
feature with JUnit Eclipse support, so it might not be there either
but after more than a year of having switched from jUnit to TestNG, I
have to say that the JUnit Eclipse plugin seems much more
stable/mature) or support for this is implemented in the Eclipse
plugin (which I doubt I will do myself).

Disclaimer: I know that everything I get here is free and I am
thankful for the software. I am merely trying to make an informed
decision on how to continue.

[Mark Derricutt]

Cédric Beust ♔ wrote:
> You might already know this, but FYI, this feature is supported in
> TestNG (you can specify in your testng.xml to run only the indices "3
> and 5" of your data provider) but not in the Eclipse plug-in.

Oh really? I didn't know that ;-) AFAIK the Idea plugin doesn't support
this, but now you mention it it's something I've often wanted myself.
The plugin generates testng.xml files to run so it shouldn't be tooooo
hard (*cough*) to add.

I've not checked the source out for awhile to try and hack on it tho.

Mark

[Cédric Beust ♔]

It's easy to see it in action: create a small test with a dataprovider that returns 3 values, throw an exception on the second one and take a look at the testng-failed.xml file that gets generated.

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to testng-users+unsubscribe@googlegroups.com.

[Mark Derricutt]

Cédric Beust ♔ wrote:
> It's easy to see it in action: create a small test with a dataprovider
> that returns 3 values, throw an exception on the second one and take a
> look at the testng-failed.xml file that gets generated.

Ok, so I just making an item in a provider fail - "rerun failed tests"
in IntelliJ reruns the full test, not just the index.

Looks like the plugin is regenerating its own testng.xml file to run
like normal, rather than using the testng generated failures.

Might go raise a ticket for Anna :)

Mark

[Cédric Beust ♔]

Yup :-)

Also, what happens in IDEA if you right click on testng-failed.xml and run that one manually?

-- 

Cédric

Mark

[Mark Derricutt]

Cédric Beust ♔ wrote:

Also, what happens in IDEA if you right click on testng-failed.xml and run that one manually?

Ok, after some playing - by default the TestNG plugin for IntelliJ disables all standard reporters, so testng-failed.xml doesn't get generated, but if you enable default reports from test run configuration screen:



Once you enable "default reporters" from the Listeners tab, you'll see all the standard reports in the test-output directory under your project, or the specified output directory.

>From here, you can add/configure ANOTHER TestNG runner configured to use a suite, and then select the testng-failed.xml file generated from the first test run.

That seems to work a treat.

Mark

[Mark Derricutt]

Amusingly tho, after a few iterations of this I see the output:




Each subsequent run prepends '(failed)' to the test name :)

[Cédric Beust ♔]

Each subsequent run prepends '(failed)' to the test name :)

Yeah, it's a feature. Give me a few hours and I might even be able to come up with a convincing explanation.

-- 

Cédric

[Mark Derricutt]

Cédric Beust ♔ wrote:

Each subsequent run prepends '(failed)' to the test name :)

Yeah, it's a feature. Give me a few hours and I might even be able to come up with a convincing explanation.

A coworker suggests:

> should really do (failed) then (failed last N times)

I like that idea.

Mark