DataProviders and configfailurepolicy

[ChaotX]

Hi!

I tried to run dataprovider tests when the
configfailurepolicy="continue" was set.
When the @BeforeMethod setup fails I expect not to skip the
forthcoming execution of the same method but after a failure in the
setup() all forthcoming setUp() was skipped for that dataProvider
test.
Am I wrong to expect the above and this a known behavior, or is there
any solution not to skip any of the setup methods when a test method
(fed by a dataprovider) runs?
The same happened in both 6.3 and 6.3.1 versions, so this is not the
cause of the added sysproperty.

[Todd Quessenberry]

I think your expectation is correct, this looks like a bug with that
use case. Unless someone else gets around to it first, I'll try to
take a closer look at it next week, as we run tests in that same
configuration as well (and I feel responsible as the original
submitter of that flag).

Todd

[ChaotX]

Thank you Todd!
This fix is specially important for me to run DataProvider tests
parallel.
If I can help in any manner just let me know!

Feri

[Cédric Beust ♔]

Great, thanks Todd. Let me know if you end up not having the time to do it so I can make sure it doesn't fall off the TODO list.

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[ChaotX]

It seems Todd did not have time, or this thread just lost in the new
year...
I try to keep it warm with this mail, don't hurt me for this
please. :)

On Dec 6 2011, 6:43 pm, Cédric Beust ♔ <ced...@beust.com> wrote:
> Great, thanks Todd. Let me know if you end up not having the time to do it
> so I can make sure it doesn't fall off the TODO list.
>
> --
> Cédric
>

[Todd Quessenberry]

Sorry I didn't follow up in this thread, but I did submit a fix back
in December (https://github.com/cbeust/testng/pull/147). I don't know
if Cédric's had a chance to look at it yet. If you want to give it a
try, you could either pull in my changes from the above link and
compile, or let me know if you'd like me to send you a jar to test.

Todd

[Cédric Beust ♔]

I just merged your patch, thanks Todd!

-- 

Cédric

[ChaotX]

Thank you very much!
Just waiting for a build that contains this. I'm so excited about
it :)

On Jan 26, 12:13 am, Cédric Beust ♔ <ced...@beust.com> wrote:
> I just merged your patch, thanks Todd!
>
> --
> Cédric
>

[Cédric Beust ♔]

Available at http://testng.org/beta

-- 

Cédric

[ChaotX]

Unfortunately does not work for me.

My code looks like:

@Parameters({ "baseHost", "basePort", "baseURL", "browserName",
"browserVersion", "platformName" })
@BeforeMethod(alwaysRun = true)
public void setUp(final Method method, @Optional("") final String
baseHost,
@Optional("") final String basePort,
@Optional("") final String baseURL,
@Optional("") final String browserName,
@Optional("") final String browserVersion,
@Optional("") final String platformName
)
{
Assert.fail("DEBUG FAIL SETUP");
}

@DataProvider(name = "DataProviderTest", parallel = false)
public Object[][] getSampleData()
{
Object[][] sampleData = {{0},{1},{2}};
return sampleData;
}

@Test(dataProvider = "DataProviderTest", groups = { "WIP" })
public void testDataProviderParallel(final Integer data)
{
Reporter.log("Data: " + data, true);
}

It fails the the first setup as expected, but still skips all
forthcoming setUp methods too.

On Jan 30, 7:21 pm, Cédric Beust ♔ <ced...@beust.com> wrote:
> Available athttp://testng.org/beta

[Todd Quessenberry]

Are you sure you have configfailurepolicy="continue" set? If you're
using Eclipse, did you replace its copy of testng.jar with the beta?
I ran your code and got 3 configuration failures and 3 skipped tests
as expected. If you're still seeing it, could you create a small
sample project that reproduces the problem for me?

Todd

[Cédric Beust ♔]

I just updated the Eclipse plug-in, please update to make sure you're using the latest testng.jar.

-- 

Cédric

[ChaotX]

I used ant call to run, but I just found out that the selenium-server-
standalone contained the testng that was hiding the testng.jar I tried
to use.
I tried and I can see the 3 failed setup now.
Thank you very much for your help, and I'm also happy to found out the
classpath issue I got.

On Feb 2, 3:13 am, Cédric Beust ♔ <ced...@beust.com> wrote:
> I just updated the Eclipse plug-in, please update to make sure you're using
> the latest testng.jar.
>
> --
> Cédric
>

[ChaotX]

I updated my project dependencies and eliminated the duplicated testng
occurrences.
I just noticed that when I use "parallel = true" in my above example,
two tests pass and one skips while all setup fails.
Can you confirm this?

I used the following xml to run it:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Work In Progress Tests" verbose="10" parallel="tests"
thread-count="4" data-provider-thread-count="8">
<test name="Work In Progress Tests" parallel="methods" thread-
count="2">
<groups>
<run>
<include name="WIF" />
</run>
</groups>
<packages>
<package name="com.company.tests" />
</packages>
</test>
</suite>

I tested with data-provider-thread-count="2" it works fine, but if the
dataprovider threadcount is >= than the number of argument sets
defined by the dataproveder it behaves weirdly.

[Todd Quessenberry]

You are correct, using it with more threads than invocation counts
results in unexpected behaviors. We're not using threaded tests so it
hasn't been a problem for us. I took a look, but I don't understand
well enough how results are supposed to be shared (or not) between
threads to be able to address this.

Todd

[ChaotX]

Hi!

I just made some tests again and I figured out that not only the dataprovider tests but tests with invocationCount > 1 are also behave strange when executed in parallel with configfailurepolicy=continue.
What I also found is that I got passed test results for tests that had a failed setup configuration regardless of the threadpool size (being > 1 ofc.) in contrast to In my last mail where I got such result only when the thread count was over the number of executed tests.. (it had to be due to the random behavior and lack of luck)
I tried this with the old 6.4beta and the latest 6.5.1 versions too.
In the lates version I also got skipped configurations (not sure if this can not happen with the 6.4b). With configfailurepolicy=skip it worked (sum of threads failed setup, rest skipped setup and all tests skipped).
I also tried to look into the source but like Todd neither me have the insight to fix this in a reasonable time.

Best regards,
Feri

> > > > > > > > > > > > testng-users+unsubscribe@googlegroups.com.

> > > > > > > > > > > > For more options, visit this group at
> > > > > > > > > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > > > > > > > --
> > > > > > > > > You received this message because you are subscribed to the
> > > > Google
> > > > > > > Groups
> > > > > > > > > "testng-users" group.
> > > > > > > > > To post to this group, send email to
> > > > testng...@googlegroups.com.
> > > > > > > > > To unsubscribe from this group, send email to

> > > > > > > > > testng-users+unsubscribe@googlegroups.com.

> > > > > > > > > For more options, visit this group at
> > > > > > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the Google
> > > > Groups
> > > > > > > "testng-users" group.
> > > > > > > To post to this group, send email to testng...@googlegroups.com.
> > > > > > > To unsubscribe from this group, send email to

> > > > > > > testng-users+unsubscribe@googlegroups.com.

> > > > > > > For more options, visit this group at
> > > > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > > --
> > > > You received this message because you are subscribed to the Google Groups
> > > > "testng-users" group.
> > > > To post to this group, send email to testng...@googlegroups.com.
> > > > To unsubscribe from this group, send email to

> > > > testng-users+unsubscribe@googlegroups.com.

[Cédric Beust ♔]

This is getting a bit confusing, can someone create a small test case that shows the issue?

Thanks.

-- 

Cédric

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/wBF5r_EOy_YJ.

To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[ChaotX]

I extended the config failure policy tests with parallel invocations here:
https://github.com/ChaotX/testng/commit/d3d6ca8387dee2610a55452ef2edd8f6ad8c74fc

They failed all the time for me (with different results). I complied it with java 1.7, but I don't think that should be a problem.

[ChaotX]

Hi!

If there is some problem with the linked tests, or I should make a pull request, or anything that helps digging down to the problem please let me know.

Thanks,

Feri

-- 

Cédric

> > > > > > > > > > > > testng-users...@googlegroups.com.

> > > > > > > > > > > > For more options, visit this group at
> > > > > > > > > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > > > > > > > --
> > > > > > > > > You received this message because you are subscribed to the
> > > > Google
> > > > > > > Groups
> > > > > > > > > "testng-users" group.
> > > > > > > > > To post to this group, send email to
> > > > testng...@googlegroups.com.
> > > > > > > > > To unsubscribe from this group, send email to

> > > > > > > > > testng-users...@googlegroups.com.

> > > > > > > > > For more options, visit this group at
> > > > > > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the Google
> > > > Groups
> > > > > > > "testng-users" group.
> > > > > > > To post to this group, send email to testng...@googlegroups.com.
> > > > > > > To unsubscribe from this group, send email to

> > > > > > > testng-users...@googlegroups.com.

> > > > > > > For more options, visit this group at
> > > > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > > --
> > > > You received this message because you are subscribed to the Google Groups
> > > > "testng-users" group.
> > > > To post to this group, send email to testng...@googlegroups.com.
> > > > To unsubscribe from this group, send email to

> > > > testng-users...@googlegroups.com.

> > > > For more options, visit this group at
> > > >http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/wBF5r_EOy_YJ.

To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Cédric Beust ♔]

Hi,

A pull request would be great, thanks!

-- 

Cédric

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/JAHZdYG5ROQJ.