NUnit - Repeat test case for atleast 3 times, to avoid false alarm

[kumar]

Hi,

I have few test cases for Web Site UI Automation.

I want to try to run my test case at least three times, if it fails
for first and second time. I am doing this to avoid false alarm.

Please let me know, if we have any option to use in NUnit. I am using
C# with NUnit.

Thanks, Kumar.

[Charlie Poole]

Hi Kumar,

NUnit has a RepeatAttribute, but that will repeat the test n times
or until it fails. Do folks think we should add an attribute like
'Retry' that does
the opposite?

Meanwhile, your best recourse is to use a loop in your test itself.

Charlie

> --
> You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
> To post to this group, send email to nunit-...@googlegroups.com.
> To unsubscribe from this group, send email to nunit-discus...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nunit-discuss?hl=en.
>
>

[prabhu kumar]

Hi Charlie,

Thanks for your quick response. 

Yes. It will be really very useful, If you guys can attribute like 'Retry'. For now, I am using C# looping.

But, If there is any chance of adding 'Retry', that would be great. Please let me know, If you can just add that attribute and provide us some private build.

Regards,

Prabhu.

[Charlie Poole]

Hi All,

What do others think about this idea, i.e.:

An attribute [Retry(int numTimes)] that would cause a test to be rerun up
to the specified number of times until it passes.

Alternatively, we could add some sort of property to the existing
RepeatAttribute
to make it work in that way, but I'm more inclined toward a separate attribute
if we do it at all.

Charlie

[Frank Schwieterman]

It sounds like a test smell. I think its better to let it be
smelly, rather than provide direct support in the test framework. I
don't mean to be insulting, plenty of my tests smell at some time or
another.

[Alona]

RetryAttribute or property to RepeatAttribute would be very useful for
me also.
And I think that it should be separate attribute. You can use it not
only for repeat the same test few times but for test with
ValuesAttribute or RangeAttribute that you want that at least one of
the cases will succeed (I don't know where you can use it in a real
life, but theoretically it's nice :-))

On Aug 11, 1:49 am, Charlie Poole <nunit...@gmail.com> wrote:
> Hi All,
>
> What do others think about this idea, i.e.:
>
> An attribute [Retry(int numTimes)] that would cause a test to be rerun up
> to the specified number of times until it passes.
>
> Alternatively, we could add some sort of property to the existing
> RepeatAttribute
> to make it work in that way, but I'm more inclined toward a separate attribute
> if we do it at all.
>
> Charlie
>
>
>
>
>
>
>
> On Wed, Aug 10, 2011 at 3:31 PM, prabhu kumar <prabhu10...@gmail.com> wrote:
> > Hi Charlie,
> > Thanks for your quick response.
> > Yes. It will be really very useful, If you guys can attribute like 'Retry'.
> > For now, I am using C# looping.
> > But, If there is any chance of adding 'Retry', that would be great. Please
> > let me know, If you can just add that attribute and provide us some private
> > build.
> > Regards,
> > Prabhu.
>

> > On Wed, Aug 10, 2011 at 12:36 PM, Charlie Poole <nunit...@gmail.com> wrote:
>
> >> Hi Kumar,
>
> >> NUnit has a RepeatAttribute, but that will repeat the test n times
> >> or until it fails. Do folks think we should add an attribute like
> >> 'Retry' that does
> >> the opposite?
>
> >> Meanwhile, your best recourse is to use a loop in your test itself.
>
> >> Charlie
>

[John Brett]

I'm afraid I agree with Frank. If the test 'sometimes passes' then I
fail to see the value in it.

I know it can be a pain (particularly with the more integration-level
testing), but the value
of the test is based upon its determinism, not statistical analysis of
its probability of giving
useful results.

Far too often I've seen failing tests ignored completely because "Oh,
they sometimes fail - it's
just a timing thing" only for a real bug to be overlooked. Not worth
it, IMHO.

John

[Brad Stiles]

It depends on why it "sometimes" fails. If it's a dependency on an
intermittent resource, then that resource should likely be mocked for
the purpose of most of the tests, and there should be a separate test
for dealing with the intermittent nature of the resource.

Off the top of my head, I can't see any reason for ignore a
"sometimes" failing test that doesn't have the potential for trouble
down the road, as John suggests below.

I'm not going to argue against the inclusion of such functionality, I
just can't see a reason to use it.

[kumar]

Retry attribute really helps when we are dealing with UI Automations.
Sometimes, UI Automations are not consistent and throws false alarm.

We just want to avoid that and try for more than once and see the
consistent failure.

When we are dealing with Web Site UI Automation, There are cases like
network issues, Server problems, Cache, Cookie issues, etc.,
We want to double check the issue, before raising the alarm.

Thanks,
Kumar.

On Aug 11, 5:19 am, Brad Stiles <bradley.sti...@gmail.com> wrote:
> It depends on why it "sometimes" fails.  If it's a dependency on an
> intermittent resource, then that resource should likely be mocked for
> the purpose of most of the tests, and there should be a separatetest
> for dealing with the intermittent nature of the resource.
>
> Off the top of my head, I can't see any reason for ignore a

> "sometimes" failingtestthat doesn't have the potential for trouble

> down the road, as John suggests below.
>
> I'm not going to argue against the inclusion of such functionality, I
> just can't see a reason to use it.
>
>
>

> On Thu, Aug 11, 2011 at 5:23 AM, John Brett <john.oz.br...@gmail.com> wrote:
> > I'm afraid I agree with Frank. If thetest'sometimes passes' then I

> > fail to see the value in it.
>
> > I know it can be a pain (particularly with the more integration-level
> > testing), but the value

> > of thetestis based upon its determinism, not statistical analysis of

> > its probability of giving
> > useful results.
>
> > Far too often I've seen failing tests ignored completely because "Oh,
> > they sometimes fail - it's
> > just a timing thing" only for a real bug to be overlooked. Not worth
> > it, IMHO.
>
> > John
>
> >> Hi All,
>
> >> What do others think about this idea, i.e.:
>

> >> An attribute [Retry(int numTimes)] that would cause atestto be rerun up
> >> to the specified number oftimesuntil it passes.

[Roman Proszynski]

Sounds like you need to have more reliable hooks in the UI to be
certain of its state.

Ex, rather than expecting to instantly navigate between form/page,
have some function in UI to notify when loading UI is finished.

Also, if you have any error popup messages, likewise have some event
driven way to handle them...

> > > For more options, visit this group athttp://groups.google.com/group/nunit-discuss?hl=en.- Hide quoted text -
>
> - Show quoted text -

[Ken Johnson]

Any update on this? While running tests using NUnit, Selenium on TeamCity for continuous integration, being able to re-run tests that are known to be flaky would be quite helpful. The flaky tests seem to be ignoring .click() very infrequently. Of course with a large test suite in CI, even a failure of this nature .05% of the time is unacceptable.

Thanks,

Ken 

[Charlie Poole]

Hi Ken,

There was a fair amount of discussion about this on the list, mostly
negative. Quoting myself, as of February 15:

> Because of the lack of enthusiasm for the feature, which is clear
> in most of the replies to the earlier discussion, this was not done
> for NUnit 2.6. I'll reconsider how Repeat/Retry might work in the
> new 3.0 release.

A good way to ensure this doesn't get lost will be for somebody to
submit a feature request at http://bugs.launchpad.net/nunit-3.0 :-)

Charlie

> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/nunit-discuss/-/aDFD4SnSL_QJ.

[Sylvain Perron]

Hi,

I know it's been a while since this discussion started but I happened to need the same and so I wrote a NUnit plugin that will do it. It can be found on NuGet or on GitHub:

NuGet: https://www.nuget.org/packages/NUnit-retry/

GitHub: https://github.com/slvnperron/NUnit-retry

Please let me know if this solves your problem