Debugging Parallelized TestFixtures

[Jeremy Dunn]

I've noticed that when using the VS test adapter, when debugging a parallelized TF, my tests are run in series. My code looks something like this:

[TestFixtureSource(typeof(UITestsSource)), Parallelizable]

public class TF_UITest1 : UITest
{
    public void GuestCheckout1Apparel1Shoe()
    {
        //do test
    }
}

If I just run the tests without debugging(both in the test adapter or via the console app) they all run as expected in parallel, but when selecting 'Debug Selected Tests' in the test adapter, it forces them into series. Is this intentional? Is there a configuration I'm missing to allow debugging in this manner, or is it intentionally disallowed?

[Charlie Poole]

Do you see this only when debugging selected tests or also if you run
selected tests without debugging? Does it appear even if you select
tests from multiple fixtures? (Remember, NUnit does not yet implement
parallel execution of individual test methods within a fixture)

> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nunit-discus...@googlegroups.com.
> To post to this group, send email to nunit-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/nunit-discuss.
> For more options, visit https://groups.google.com/d/optout.

[Greg Young]

To be fair I would consider running serially when a debugger is
attached a feature in most cases :)

--
Studying for the Turing test

[Rob Prouse]

Charlie, I thought this was intentional. Didn't we change the adapter to run the tests non-parallel when debugging because people were complaining about stepping through their tests when they were running parallel and it jumping all over their code as the threads switched?

--

Rob Prouse

 

I welcome VSRE emails. Learn more at http://vsre.info/

[Charlie Poole]

Rob, that does sound familiar... yup, I see the code.

So it's a feature! :-)

[Charlie Poole]

To complete the explanation: we don't do this in the console runner
because the user has the ability to control it through the command
line. That option isn't available for the adapter, so we decided to
make it automatic.

[Jeremy Dunn]

If I run the tests without debugging, they run in parallel as expected, in our code we've got a 1:1 relationship between fixture and test specifically to overcome individual tests not running in parallel. Selecting multiple fixtures and debugging also results in serial execution.

I can absolutely see running serially with debugger attached as a feature in a lot of use cases, but, at least in my case, there would be a ton of value in being able to choose whether or not it will ignore parallelization. Serial is great when debugging a specific test, but in the parallel world it makes hunting down bugs relating to thread-safe objects and locking transactional SQL much, much harder.

[Greg Young]

Then run through console and attach debugger. You can even make a
project that when you hit f5 runs nunit-console and attaches debugger
to it.

For debugging tests you almost never want this behaviour.

[Charlie Poole]

I agree with "almost never."

We discussed having the --debug option of the console automatically
turn off parallel execution but decided to keep them separate to allow
for those rare times. For console debugging, I generally use --debug
--workers=0

[Jeremy Dunn]

Unfortunately your almost never == ~1/3 of the time for me; alas, different worlds :). I'd argue that single threading a suite that is designed to run across many threads is sort of un-intuitive, and that users who run into issues when debugging multi-threaded code should be able to utilize the thread handling that is built into VS(specifically- assigning a breakpoint a threadID), rather than disabling the capability wholesale.

All that being said, it doesn't sound like I'm changing any minds. While somewhat cumbersome, the workaround is very helpful to have. As always, thanks for the advice!

[Greg Young]

Also a step debugger is slightly above useless debugging most threading issues.

Most people will be writing tests that can be run in parallel but want
to debug serially (they have bohrbugs in tests not heisenbugs).

If dealing with heisenbugs in tests I think the debugging the
nunit-console is best

[Charlie Poole]

Hi Jeremy,

I think you may have misunderstood some of the comments. What we are
talking about disabling is the parallelism that NUnit itself adds for
efficient test execution. Any thread switching etc. that is built into
your application and tests will continue to operate. In fact, if I am
debugging a multi-threaded application, I definitely want to disable
NUnit's running of unrelated threads on separate threads while I do
it.

Charlie

[Jeremy Dunn]

I don't think I've misunderstood, I'm definitely talking about the NUnit implementation of parallelism; I think this stems from how I'm using NUnit as a driver of functional Selenium tests vs. what it was designed to do as a discreet unit testing solution. We have a couple layers built on top of NUnit to support advanced reporting and distribution a load of Selenium tests across local and cloud hosted VMs. In most cases, NUnit is driving those layers. When implementing features on those layers it isn't uncommon(definitely not almost never) to need to debug the suite by running a few dozen(or hundred) tests, and then hitting a break point to inspect the state of the suite. Running in parallel is not only for convenience of not having to wait for a large number of tests to complete, but also to ensure the layers above NUnit are performing in a thread-safe manner. Maybe I'm not using your tool in the manner it was intended, I'm not totally sure at this point, but here I am :). 

It boils down to a manner of convenience- is it more convenient to debug in parallel or series by default? For me the answer is that the option for either is important. I'll concede that it sounds like I'm in the minority with ever needing to debug in a parallel manner, and perhaps that is because I'm using NUnit in an unintended way. From my perspective- there would be a good amount of value(most definitely slightly above useless) in the ability to tell the test adapter whether or not I want it to run my parallel tests in series; I don't imagine I'm alone in that if there are others out there using NUnit as a driver of Selenium tests.

I'm wholly unfamiliar with how it's been implemented, but it sounds like this is something I might be able to enable in my own extension of the VS adapter?

I hope I'm not coming off as standoff-ish, I just want to provide perspective for how I'm using NUnit. As I said before, I can get by just attaching to the console app, and I'm very happy that I at least have that workaround; it's just a little cumbersome and not as friendly to devs with less experience in VS.

[Greg Young]

I hope I'm not coming off as standoff-ish, I just want to provide
perspective for how I'm using NUnit. As I said before, I can get by
just attaching to the console app, and I'm very happy that I at least
have that workaround; it's just a little cumbersome and not as
friendly to devs with less experience in VS.

You can also set your project to run nunit-console.exe your.dll then
just use f5 debugging as opposed to attaching.

Cheers,

Greg

[Rob Prouse]

Jeremy,

I think that is reasonable to provide the option to run in parallel even when debugging in the adapter. The default will remain as it is, but we could accommodate your needs with a setting. Would you enter an issue requesting it? https://github.com/nunit/nunit3-vs-adapter/issues

Rob

[Charlie Poole]

Well, it sounds like there's one thing that wasn't understood -
probably my fault at that: I'm saying, along with you, that both
options have to be available.

And they are! That's why the --debug option does **not** (for example)
turn off parallel execution.

For my own debugging, there are different cases:

In the "standard" case, I want only one thread running, because it's
just one thing to look at.

In the case of a threaded application, I want all those application
threads to keep running but be paused (which is what many debuggers to
anyway) but I still don't want any extra threads created by NUnit in
my face.

In debugging NUnit itself, I want all the threads operating. That's a
special case of the one above, since NUnit is my threaded application.

It sounds like your situation is closer to the last case.

As close as I can tell, we have all the facilities you are asking
about. The difficulty is that you are asking them to be selectable
from within the VS Adapter. That's not currently possible because we
don't have a general way to tell NUnit what settings to use when
running under the adapter. We have hacked in a few optons using the
registry, but that's really not a good long term approach. See this
issue for a discussion:
https://github.com/nunit/nunit3-vs-adapter/issues/49

Rather than extending the adapter, you might consider becoming a contributor!

Meanwhile, as others have said. Running your tests under the console
is your best option for those cases where you need such a level of
control.

Charlie

[Jeremy Dunn]

I went ahead and created https://github.com/nunit/nunit3-vs-adapter/issues/92. Thanks again for the help all.

[Charlie Poole]

That won't hurt.

Most likely, when we implement issue 49 we will probably expose all
the existing NUnit package settings, which will give the same degree
of control that the console has.