Parallel test execution and scenario re-use

[Edgars Šults]

I have been trying to refactor a Specflow solution to enable parallel test execution with NUnit 3. I have changed all static references to FeatureContext and ScenarioContext in my steps to use injected instances. All was well.

But due to the business requirements our scenarios are quite long and we want to be able to re-use scenarios from other scenarios in order to set up some test data. So we implemented a step that lets us find and invoke other scenario methods via reflection. This is the refactored version:

[Given(@"Test data is prepared by feature '(.*)' and scenario '(.*)'")]
public void ExecuteScenario(string feature, string scenario)
        {
            try
            {
                // Invoke the other scenario in a separate thread so that context isn't shared
                Task.Run(() =>
                {
                    Type type = null;
                    MethodInfo scenarioMethod = null;
                    Utils.GetScenarioRelatedMethod(feature, scenario, out type, out scenarioMethod);
                    var featureInstance = Activator.CreateInstance(type);

                    // Invoke the scenario set-up method
                    var setupMethod = type.GetMethod("FeatureSetup");
                    setupMethod.Invoke(featureInstance, null);

                    // Get the new test runner and copy the values that we want to share to the feature context
                    var testRunner = type
                        .GetField("testRunner", BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.GetField)
                        .GetValue(featureInstance) as ITestRunner;
                    testRunner.FeatureContext.Fill(FeatureContext);

                    // Use the same driver for the new test runner
                    if (!testRunner.FeatureContext.ContainsKey("driver"))
                    {
                        testRunner.FeatureContext.Add("driver", _driver);
                    }

                    // Start the scenario
                    scenarioMethod.Invoke(featureInstance, null);

                    // Copy back updated context values
                    FeatureContext.Fill(testRunner.FeatureContext);
                }).Wait();
            }
            catch (Exception ex)
            {
                throw ex;
            }
        }

Note that the driver of the original scenario is re-used. There is some code in the injected driver class that checks whether the context has a reference to a driver and uses that instead of starting a new one. This works well when test are run serially.

The problem is that when running tests in parallel they start failing as soon as one worker has no more tests to run and finishes. At least that's what I've observed. As soon as that happens there's a NullReferenceException in TestExecutionEngine.GetStepMatch because the FeatureContext is null.

I've tried invoking the DisableSingletonInstance method on all the contexts, thinking that maybe the static references caused some issues, but no. I haven't been able to find what destroys the FeatureContext for other workers when one worker finishes. Any ideas?

[Edgars Šults]

I've given up on NUnit parallel capabilities in conjunction with Specflow. Implemented my own parallel runner that ensures process isolation.

[Andreas Willich]

Hi

About executing whole scenarios from another one:

I am not sure if SpecFlow can this, as this is normally a bad smell when you write scenarios in a BDD style.

The least bad thing that can happen, is that you have hooks executed more than one.

Probably some internal state will get also confused.

About parallel execution:

AFAIK NUnit parallel execution is all done in the same AppDomain.

If you need process/appdomain isolation, have a look at the SpecFlow+Runner (http://specflow.org/plus/runner/). It has this functionality.

Best regards

Andreas

--
You received this message because you are subscribed to the Google Groups "SpecFlow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to specflow+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.