Large number of feature files causes significant lag in starting tests

[Steve Ferguson]

We're running integration tests using SpecFlow 3 (about 950 feature files spread among 20 projects) in a Visual Studio .NET 4.8 solution.  Executing a test from the VS Test Runner, we have a number of developers that experience a significant delay (10+ minutes) after the .feature.cs files are regenerated, but before the tests actually begin executing.  In the interim, VS shows no signs of activity, logs nothing, but shows "Run" as disabled when right-clicking scenarios in the test explorer.

I have tracing enabled in the SpecFlow extension and the time-stamped output stops around the same time the build completes, with no additional output during the window before the tests begin executing.

I have the discover tests in real-time option disabled for testing.

We're using xunit as the test runner.

All of the CPU activity is in the main VS process.  None of the other background processes appears to be doing any work while looking at Task Manager.

Has anyone else seen similar behavior?  Any suggestions on what else I can look into (besides splitting up the solution, which we're working on) in order to determine root cause here?

Regards,

Steve

[Steve Ferguson]

I can add that in the trace output for SpecFlow I'm seeing about 2000 FileChangeEventsListeners registered, one for each .feature file, one for each .feature.cs file, and a handfull of dlls.  Likewise, I see about 1000 GherkinProcessingScheduler "Analyzing request" items queued up on a single thread.

[dip biswas]

I used NUnit test runner enable parallel  test runs on VS 2019 on a multi core workstation, and ran 70+ feature files to test UI for web app. So far no issues. In fact the only issues I had was loading test data at the start of the tests, which was slowing things down and making tests brittle. I was able to negotiate that issue by enabling LAZY loading, so test data is only loaded when needed for scenario. 

--
You received this message because you are subscribed to the Google Groups "SpecFlow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to specflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/specflow/debe9d57-a363-4f9d-9e56-c1bca4eb9586n%40googlegroups.com.

[Darkrod 99]

Really? I hope that you can help to achieve that because I'm trying Nunit with Specflow and Selenium, right now, and is not going well.

[Steve Ferguson]

What seems to be happening is that, due to the large number of feature files in our solution, when we do a build, it seems to rebuild all of them rather than just ones where the .feature.cs file is older than the .feature file.  This triggers all 950 file change listeners to trigger and queue up at once, which overwhelms Visual Studio, causing it to take a long time processing them (at least 30 minutes in one test run, before I killed it and stopped waiting).  Stopping and restarting VS I could then run the test normally.

The test run itself has no trouble executing, once everything is in place.  And running the build and test from the command line without Visual Studio open works fine.

The options I've come up with so far are:

• Create a filtered solution file (.slnf) with the test project plus all of the code projects needed to work on the application
• Revert to using the single file generator

Regards,

Steve

[Andreas Willich]

Hi Steve

Please open an issue for this on GitHub (https://github.com/SpecFlowOSS/SpecFlow). 

Did you try the new 3.4.3 version already? There was a small change, that we write not always the code-behind file to disk. We are only doing it now, when the content changed.

Cheers,

Andi

To view this discussion on the web visit https://groups.google.com/d/msgid/specflow/20127c5c-ecea-4c4a-ace1-8688d94e4b1bn%40googlegroups.com.

[Steve Ferguson]

We are still using 3.3.30 right now.  I"ll try updating to 3.4.3 and see if that helps alleviate the problem.

Regards,

Steve

[Steve Ferguson]

@Andi Is there a link to the change you noted in 3.4.3?  I've updated to 3.4.3 and my testing is showing mixed results.  I'd like to see how closely our situation mirrors the one addressed by the change, which might help me better understand which situations might be improved and which might still run slowly.  I have trace logging enabled for SpecFlow, so if there's something I can look for in the trace log output, that would be helpful.

Regards,

Steve

[Steve Ferguson]

Per your request @Andi, I've opened  https://github.com/SpecFlowOSS/SpecFlow/issues/2109.