I am encountering an OutOfMemoryException when running large numbers of tests.

[Chris Edwards]

I have a test run of 2000+ unit tests that I am trying to run with SpecRun. If I keep the number of threads below 8, it runs fine, but I see the memory usage climb up consistently. If I use 8 or more, I get an OutOfMemoryException. When it crashes, it is using 1.5gig of ram. When I run with NUnit, memory usage never climbs above a few megabytes. Ideas?

[Gáspár Nagy]

We were able to track down (and hopefully fix) the issue. Altogether it seems to be related to the fact that SpecFlow loads the bindings for each test thread appdomain. If you have quite many step definitions and SpecFlow processes the regular expression for each of these. For some reason (hoping for better performance), we use the "Compiled" option for these regular expression, that increases the memory usage quite much (~5 times more than w/o). I have also measured now, that this regex option does not really give any performance benefit for the kind of regex patterns people use for step definitions, so removing this "Compiled" flag should be the core of the solution here.

I have created a patched version of TechTalk.SpecFlow.dll, that does not have the Compiled regex flag. You can download it from https://dl.dropboxusercontent.com/u/51289/SpecRun/TechTalk.SpecFlow_regex_fix.zip.

Please replace the specflow dll in your packages folder with this one, recompile everything and try again.

I'm looking forward to hear the results!

Br,
Gaspar

[Markus Mayer]

Hi Gaspar,

i have similar problems with random stack-overflow crashes every week or so. Seems to be a memory-issue although the machine has enough of it (4 GB for a 4 thread 600 scenario-Testrun) . My question: Is this patch now part of the most recent SpecFlow? Or is it worth testing it?

Kind regards,
Markus

[Gáspár Nagy]

There were no SpecFlow release since then, so the fix is still only in the patch.

[Markus Mayer]

Hallo Gaspar,

 

since a few weeks SpecRun crashes got very bad. Only seldomly a Testrun manages to get through.

 

Yes - I tried the patch and sorry it didn't help.

 

We've got some 640 szenarios running in 4 threads on a Windows Server 2008 R2 with 4 GB of Memory (it is a virtual Server though running on an ESX).

SpecFlow i using Selenium to automate the browser to test a Webapplication.

 

Some more Infos:

* The same scenarios are running fine single threaded, but of course take a lot more time (too much) to execute.

* We have been experiencing these problems since a year or so but only "once in a while" and when it got bad, it normally helped to reboot the Server. Now this doesn't help anymore.

* Memory Usage on the whole sever is always < 2GB, usually arround 1,4 GB. Memory Usage of SpecRun is between 100 and 200 MB and every automated Firefox uses around 200 MB. So for me there seems to be no Memory issue.

* When I take a look into the SpecRun.log - it just ends in "the middle of a sentence" without any sign of errors or problems before.

* When SpecRun crashes a Windows-Pop-Up ist coming up, offering to take a look at crash details. They look like the following:

 

Problemsignatur:

Problemereignisname: APPCRASH

Anwendungsname: SpecRun.exe

Anwendungsversion: 1.2.0.15

Anwendungszeitstempel: 520e3837

Fehlermodulname: clr.dll

Fehlermodulversion: 4.0.30319.18444

Fehlermodulzeitstempel: 52717f9a

Ausnahmecode: c00000fd

Ausnahmeoffset: 0000000000224f70

Betriebsystemversion: 6.1.7601.2.1.0.400.8

Gebietsschema-ID: 1031

Zusatzinformation 1: 87dd

Zusatzinformation 2: 87dd6cccb79005c6f94a0d9103803118

Zusatzinformation 3: b1e5

Zusatzinformation 4: b1e55aeb6586582f617f1cee2ae9208d

The first 8 lines seem to be invariant with every crash.

 

Is there anything I can do to assist the bugsearch?

* Would it be of any use to try out the new AlphaVersion of SpeFlow+ Runner?

* Should I send you some more Infos?

* Maybe I can start SpecRun with some "Debugging-Option" turned on?

 

In hope of a solution,

Markus

 

PS: We like the new Roadmap of SpecFlow+ (with "Cloud", MVC- and JIRA-Support)