Embedded DB out of memory Exception

[nick]

Every so often we get the following error from our CI build.  This is from our service bus CI build and we have around 540 unit tests that are using an embedded ravendb.  When this happens the machine is not tapped out on memory.  If we just re run the build it usually works like a champ so this isn't a show stopper problem, just supper annoying.  Has anyone seen this before?  BTW we are still on build 701. 

Thanks,

Nick

System.Threading.Tasks.TaskSchedulerException : An exception was thrown by a TaskScheduler. ---- System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.

 at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection) at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Object action, Object state, CancellationToken cancellationToken, TaskScheduler scheduler, TaskCreationOptions options,

 InternalTaskOptions internalOptions, StackCrawlMark& stackMark) at System.Threading.Tasks.TaskFactory.StartNew(Action action, CancellationToken cancellationToken, TaskCreationOptions creationOptions, TaskScheduler scheduler) at Raven.Database.DocumentDatabase.SpinBackgroundWorkers()

 in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 386 at Raven.Client.Embedded.EmbeddableDocumentStore.InitializeInternal() in c:\Builds\RavenDB-Stable\Raven.Client.Embedded\EmbeddableDocumentStore.cs:line 144 at Raven.Client.Document.DocumentStore.Initialize()

 in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\DocumentStore.cs:line 483 at RavenDb.Components.RavenDBConfiguration.InitializeDocumentStore() in e:\TeamCity\buildAgent\work....

[Matt Warren]

See below

On Wednesday, 27 June 2012 16:15:58 UTC+1, nick wrote:

Every so often we get the following error from our CI build.  This is from our service bus CI build and we have around 540 unit tests that are using an embedded ravendb.  When this happens the machine is not tapped out on memory.  If we just re run the build it usually

This is normal behaviour, OOM just means there isn't a large enough chunk of memory or the process has reached its limit. It doesn't mean that there is no memory left. See http://blogs.msdn.com/b/ericlippert/archive/2009/06/08/out-of-memory-does-not-refer-to-physical-memory.aspx for a better explanation

 

works like a champ so this isn't a show stopper problem, just supper annoying.  Has anyone seen this before?  BTW we are still on build 701. 

Do you clear out the database between tests or do you leave all the docs still in there? Also are you running as 32-bit or 64-bit?

[nick]

Each unit test is creating a new embedded database each time it runs.  Currently we have the unit tests running as 32-bit.  We made that changed to allow resharper to run our xunit tests in visual studio.  Without making that change the resharper IDE would give us an error when running xunit tests that used a sql lite database.

Thanks,

Nick

[Oren Eini (Ayende Rahien)]

The 32 bits is probably key, it means that we have just 1.5 GB of virtual space to play with, and because all of the dbs are actually created in the same process, we are probably running out of addresses.

In more recent builds, we have better behavior for running under 32 bits.

[nick]

Changed our CI build to build the unit tests as 64-bit.  Will watch it for awhile and see what happens.

Thanks again for the help,

Nick

[nick]

We switched our builds to 64 and still seeing the problem. Any ideas on other ways to fix this?  Again this is not a show stopper for us just really annoying.  

Thanks,

Nick

On Wednesday, June 27, 2012 1:34:14 PM UTC-6, Oren Eini wrote:

[Matt Warren]

It's strange that you're still seeing it, even when running 64-bit. If you look at the process using Process Explorer, what does it list the image type as, 64-bit or 32-bit? Is there any chance the build process or test runner is 32-bit and so it is forcing Raven to run as 32-bit?

[Oren Eini (Ayende Rahien)]

Is it reproducible? What build are you running?

[nick]

We did check the assemblies and everything is set to 64-bit.  As for reproducible, it happens about 30% of the time.  We are still on build 701, using xunit and running under Team City with 4 build agents. We are planning on upgrading to the latest version of RavenDB this iteration.  Once we are complete with the upgrade and we are still seeing the problem I will circle back to this post.

Thanks,

Nick

[Matt Warren]

When you say "4 build agents" do you mean that 4 unit tests could be running in parallel on 1 machine? If so I guess this is where the large memory usage could come from?

[nick]

Sorry for the confusion.  4 build agents on two build machines.  There is the possibility that two build agents can be running on one machine.  We were seeing this problem when we had one build agent per build server but not as often.

[nick]

Ok we have upgraded to the latest version in our dev environment and so far so good.  I think the real fix was from some helpful debug statements that have been added.  It looks like our unit tests were not disposing the embedded store correctly.  The new version gave us a friendly nudge stating there was an existing instance open and that we might want to dispose it before creating a new one.  

Hopefully this will be the last of that annoying error.  

Once again, thanks for cleaning up our code :).