Massive memory usage in web app using RavenDB

[Mircea Chirea]

I have a web app that regularly is using over 1GB of memory when it is mostly idle. I've taken a few memory dumps and most of this memory is used by Raven.Json.Linq.* objects. Here is a screenshot of the differences between two dumps taken a couple of hours apart:

http://i.imgur.com/e0JdC8e.png

Some interesting things I see in the dumps:

• Most of the memory is used by Dictionary<string, RavenJToken>.
• There's over 3 million RavenJValue objects in memory.
• There's 6000 instances of the restaurant document (we only have 30 in the database), which is the largest document by far.
• SimpleInjector (the IoC container we use) appears to have 223 contexts still alive, each with an async document session attached.

I am not sure what to make of that.

[Oren Eini (Ayende Rahien)]

At a guess, you are still holding to your document session, which is causing a memory leak, since it hold all the entities you are holding.

Do all the memory path points back to the injector?

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mircea Chirea]

Yes, all RavenDB classes are held by the container, directly or indirectly. The fun part is finding out why.

[Oren Eini (Ayende Rahien)]

Out of scope for us, then. Probably you need to configure lifespan policy, or something like that.

[Mircea Chirea]

I'll look into it. It appears to be a circular dependency somewhere (between the scope and controller), which never gets released and thus never releases the sessions.

In the meantime is there a way to make the document session release all references to all documents? It is being properly disposed, but the GC never gets around to collect it.

[Oren Eini (Ayende Rahien)]

You can call session.Advanced.Clear(); But that is a REAL WORKDAROUND and doesn't actually help you.

Disposing != releasing the reference.

[Mircea Chirea]

I know that disposing won't release the reference; but I was under the impression that the document session cleared its cache when disposed.

The real issue is the container keeping references around, but until I find why and fix it, I need workaround to get rid of those sessions, as this occurs only in production as luck would have it.

[Oren Eini (Ayende Rahien)]

No, it doesn't do that. It make no sense to do so because once you dispose the session, there is no point in holding on to it. 

So the GC can clear everything when this happens, it doesn't matter if we remove the references or not.

[Chris Marisic]

What container are you using? Can you post your container setup?

[Mircea Chirea]

SimpleInjector.

The container is set up like this for all per-request dependencies:

container.RegisterMvcAttributeFilterProvider();

container.RegisterMvcControllers(Assembly.GetExecutingAssembly());

container.RegisterPerExecutionContext<IRavenSession, RavenSession>();

(RavenSession is just a wrapper over IAsyncDocumentSession that add some functionality).

Now it gets a little bit more compliated, as the app makes use of both MVC and SignalR.

• For MVC I have subclassed DefaultControllerFactory: https://privatepaste.com/7235a09495
  
• For SingalR I've implemented IHubActivator and have a base class that all hubs inherit from, which disposes of the scope when the hub is diposed by the SignalR infrastructure: https://privatepaste.com/2df289921a

For SignalR that is the only option; pretty crappy: https://github.com/SignalR/SignalR/issues/2908

[Chris Marisic]

Simplest solution might be just pass around the document store and create sessions as needed. Yes that means you lose some advantages of the session per request pattern that some caching might be useful across layers but I also personally view if you need the session cache during a single request you're probably not at a good design anyway. You could theoretically lose some of the benefits of change tracking through layers by not using a single session.

[Mircea Chirea]

That would be one way. In fact I removed the IoC container from the app in question, directly instantiating all dependencies, and it still managed to hold onto a lot of memory after a few hours. I think it's a deeper problem with the fact that SimpleInjector's container doesn't track objects it created by default, hence it cannot dispose of them; you have to use various lifestyles. I was using ExecutionContextLifestyle (which is what is recommended for async/await code) and for some reason the scopes were still kept in memory, thus everything from RavenDB sessions to other dependencies resolved during the scope would be left rotting in memory.

I'm working on replacing the container with Castle Windsor, which can do explicit dependency release (so when the controller is done with the request I can tell it to release all of its dependencies).

Thank you Oren and Chris for you input.

[Oren Eini (Ayende Rahien)]

This has nothing to do with disposing.

It has to do with something (probably the container) holding references to the sessions.

[Mircea Chirea]

Yes; the scope is still kept alive well after it has ended. From the memory dump all references of all objects created by the container trace back to a scope. I don't really need scoping, transient instances are fine - except that SimpleInjector cannot dispose of them, so it can create a controller with the whole dependency graph, but I have to call Dispose on the whole grapth myself. Or I can use a different lifestyle, but the only one that would dispose objects at the end still keeps references to them.

:-/

[Chris Marisic]

I'm going to throw out my plug, use StructureMap it is the best container.

[Kijana Woodard]

Haha. Container wars.

I've used and liked SimpleInjector. The decorator features are slick. I never used it with async though.

I also like autofac.

I also like no container. :-D

[Mircea Chirea]

I have fixed the issue. It wasn't RavenDB's or the IoC container's fault; it was me using HttpResponse.Redirect(url, endResponse: true) in some very old code (to redirect to HTTPS). Never do that, that's incredibly stupid (it writes the headers and aborts the request thread).

[Justin A]

so aborting the requested .. caused what to happen ??

[Oren Eini (Ayende Rahien)]

Because the thread was aborted, cleanup (removing the tracked sessions) didn't happen

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

On Mon, Sep 1, 2014 at 9:11 AM, Justin A <jus...@adler.com.au> wrote:

so aborting the requested .. caused what to happen ??

--

[Mircea Chirea]

Yep. Now the code deep inside MVC had guards for both exceptions being thrown (finally) and the thread being aborted (catch ThreadAbortException), but not when executing filters... which is exactly where I was doing the redirect with end response.
Thus neither SimpleInjector's scope got disposed and dereferenced, nor Castle Windsor's Release method got called, because the world was yanked out from below the code.