Multi-tenant memory issues with RavenDB upgrades

[Ian Cross]

Hi guys,

We just went through the process last night of upgrading a RavenDB server to the latest release (3660) and I wanted to share some observations. These issues have existed in previous RavenDB 3.0 builds as well and I was hoping this one might be different.

What I have observed is that when you upgrade a RavenDB server or import a database dump from an older version, then when you access the database, RavenDB goes through the process of rebuilding the indexes, or something that is very CPU and memory intensive with indexes for a long time. I have no real issue with this and I understand it, but the issue I have is what happens with the memory on the server, especially in a multi-tenant scenario.

During this indexing process, a large amount of memory is allocated and I am guessing, but here is what I think is happening: Based on the default setting for Raven/MemoryLimitForIndexing, Raven allocates 1024 MB of memory to the first tenant (although I think it actually takes more than this). Memory use in the machine starts to grow. Then another tenant is accessed and that also gets 1024 MB and starts competing for processor time for the indexing. Eventually things continue and the machine ends up at 96-97% memory usage and everything on the machine grinds to a halt. It takes 2-3 minutes to even get the Services window open to be able to attempt to stop the RavenDB Windows Service. 

Eventually after several attempts, a single tenant's indexes will be rebuilt ok and that tenant has then been upgraded, but the pain will repeat until all tenants have been through this process. 

By reducing the Raven/MemoryLimitForIndexing setting down to 128 we have been able to delay the pain a while but the big issue I have is that the memory does not ever seem to be reclaimed after a tenant has been upgraded, cpu is back down to 3-5% - memory is still high up at 90% plus and the databases are not even being accessed. I would have thought that whatever memory was allocated for this major indexing task would be released so the machine can get back to normal.

We have also set the Raven/DisableInMemoryIndexing setting to true but that did not seem to help.

For info, at the moment we have about 10 tenants on one of the servers. Each tenant has 60 indexes and there are roughly 10,000 - 20,000 documents per tenant. No map / reduce just map only. That doesn't sound like a lot of load to me, but I may be wrong. We also see this issue happen if we delete all indexes and call the code to recreate them from scratch.

There was talk previously about this being related to a browser memory leak thing that got fixed, but it has been there a while and I suspect it is more to do with the core behaviour of the engine.

If we are doing something wrong, please let me know.

Cheers,

Ian

 

 

 

[Oren Eini (Ayende Rahien)]

The likely problem is that each database start the work at roughly the same time, see that they have free memory available, then they all consume it and run out.

I created an issue for this:

http://issues.hibernatingrhinos.com/issue/RavenDB-3415

The other issue with the memory usage is related to this:

http://issues.hibernatingrhinos.com/issue/RavenDB-3270

This isn't a memory leak per se, the memory isn't lost. But we aren't releasing it back, so it looks the same way.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ian Cross]

Thanks Oren… I have seen your exchange with Arkadiusz on the memory release today. We are using the Changes API in one area because we are trying to get application cache updates triggered between different web server processes. Are you saying that if we avoided using the Changes API completely that the memory would in fact be released? We could probably work with this limitation and get rid of using the Changes API if it would help this issue in the interim?

 

Cheers,

 

Ian

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/gXR99eQD0GQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Please don't do that. It is likely that there are other things that are causing this, and it will be fixed soon.

[Christophe Dubray]

Will http://issues.hibernatingrhinos.com/issue/RavenDB-3415 make it into the next stable?

[Oren Eini (Ayende Rahien)]

No, this is going to be in 3.5

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Fri, May 15, 2015 at 5:17 PM, Christophe Dubray <christop...@gmail.com> wrote:

Will http://issues.hibernatingrhinos.com/issue/RavenDB-3415 make it into the next stable?

--