RavenDB - High Memory usage and writing to system paging file causes performance degradation.

[kurtic...@gmail.com]

Hello there,

We have two RavenDb instances running in a Master\Slave setup on two machines with 8GB of RAM each. When we started testing a particular operation of our application with larger data sets, we started witnessing a spike in memory usage that causes Raven to start writing to the system paging file, severely degrading performance of the system and affecting other apps on the machines. 

Operation consists of sending around 20000 entities to the application and performing a number of bulk inserts, queries and regular save operations against multiple databases. Bulk insert is used in all cases where we store a large number of documents to the DB. Documents are ranging in size from a few KB to a few hundred KB and a number of exceptional ones (not more than 5) that can be a few MB big. We have around 10 regular indexes and 2 map reduce indexes on the dbs and collections that are affected by this operation.

During the operation ,we get this event logged: 'A significant portion of the database buffer cache has been written out to the system paging file. This may result in severe performance degradation.' Raven Studio is usually not responsive during that time but it eventually recovers.

We tried disabling replication, reducing the MaxIndexOutputsPerDocument to 500 for all the indexes where we use this property( 3 of them) and even disabling indexing completely but we didn't notice any difference in behaviour. 

Here are the paging warning for the event log, server log files, stats and mini dumps from before, during and after the operation:

https://drive.google.com/open?id=0B_d6GYozqfZvSTVwZEZMMWlqbms

All are from the Master instance.

What would be the correct way to handle these spikes of memory usage? Could limiting memory usage of RavenDb help and if yes, what would be the correct way to do it?

Thanks for your time,

Ognjen

[kurtic...@gmail.com]

And yes, we are using build 35183.

[Oren Eini (Ayende Rahien)]

What happen if you try to disable caching? We are caching things by default as they enter, and under load, it may get to a point where the memory usage is high enough to induce paging before we have a chance to evict things. 

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Sat, Aug 19, 2017 at 4:44 PM, <kurtic...@gmail.com> wrote:

And yes, we are using build 35183.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oren Eini (Ayende Rahien)]

Do you have LoadDocument in your indexes? Suggestions?

If you space the inserts out, does this help?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

[kurtic...@gmail.com]

Our stores are initialized in the following way:

var store = new DocumentStore

            {

                Url = url,

                DefaultDatabase = databaseName,

                ApiKey = $"{apiKey}/{apiSecret}",

                EnlistInDistributedTransactions = false

            }.Initialize(true);

            store.Conventions.SaveEnumsAsIntegers = true;

            // disable client-side caching

            store.Conventions.ShouldCacheRequest = _ => false;

            store.DatabaseCommands.DisableAllCaching();

            return store; 

This should disable all caching on the server side?

Regarding the indexes, we dont use LoadDocument. Here is a typical index: 

this.Map = Type1Entities=>

                from Type1Entity in Type1Entities

                from Type2Entity in Type1Entities.Type2Entities

                select new Result

                {

                    SomeProp= Type1Entity .SomeProp,

                    SomeProp= Type1Entity .SomeProp,

                    SomeProp= Type2Entity.SomeProp,

                    SomeProp= Type1Entity .SomeProp

                };

            this.StoreAllFields(FieldStorage.Yes);

            this.MaxIndexOutputsPerDocument = 30000;

The MaxIndexOutputsPerDocument is however not used on all of them and we already tried to lower that to 500.

What do you mean by spacing out the inserts?

Thanks,

Ognjen

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Grisha Kotler]

Hi,

This disables the cache on the client side.

You can set "Raven/CacheDocumentsInMemory" to "false" in the database settings page in order disable server side caching.

https://ravendb.net/docs/article-page/3.5/http/server/configuration/configuration-options

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[kurtic...@gmail.com]

Thanks, 

we 'll try it out together with Raven/MemoryCacheLimitMegabytes and get back to you.

[kurtic...@gmail.com]

Tried out to limit caching memory with Raven/MemoryCacheLimitMegabytes set to 1024 - we still see high memory usage and occasional writes to the system paging file.

We 'll try to disable caching completely to see if that makes any difference.

[kurtic...@gmail.com]

Disabling caching on the server using Raven/CacheDocumentsInMemory  flag solved the issue for us, at least for now. Performance impact was almost non-existent, which is rather strange.