Possible memory leak?

[Johan Börjesson]

Hi!

We are currently running a test in our lab environment. We have two identical setups where the only difference is the version of Raven DB.

Short description of the test bench: 

- Host 1: A Simulator saving data in Raven DB. 

- Host 2: Hosting Raven DB

- Windows Server 2012 

- Intel Core i7-2660 @ 3.40 GHz

- 16 GB RAM

- C Drive 500 Gb rotating disk

- E Drive 1 TB SSD (Samsung 850 Evo) <-- This is were RavenDB is installed and also where the database (data + indexes) is located.

- Host 3: Client querying Raven DB

Setup 1: Raven DB version 3.0.30115.

Setup 2: Raven DB version 3.5.4 build 35247.

In both setups we have set the parameter "CacheDocumentsInMemory" = "false". 

Both test setups the simulator is generating the same amount of data (about 100 docs per second with a size of 3 - 10 KB each), and the client is querying for the same amount of data.

If we let these two system run for a week or so we can see that the older version 30115 is perfectly stable at around 4 GB of memory usage, while the newer version 3.5.4 build 35247 has used up all system memory (15+ GB) and finally stopped working.

We have tried to disable the document cache but it seems that did not help. 

We created a debug package from the web interface and can confirm that the memory usage is very high, but can we somehow find out what exactly this is? 

  "TotalNumberOfRequests": 9966241,

  "Uptime": "11.21:11:26.8475037",

  "Memory": {

    "DatabaseCacheSizeInMB": 2.54,

    "ManagedMemorySizeInMB": 16756.37,

    "TotalProcessMemorySizeInMB": 17452.38

  }

  

[Oren Eini (Ayende Rahien)]

Can you take a dump of the process of 3.5 when it is using a lot of memory and send it to us?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johan Börjesson]

I managed to take a process dump of the RavenServer.exe... It took a while and it is 63 GB large. I'm currently trying to compress the file, but I don't know how much that will do. 

Anyway, how can I upload this to you? 

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Johan Börjesson]

9 GB zipped. 

What would be the best way to transfer a 9 GB file to you?

[Oren Eini (Ayende Rahien)]

Google Drive

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Johan Börjesson]

I uploaded to google drive. See link below.

https://drive.google.com/open?id=1WreJV-5nsHLih3hZilyKCKfjTd_ZgdUu

I also want to mention one thing... After I did the process dump I noticed that the memory consumption of Raven dropped to just a few hundred megs. I checked the webinterface to see what was going on and most of the databases was not possible to open. It seems that the process dump somehow caused Raven corrupted its state in some kind of way. Now I can't even access the webinterface anymore.. 
That said, I don't know if the process dump is reliable or it's non-sense because of what happend directly after (or during the process dump).

Let me know if I should restart the test and make the process dump earlier before the memory is completely used up.

[Oren Eini (Ayende Rahien)]

Hi,

That was _very_ useful, yes.

We identified what this can cause this.

Can you provide some details about your scenario? You seems to be deleting a LOT of documents, is that the case?

Did you introduce a new index during this process?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Johan Börjesson]

Hi,

We have a flow of incoming data which we store in Raven. For several reasons we are creating a new database on a daily basis. That means that every midnight basically a new database is created and all of our indexes are registered for that new database. These databases are then kept for a few months and then they are dropped. So basically we have several databases containing data for 24 hours. 

Then we also have a much smaller database which only contains data for the last hour or so. So yes, in this short-term database we are actually deleting documents. A timer is ticking every 5:th minute and deleting all documents older than a certain time. (one hour old). For this we are using the DeleteByIndex method.

Indexes are only introduced when a new database is created. 

The reason for having the split between several 24hour databases and one with a shorter timeframe is that the smaller one is queried quite a lot by our monitoring system, and the 24hour databases are queried only on demand when digging into older data. 

[Oren Eini (Ayende Rahien)]

Okay, we are working on a fix, which you can track here: http://issues.hibernatingrhinos.com/issue/RavenDB-11118

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Johan Börjesson]

Ok! Thanks a lot for support! I'm looking forward to test the fix when its ready!

[Johan Börjesson]

Hello

I see that you have made some changes to 3.5 regarding the issue. I'll be happy to test it. How can I get hold of a build? I could only find nightly build on version 4.0 on the website. Will there be a new release or a patch available some time soon?

BR

Johan

Den tisdag 15 maj 2018 kl. 14:37:31 UTC+2 skrev Oren Eini:

[Oren Eini (Ayende Rahien)]

A build should be out today

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Johan Börjesson]

Just wanted to let you know that we have run the new version (RavenDB-3.5.7-patch-35263) for a couple of weeks now in our lab envrionment and it seems that the problem is solved. We are running two parallell setups, one with the cache enabled and one disabled. Both of them are still running with any breakdowns.

Thanks for your support!

[Oren Eini (Ayende Rahien)]

That's great to hear

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.