Memory Limit

[Johan]

We have an embedded server (5.0.8) that is deployed as a container application.

We have noticed that either the embedded server or the main process sometimes get terminated by the linux memory manager (due to low memory or out of memory).

I can control the memory requirements of our main process, but can't control the memory limit of the database (or at least not that I know of).

There is a low memory limit configuration, but that wants to keep a certain amount of memory free. This doesn't set a memory limit in Raven itself and with an embedded server the main process will need to take memory too. 

Is there a way to set a memory limit for the embedded server?

[Egor Shamanaev]

Hi,

There is no configuration to limit the memory currently,

We have an issue ticket open for this (for windows), please see: https://issues.hibernatingrhinos.com/issue/RavenDB-15312 
and on windows, you can also limit the memory using PowerShell (Get-Process RavenDB.Server).MaxWorkingSet = 1024*1024*1024

Regarding the OOM, are you running encrypted DBs ?

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/0f4405e0-0f39-4b0a-8151-d0a2e3008d56n%40googlegroups.com.

--

Egor Developer   /   Hibernating Rhinos LTD Sales:  sa...@ravendb.net Support:  sup...@ravendb.net

[Johan]

We are running linux containers. This happens with non-encrypted databases.

For example the container has 8Gb of memory allocated (and the cloud provider doesn't provision swap for containers). The embedded database will use the low memory limit configuration (the default) to trigger low memory conditions when 10% of the memory is available. The database typically doesn't need more than 2Gb, but for some operations it may grow beyond that. Our own main process will take about 4Gb.  However I think the database doesn't see the 4Gb used by the main process and calculates that it still has 6Gb of available memory left. It therefore assumes it can function normally. When total memory usage (of the 2 processes) nears 8Gb then the Linux OOM manager will terminate one of them (sometimes the main process and sometimes the embedded server). We have tried to change the low memory limit configuration, but that doesn't help.

- Johan

[Oren Eini (Ayende Rahien)]

Can you check /admin/debug/proc/meminfo on your instance?

In particular, what is the MemAvailable value that you see there under low memory conditions?

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/0a3c8c5e-0a1f-4a83-93bf-50d37d08868an%40googlegroups.com.

--

Oren Eini CEO   /   Hibernating Rhinos LTD Mobile:  972-52-548-6969 Sales:  sa...@ravendb.net Skype:  ayenderahien Support:  sup...@ravendb.net

[Johan]

There is no such file. The mem available in /proc/meminfo is 1078576 kB.

However I believe these numbers are not reliable inside a container as they may reflect the hosts memory. 

We run with k8s and specify resource limits and then memory information should be obtained from cgroups (as far as I know).

[Oren Eini (Ayende Rahien)]

That isn't a file, sorry, it is an endpoint on RavenDB, like:

http://live-test.ravendb.net/admin/debug/proc/meminfo

Also, this endpoint? /admin/debug/memory/stats

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/6adb81e0-a392-4600-b992-a87fcee2cc1an%40googlegroups.com.

[Johan]

The proc/meminfo:

MemTotal:        7865012 kB

MemFree:          241776 kB

MemAvailable:     751428 kB

The memory stats:

{"MemoryInformation":{"PhysicalMemory":"6.5 GBytes","WorkingSet":"624.75 MBytes","ManagedAllocations":"177 MBytes","UnmanagedAllocations":"8.83 MBytes","EncryptionBuffers":"0 Bytes","MemoryMapped":"8.45 GBytes","ScratchDirtyMemory":"0 Bytes","IsHighDirty":false,"DirtyMemory":"3.6 MBytes","AvailableMemory":"6.06 GBytes","AvailableMemoryForProcessing":"6.056 GBytes","HighMemLastOneMinute":"6.5 GBytes","LowMemLastOneMinute":"6.056 GBytes","HighMemLastFiveMinute":"6.5 GBytes","LowMemLastFiveMinute":"6.056 GBytes","HighMemSinceStartup":"6.5 GBytes","LowMemSinceStartup":"6.056 GBytes"}

Since this is happening in a production system, I don't know when there is such a low memory condition happening. All I am seeing that every now and then a process is terminated by the oom manager.

[Johan]

Ideally I like to have a memory limit setting (for an embedded database) where I can specify the max amount of memory (or a percentage of total) the embedded database is allowed to use instead of assuming it can take all memory available in a container. On Linux you could normally achieve this by using cgroups, but this is inside a container which is already using cgroups for the specified memory restrictions on the container.

[Oren Eini (Ayende Rahien)]

These are excellent numbers, which is strange given your experience.

The MemAvailable at ~700 MB is not ideal, but RavenDB itself is taking a total of 624.75 MBytes in terms of working set.

There really isn't much that we can do to reduce the memory usage from that level, I think. 

Is it possible that some the client application is querying RavenDB with big result set without using streaming? That would force us to use a lot of memory.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/a37983f3-269c-43da-9ecc-2d8c3fb227ean%40googlegroups.com.

[Johan]

We have seen a process being terminated now 6 times (over 3 machines) in the last month, so  it is hard to pin down or provide numbers when it is close to happening

[Oren Eini (Ayende Rahien)]

The OOM can give detailed logs, IIRC

See: https://haydenjames.io/how-to-diagnose-oom-errors-on-linux-systems/

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/0873e8df-d01d-46cd-9703-3e406842b060n%40googlegroups.com.