Event Sourcing, Embedded RavenDB and memory consumption using 3.0.3800

[Dennis Doomen]

Hi all,

 

We're doing a spike in which we're using an embedded instance of RavenDB 3.0.3800 as a denormalization store for events coming from a SQL Server database (using NEventStore). After reading pages of discussions, blogs as well as the documentation site, I'm still in doubt whether we should continue with RavenDB. Our current approach is maybe a bit naive, but while denormalizing, we're essentially loading the projection (which is about 30-40KB), updating the projection and then calling SaveChanges on the session. We have no indexes whatsoever.

 

We're observing a couple of things:

• Memory consumption goes up pretty quickly with Esent. I can't get any of the memory-related settings such as CacheSizeMax and MaxVerPages to work (neither through the app.config or through the Configuration.Settings dictionary. In fact, if I use management studio to get the configuration, all the settings are on the defaults. On a 16GB machine, memory jumps to 10GB.
• If we use Voron, again, none of the memory settings for MaxBufferPoolSize, MaxScratchBufferSize or MemoryCacheLimitMegabytes are picked up. Memory goes up to 8GB and then drops back to about 1GB.
• While running this setup, the Windows System process is also taking up to 2GB of memory.
• Somewhere after the database contained about 250000 documents, we observed a FatalExecutionEngine while using Voron.

 

Some questions:

• Should we use Esent or Voron. Does one have an advantage over the other? I have the feeling that Voron is slower than Esent, but has less excessive usage (7-8GB instead of 10-11GB).
• How can we limit the working set for the embedded process to about 2 GB?

The fact that we don't have control of what's going on, makes us doubt on this potentially fabulous product. So any advice is much appreciated.

Cheers,

Dennis

[Oren Eini (Ayende Rahien)]

This doesn't seem right.

Can you send us a way to reproduce this?

Voron tends to be faster than Esent for most tasks.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dennis Doomen]

I could send you the code, but to reproduce it, you'll need the 4GB database as well. Instead, could you say anything about the memory-related settings? For instance, should they work in embedded scenarios as well?

[Oren Eini (Ayende Rahien)]

If you could do that, it would be great.

See this article:

http://ravendb.net/docs/article-page/2.5/Csharp/faq/low-memory-footprint

Note that you are continuously increasing the documents size, which means that we have to move them to new locations, which creates additional work.

You also probably have the default indexes.

[Dennis Doomen]

The problem is that the database contains some sensitive client data that I can't share.

I saw that article, but how do I add that to the IDocumentStore configuration? They don't exist as properties and the app.config is not picked up.

I don't need any indexes yet. All the activity involves Loads by ID.

[Oren Eini (Ayende Rahien)]

How do you init the db?

[Dennis Doomen]

            var documentStore = new EmbeddableDocumentStore
            {
                DataDirectory = dataDir,
                Conventions =
                {
                    MaxNumberOfRequestsPerSession = 100,
                    ShouldCacheRequest = (url) => false
                },
                Configuration =
                {
                    MemoryCacheLimitMegabytes = 256,
                    DataDirectory = dataDir,
                    CountersDataDirectory = Path.Combine(configuration.RavenDBDataFolder, "Counters"),
                    CompiledIndexCacheDirectory =
                        Path.Combine(configuration.RavenDBDataFolder, "CompiledIndexCache"),
                    DefaultStorageTypeName = "Voron",
                    Storage = { Voron = { MaxBufferPoolSize = 2, MaxScratchBufferSize = 1024 }}
                },
                EnlistInDistributedTransactions = false,
               
            };

[Oren Eini (Ayende Rahien)]

what build are you running?

Why do you have counters there?

[Dennis Doomen]

3.0.3800

 

W.r.t. the counters, I copied that from a previous project. Previous builds were dropping output in folders we didn’t’ except and fixed all folders to the same location. Does it cause any side-effects?

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/ZCKsa7gNc-I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

It shouldn't.
The config looks fine

[Dennis Doomen]

Instead of sending the sources and/or the original event sourcing database, would it help if I send you the NLog files I collected? If so, where can I upload these?

[Oren Eini (Ayende Rahien)]

It is easier if we can send dump files from when this happens.

You can send them via google drive

[Dennis Doomen]

So if the settings are correct, why does Management Studio => Debug => Config return this?

 

{

    "Replication": {

        "FetchingFromDiskTimeoutInSeconds": 30,

        "ReplicationRequestTimeoutInMilliseconds": 60000,

        "ForceReplicationRequestBuffering": false,

        "MaxNumberOfItemsToReceiveInSingleBatch": null

    },

    "Prefetcher": {

        "FetchingDocumentsFromDiskTimeoutInSeconds": 5,

        "MaximumSizeAllowedToFetchFromStorageInMb": 256

    },

    "Storage": {

        "PreventSchemaUpdate": false,

        "Voron": {

            "MaxBufferPoolSize": 4,

            "InitialFileSize": null,

            "MaxScratchBufferSize": 6144,

            "ScratchBufferSizeNotificationThreshold": 1024,

            "AllowIncrementalBackups": false,

            "TempPath": null,

            "JournalsStoragePath": null,

            "AllowOn32Bits": false

        },

        "Esent": {

            "JournalsStoragePath": null

        }

    },

    "FileSystem": {

        "MaximumSynchronizationInterval": "00:01:00",

        "DataDirectory": "C:\\Workspaces\\VisionSuite.IntegrationHub\\Sources\\IntegrationHub.TestHost\\bin\\Debug\\FileSystems\\",

        "IndexStoragePath": "C:\\Workspaces\\VisionSuite.IntegrationHub\\Sources\\IntegrationHub.TestHost\\bin\\Debug\\FileSystems\\Indexes",

        "DefaultStorageTypeName": null

    },

    "Encryption": {

        "UseFips": false,

        "EncryptionKeyBitsPreference": 128,

        "UseSsl": false

    },

    "Indexing": {

        "MaxNumberOfItemsToProcessInTestIndexes": 512,

        "DisableIndexingFreeSpaceThreshold": 2048,

        "DisableMapReduceInMemoryTracking": false

    },

    "WebSockets": {

        "InitialBufferPoolSize": 131072

    },

    "DatabaseName": "IntegrationHub",

    "FileSystemName": null,

    "CountersDatabaseName": null,

    "MaxClauseCount": 1024,

    "MaxSecondsForTaskToWaitForDatabaseToLoad": 5,

    "IndexAndTransformerReplicationLatencyInSec": 600,

    "AllowScriptsToAdjustNumberOfSteps": false,

    "TombstoneRetentionTime": "14.00:00:00",

    "MaxConcurrentServerRequests": 512,

    "MaxConcurrentRequestsForDatabaseDuringLoad": 50,

    "MaxConcurrentMultiGetRequests": 192,

    "PrefetchingDurationLimit": 5000,

    "BulkImportBatchTimeout": "00:01:00",

    "DatabaseOperationTimeout": "00:05:00",

    "TimeToWaitBeforeRunningIdleIndexes": "00:10:00",

    "TimeToWaitBeforeRunningAbandonedIndexes": "03:00:00",

    "TimeToWaitBeforeMarkingAutoIndexAsIdle": "01:00:00",

    "TimeToWaitBeforeMarkingIdleIndexAsAbandoned": "3.00:00:00",

    "ActiveBundles": [],

    "HeadersToIgnore": [],

    "CustomTaskScheduler": null,

    "RedirectStudioUrl": null,

    "UseDefaultOAuthTokenServer": true,

    "Settings": [

        "Raven/RunInMemory",

        "Raven/CompiledIndexCacheDirectory",

        "Raven/StorageEngine",

        "Raven/DataDir",

        "Raven/Esent/LogsPath",

        "Raven/TransactionJournalsPath",

        "Raven/VirtualDir"

    ],

    "ServerUrl": "http://ws002:38080/",

    "ResetIndexOnUncleanShutdown": false,

    "MaxPageSize": 1024,

    "MemoryCacheLimitPercentage": 0,

    "MemoryCacheLimitMegabytes": 7011,

    "MemoryCacheLimitCheckInterval": "00:02:00",

    "IndexingClassifier": {},

    "MaxNumberOfItemsToProcessInSingleBatch": 131072,

    "InitialNumberOfItemsToProcessInSingleBatch": 512,

    "MaxNumberOfItemsToReduceInSingleBatch": 65536,

    "InitialNumberOfItemsToReduceInSingleBatch": 256,

    "NumberOfItemsToExecuteReduceInSingleStep": 1024,

    "MaxNumberOfParallelProcessingTasks": 8,

    "NewIndexInMemoryMaxBytes": 67108864,

    "HostName": null,

    "Port": 38080,

    "ExposeConfigOverTheWire": "Open",

    "AccessControlAllowOrigin": [],

    "AccessControlMaxAge": "1728000",

    "AccessControlAllowMethods": "PUT,PATCH,GET,DELETE,POST",

    "AccessControlRequestHeaders": null,

    "VirtualDirectory": "/",

    "HttpCompression": true,

    "AnonymousUserAccessMode": "Admin",

    "AllowLocalAccessWithoutAuthorization": false,

    "RejectClientsMode": false,

    "OAuthTokenKey": "<not shown>",

    "IgnoreSslCertificateErrors": "None",

    "WorkingDirectory": "C:\\Workspaces\\VisionSuite.IntegrationHub\\Sources\\IntegrationHub.TestHost\\bin\\Debug\\",

    "DataDirectory": "C:\\Workspaces\\VisionSuite.IntegrationHub\\Sources\\IntegrationHub.TestHost\\bin\\Debug\\.\\Data\\IntegrationHub\\",

    "CountersDataDirectory": "C:\\Workspaces\\VisionSuite.IntegrationHub\\Sources\\IntegrationHub.TestHost\\bin\\Debug\\Data\\Counters\\",

    "DefaultStorageTypeName": "Voron",

    "RunInMemory": false,

    "DisableInMemoryIndexing": false,

    "TransactionMode": "Safe",

    "WebDir": "C:\\Workspaces\\VisionSuite.IntegrationHub\\Sources\\IntegrationHub.TestHost\\bin\\Debug\\Raven/WebUI",

    "PluginsDirectory": "C:\\Workspaces\\VisionSuite.IntegrationHub\\Sources\\IntegrationHub.TestHost\\bin\\Debug\\Plugins\\",

    "AssembliesDirectory": "C:\\Workspaces\\VisionSuite.IntegrationHub\\Sources\\IntegrationHub.TestHost\\bin\\Debug\\Assemblies\\",

    "EmbeddedFilesDirectory": "",

    "CreatePluginsDirectoryIfNotExisting": true,

    "CreateAnalyzersDirectoryIfNotExisting": true,

    "CompiledIndexCacheDirectory": "C:\\Workspaces\\VisionSuite.IntegrationHub\\Sources\\IntegrationHub.TestHost\\bin\\Debug\\.\\Data\\CompiledIndexCache\\",

    "OAuthTokenServer": "http://ws002:38080/OAuth/API-Key",

    "DisableDocumentPreFetching": false,

    "MaxNumberOfItemsToPreFetch": 131072,

    "RunInUnreliableYetFastModeThatIsNotSuitableForProduction": false,

    "MemoryCacheExpiration": "01:00:00",

    "CreateAutoIndexesForAdHocQueriesIfNeeded": true,

    "MaxIndexCommitPointStoreTimeInterval": "00:05:00",

    "MinIndexingTimeIntervalToStoreCommitPoint": "00:01:00",

    "MaxNumberOfStoredCommitPoints": 5,

    "MemoryLimitForProcessingInMb": 1024,

    "DynamicMemoryLimitForProcessing": 1073741824,

    "IndexStoragePath": "C:\\Workspaces\\VisionSuite.IntegrationHub\\Sources\\IntegrationHub.TestHost\\bin\\Debug\\.\\Data\\IntegrationHub\\Indexes",

    "AvailableMemoryForRaisingBatchSizeLimit": 768,

    "MaxProcessingRunLatency": "00:05:00",

    "IsTenantDatabase": true,

    "DisableClusterDiscovery": false,

    "TurnOffDiscoveryClient": false,

    "ServerName": null,

    "MaxStepsForScript": 10000,

    "MaxRecentTouchesToRemember": 1024,

    "AdditionalStepsForScriptBasedOnDocumentSize": 5,

    "MaxIndexWritesBeforeRecreate": 262144,

    "MaxSimpleIndexOutputsPerDocument": 15,

    "MaxMapReduceIndexOutputsPerDocument": 50,

    "PrewarmFacetsOnIndexingMaxAge": "00:10:00",

    "PrewarmFacetsSyncronousWaitTime": "00:00:03",

    "FlushIndexToDiskSizeInMb": 5,

    "EnableResponseLoggingForEmbeddedDatabases": false,

    "NewIndexInMemoryMaxTime": "00:15:00",

    "ImplicitFetchFieldsFromDocumentMode": "Enabled"

[Dennis Doomen]

I have a ZIP including the sources, the data files and everything needed to start the application and reproduce the problem. But since this code is confidential, where I can I send a Google Drive access link to?

...

[Oren Eini (Ayende Rahien)]

On what database?

And what is shown in the db settings for this?

[Oren Eini (Ayende Rahien)]

sup...@ravendb.net

[Dennis Doomen]

If, with “on what database”, you mean the data base type, it’s Voron.

 

DB settings?

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Oren Eini (Ayende Rahien)
Sent: Monday, 23 November, 2015 09:39
To: ravendb <rav...@googlegroups.com>
Subject: Re: [RavenDB] Event Sourcing, Embedded RavenDB and memory consumption using 3.0.3800

 

On what database?

And what is shown in the db settings for this?

[Oren Eini (Ayende Rahien)]

No, you have multiple tenant databases, I assume, what is the db document like for that?

Although, if you have a repro, that would be much easier.

--

[Dennis Doomen]

In the spike, we only have a single database (next to the System one).

 

But I’ll send you a full repo momentarily through the support email.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/ZCKsa7gNc-I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Dennis Doomen]

Send the details directly to that email address..

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Oren Eini (Ayende Rahien)

Sent: Monday, 23 November, 2015 09:39

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/ZCKsa7gNc-I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Dennis Doomen]

Oren, any progress on analyzing that repo I sent you guys? I would like to learn whether we did something wrong or if it is Raven. It's the final step for making the decision to use it more broadly. 

[Michael Yarichuk]

Hi,
I looked into what you've sent.

First of all I am seeing ExecutionEngineException consistently while running your app.

Is this the exception that you are seeing?
Faulting application name: eVision.VisionSuite.IntegrationHub.TestHost.exe, version: 1.0.0.0, time stamp: 0x56520393
Faulting module name: clr.dll, version: 4.6.96.0, time stamp: 0x55b96716
Exception code: 0xc0000005

Application: eVision.VisionSuite.IntegrationHub.TestHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 00007FF870BCF315 (00007FF870A50000) with exit code 80131506.

Also, how are you hosting RavenDB instance? This doesn't happen if I run your database in a separate instance of RavenDB.

Memory usage:

It seems that it is related to _very_ large documents you have there - I've seen at least one document with size of 200MB, with according to statistics is not the only large document.

Other than sheer document size, I do not see any issues with how memory is being used in this case.

(For example -> WorkItemProjection/c04e4557-70d4-593c-be5f-b961e52e8747 is ~200MB)

--

Best regards,

 

Michael Yarichuk

RavenDB Core Team

Tel: 972-4-6227811

Fax:972-153-4-6227811

Email : michael....@hibernatingrhinos.com

 

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

[Dennis Doomen]

Hi Michael,

Thanks for analyzing my repo. Are you sure that document is 200 MB? I was expecting something like 20-30 KB. That must be the reason it failed. So how did you analyze that? I couldn’t use Raven Management Studio myself. Or did you use a memory profiler?

 

The FatalExecutionEngine only happened once to me, but I don’t remember when. Maybe before I switched to 64-bit and Voron...

 

So why is Raven ignoring the memory limits I’m setting? When I go to the debug page to show the full config, it seems to be displaying all the details.

 

Thanks,

Dennis

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/ZCKsa7gNc-I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Dennis Doomen]

Oh, and is there some way I can see the contents of that large document myself. That’ll help me find the design error in our solution…

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Michael Yarichuk

Sent: vrijdag 27 november 2015 18:27

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/ZCKsa7gNc-I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Michael Yarichuk]

1) I looked on a dump that I took when the database used lots of memory
2) I loaded your database in standalone RavenDB instance and waited for some time. The studio is working in this case

Take a look at this document for example : WorkItemProjection/c04e4557-70d4-593c-be5f-b961e52e8747 -> it is huge

About looking at the contents, the simplest way is using a debug endpoint - the following endpoint should do it:
/databases/{databaseName}/debug/raw-doc?id={id}

(list of Raven's debug endpoints http://ravendb.net/docs/article-page/3.0/csharp/server/troubleshooting/debug-endpoints)

[Dennis Doomen]

Thank you Michael. This information is been VERY helpful and allowed me to trace the problem to a bug in our denormalization code.

The only thing that still worries me is that the configuration listed through the Debug page still shows the defaults for the memory settings.

On Saturday, November 28, 2015 at 2:46:25 PM UTC+1, Michael Yarichuk wrote:

[Michael Yarichuk]

Not sure that I follow - Debug page?

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Grisha Kotler]

Are you seeing the debug page using the System Database?

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

--

[Dennis Doomen]

Nope, using something like http://localhost:38080/studio/index.html#databases/status/debug/config?&database=IntegrationHub

 

I’m adding the settings for Esent like this:

 

            documentStore.Configuration.Settings.Add("Raven/Esent/CacheSizeMax", "256");

            documentStore.Configuration.Settings.Add("Raven/Esent/MaxVerPages", "32");

 

And for Voron, like this:

 

                Configuration =

                {

                    MemoryCacheLimitMegabytes = 256,

                    DefaultStorageTypeName = "Voron",

                    Storage = { Voron = { MaxBufferPoolSize = 2, MaxScratchBufferSize = 1024 }}

                },

 

Neither of the settings seem to work

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Grisha Kotler
Sent: Monday, 30 November, 2015 10:31
To: rav...@googlegroups.com
Subject: Re: [RavenDB] Event Sourcing, Embedded RavenDB and memory consumption using 3.0.3800

 

Are you seeing the debug page using the System Database?

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/ZCKsa7gNc-I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Grisha Kotler]

How are you creating this database?

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

[Grisha Kotler]

For Voron you need to specify the configuration in the settings (like it's done in Esent):

documentStore.Configuration.Settings.Add("Raven/Voron/MaxBufferPoolSize", "2");

documentStore.Configuration.Settings.Add("Raven/Voron/MaxScratchBufferSize", "1024");

For Esent: the settings are applied but aren't displayed in the debug page.

We'll fix this shortly.

http://issues.hibernatingrhinos.com/issue/RavenDB-4073

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

On 30 November 2015 at 12:48, Dennis Doomen <dennis...@gmail.com> wrote:

[Dennis Doomen]

Thanks. That explains the large memory usage. Running the tests again.

 

So, what advantage or disadvantage has Voron over Esent? In other words, based on what criteria should I select one over the other?

[Oren Eini (Ayende Rahien)]

Voron is our own storage engine, and we built it so we can control every aspect of how it is used.

Esent is a windows component that we use. We have very little insight into how/what it does.

Esent has been around for a very long time, Voron has been around for a few years.

We are moving more and more into Voron.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

[Dennis Doomen]

OK, so it doesn’t really matter right now which I use. Thanks for clarifying.

[Dennis Doomen]

Looking at the settings available, it seems that Voron is not really suitable for low-memory situations, correct? I mean, even with minimum settings mentioned in this thread, the process working set stays at around 2.5-3 GB.

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Oren Eini (Ayende Rahien)

Sent: Tuesday, 1 December, 2015 08:33

[Oren Eini (Ayende Rahien)]

Voron is using mmap for the entire file, the OS is going to manage what is loaded into actual RAM for us

[Dennis Doomen]

Which is not a direct answer to my question ;-)

[Oren Eini (Ayende Rahien)]

There is a difference between used memory and allocated memory, basically.

[Dennis Doomen]

So to make question more specific, if I want to use RavenDB in embedded mode in a light weight console application, I want to keep the memory usage maxed at 1 GB, should I use Voron or Esent? And with what settings?

PS. I'll blog about all this stuff as well. I want to build up a good example of using RavenDB in a low memory situation.

[Kijana Woodard]

Maybe you can get Tobi to chime in on experience with such systems [embedded pos, iirc]. 

Search for posts in the forum history.

[Oren Eini (Ayende Rahien)]

Okay, we need to define something.

Memory usage is what the app is currently using, it isn't what it has allocated.

The OS can quite easily discard the memory we have allocated, if it is can be loaded from a file, frex.

This also relates to the size of your db, Voron maps the whole file, but only access the used portioned, and the OS will make sure that it will not use more memory than is available.

[Michael Yarichuk]

What kind of load do you expect to have in RavenDB?
I mean - how many documents, their sizes, how many queries/changes per sec.,  how many indexes, their complexity, etc

Best regards,

 

Michael Yarichuk

RavenDB Core Team

Tel: 972-4-6227811

Fax:972-153-4-6227811

Email : michael....@hibernatingrhinos.com

 

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

[Dennis Doomen]

I understand, but somehow I suspect SQL Server is using a similar approach. And if I configure that one to not consume more memory than 2GB, then it never will. I was hoping to be able to do something similar with Voron or Esent.

 

We’re considering to have a Windows Service running next to IIS on a front-end web server which hosts Raven embedded and serves queries over projections (stored as documents) and indexes derived from that. So being able to ‘guarantee’ RavenDB is not consuming more than an x amount is pretty important. Losing some performance because of that is not such a big deal.

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Oren Eini (Ayende Rahien)
Sent: Tuesday, 1 December, 2015 19:31
To: ravendb <rav...@googlegroups.com>
Subject: Re: [RavenDB] Event Sourcing, Embedded RavenDB and memory consumption using 3.0.3800

 

Okay, we need to define something.

[Dennis Doomen]

Millions of documents of about 30-100KB, with all kinds of indexes around them. Not sure if we’ll need map/reduce constructs though.

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Michael Yarichuk
Sent: Tuesday, 1 December, 2015 20:17
To: RavenDB - 2nd generation document database <rav...@googlegroups.com>
Subject: Re: [RavenDB] Event Sourcing, Embedded RavenDB and memory consumption using 3.0.3800

 

What kind of load do you expect to have in RavenDB?

[Oren Eini (Ayende Rahien)]

SQL Server actually uses a very different method. It uses its own paging subsystem, for various (mostly historical and 32 bits) reasons.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

[Oren Eini (Ayende Rahien)]

The easiest way to have a low memory footprint is already in the docs, IIRC, 

http://ravendb.net/docs/article-page/2.5/Csharp/faq/low-memory-footprint

[Dennis Doomen]

And what about Voron?

 

BTW, these settings don’t seem to have the desired effect. The Working Set easily reaches 3GB. But I’ll have to run a memory profiler to understand where that memory is kept.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/ZCKsa7gNc-I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Dennis Doomen]

And what about Voron?

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Oren Eini (Ayende Rahien)
Sent: Tuesday, 1 December, 2015 20:39
To: ravendb <rav...@googlegroups.com>
Subject: Re: [RavenDB] Event Sourcing, Embedded RavenDB and memory consumption using 3.0.3800

 

The easiest way to have a low memory footprint is already in the docs, IIRC, 

http://ravendb.net/docs/article-page/2.5/Csharp/faq/low-memory-footprint

[Oren Eini (Ayende Rahien)]

Those settings are for _Esent_ only.

What are you doing while this is going on?

Note that you can also reduce the max number of items to process in a single batch. But that would increase indexing time.

[Dennis Doomen]

Projecting 1.5 million events from a SQL Server database to Raven projections/documents (in batches of 1 to 255 events per session).

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Oren Eini (Ayende Rahien)
Sent: Tuesday, 1 December, 2015 20:45
To: ravendb <rav...@googlegroups.com>
Subject: Re: [RavenDB] Event Sourcing, Embedded RavenDB and memory consumption using 3.0.3800

 

Those settings are for _Esent_ only.

 

What are you doing while this is going on?

 

Note that you can also reduce the max number of items to process in a single batch. But that would increase indexing time.

[Oren Eini (Ayende Rahien)]

Indexes along the way? The internal cache?

--

[Chris Marisic]

For clarity, you're talking about database that document storage alone will consist of 250GB, 500GB or more?

Because the concerns i have 2.5GB are entirely different than 25GB and vastly different than 250GB and super vastly different than 2,500GB.

[Dennis Doomen]

If I must guess, I would say between 2.5 and 50GB.

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Chris Marisic
Sent: Tuesday, 1 December, 2015 21:04
To: RavenDB - 2nd generation document database <rav...@googlegroups.com>
Subject: Re: [RavenDB] Event Sourcing, Embedded RavenDB and memory consumption using 3.0.3800

 

For clarity, you're talking about database that document storage alone will consist of 250GB, 500GB or more?

 

[Dennis Doomen]

With “max number of items to process in a single batch”, do you mean a particular option or all four of the MaxNumberOf… options from http://ravendb.net/docs/article-page/3.0/csharp/server/configuration/configuration-options

[Tobi]

I haven't that much experience with Voron yet besides testing this
in-house. The performance issues with RavenDB 3 still keep me from updating.

One thing to note: You can use Voron only on a 64bit OS (The whole
database needs to be addressable via a single memory map!). So if you
intend to run the console application on 32 bit systems as well, you have
to stick to Esent.

Comparing the performance of Voron and Esent with an embedded RavenDB
doesn't really make much of a difference. At least not the last time I
tested this - there are more serious performance bottlenecks at higher levels.

I'm using RavenDB 2.5 in an embedded application now for about 4-5 years
and so far I didn't lost a bit of data from the Esent storage (the index
storage is much more fragile!). Besides this there's esentutl.exe which
might be able to repair a damaged database up to a certain degree.

So right now Esent has a big trust-bonus, but I can't say anything bad
about Voron.

With one of the next major releases of my application I will hopefully
switch to RavenDB 3.0 and I will introduce a setting to be able to decide
which storage engine to use. But most likely I will still use Esent on
most systems, simply because 64bit code sometimes tends to be noticeably
slower than 32 bit code.

It's just a tiny setting in EmbeddableDocumentStore so my advice would be
to simply add "--voron / --esent" options to your application and default
to Esent.

BR,

Tobias

On 01.12.2015 18:23, Kijana Woodard wrote:

> Maybe you can get Tobi to chime in on experience with such systems
> [embedded pos, iirc].
> Search for posts in the forum history.
>
> On Tue, Dec 1, 2015 at 10:54 AM, Dennis Doomen <dennis...@gmail.com
> <mailto:dennis...@gmail.com>> wrote:
>
> So to make question more specific, if I want to use RavenDB in
> embedded mode in a light weight console application, I want to keep
> the memory usage maxed at 1 GB, should I use Voron or Esent? And with
> what settings?
>
> PS. I'll blog about all this stuff as well. I want to build up a good
> example of using RavenDB in a low memory situation.
>
> On Tue, Dec 1, 2015 at 1:19 PM, Oren Eini (Ayende Rahien)
> <aye...@ayende.com <mailto:aye...@ayende.com>> wrote:
>
> There is a difference between used memory and allocated memory,
> basically.
>

> */Hibernating Rhinos Ltd /*____
>
> Oren Eini* l CEO l *Mobile: + 972-52-548-6969
> <tel:%2B%20972-52-548-6969>
>
> Office: +972-4-622-7811 <tel:%2B972-4-622-7811> *l *Fax:
> +972-153-4-622-7811
>
> __
>
> __

>
>
> On Tue, Dec 1, 2015 at 1:53 PM, Dennis Doomen

> <dennis...@gmail.com <mailto:dennis...@gmail.com>> wrote:
>
> Which is not a direct answer to my question ;-)____
>
> __ __
>
> *From:*rav...@googlegroups.com
> <mailto:rav...@googlegroups.com>
> [mailto:rav...@googlegroups.com
> <mailto:rav...@googlegroups.com>] *On Behalf Of *Oren Eini
> (Ayende Rahien)
> *Sent:* Tuesday, 1 December, 2015 11:57
>
>
> *To:* ravendb <rav...@googlegroups.com
> <mailto:rav...@googlegroups.com>>
> *Subject:* Re: [RavenDB] Event Sourcing, Embedded RavenDB and
> memory consumption using 3.0.3800____
>
> __ __

>
> Voron is using mmap for the entire file, the OS is going to

> manage what is loaded into actual RAM for us____
>
>
> ____
>
> */Hibernating Rhinos Ltd /*____
>
> Oren Eini* l** **CEO **l *Mobile: + 972-52-548-6969
> <tel:%2B%20972-52-548-6969>____
>
> Office: +972-4-622-7811 <tel:%2B972-4-622-7811> *l** *Fax:
> +972-153-4-622-7811____
>
> ____
>
> __ __

>
> On Tue, Dec 1, 2015 at 12:50 PM, Dennis Doomen

> <dennis...@gmail.com <mailto:dennis...@gmail.com>>
> wrote:____

>
> Looking at the settings available, it seems that Voron is
> not really suitable for low-memory situations, correct? I
> mean, even with minimum settings mentioned in this thread,

> the process working set stays at around 2.5-3 GB.____
>
> ____
>
> *From:*rav...@googlegroups.com
> <mailto:rav...@googlegroups.com>
> [mailto:rav...@googlegroups.com
> <mailto:rav...@googlegroups.com>] *On Behalf Of *Oren
> Eini (Ayende Rahien)
> *Sent:* Tuesday, 1 December, 2015 08:33
> *To:* ravendb <rav...@googlegroups.com
> <mailto:rav...@googlegroups.com>>____
>
>
> *Subject:* Re: [RavenDB] Event Sourcing, Embedded RavenDB
> and memory consumption using 3.0.3800____
>
> ____

>
> Voron is our own storage engine, and we built it so we can

> control every aspect of how it is used.____

>
> Esent is a windows component that we use. We have very

> little insight into how/what it does.____
>
> ____

>
> Esent has been around for a very long time, Voron has been

> around for a few years.____
>
> ____
>
> We are moving more and more into Voron.____
>
>
> ____
>
> */Hibernating Rhinos Ltd /*____
>
> Oren Eini* l** **CEO **l *Mobile: + 972-52-548-6969
> <tel:%2B%20972-52-548-6969>____
>
> Office: +972-4-622-7811 <tel:%2B972-4-622-7811> *l** *Fax:
> +972-153-4-622-7811____
>
> ____
>
> ____

>
> On Tue, Dec 1, 2015 at 9:04 AM, Dennis Doomen

> <dennis...@gmail.com <mailto:dennis...@gmail.com>>
> wrote:____

>
> Thanks. That explains the large memory usage. Running

> the tests again.____
>
> ____

>
> So, what advantage or disadvantage has Voron over
> Esent? In other words, based on what criteria should I

> select one over the other?____
>
> ____
>
> *From:*rav...@googlegroups.com
> <mailto:rav...@googlegroups.com>
> [mailto:rav...@googlegroups.com
> <mailto:rav...@googlegroups.com>] *On Behalf Of
> *Grisha Kotler
> *Sent:* Monday, 30 November, 2015 15:01____
>
>
> *To:* rav...@googlegroups.com
> <mailto:rav...@googlegroups.com>
> *Subject:* Re: [RavenDB] Event Sourcing, Embedded
> RavenDB and memory consumption using 3.0.3800____
>
> ____

>
> For Voron you need to specify the configuration in the

> settings (like it's done in Esent):____
>
> documentStore.Configuration.Settings.Add("Raven/Voron/MaxBufferPoolSize",
> "2");____
>
> documentStore.Configuration.Settings.Add("Raven/Voron/MaxScratchBufferSize",
> "1024");____
>
> ____

>
> For Esent: the settings are applied but aren't

> displayed in the debug page.____
>
> We'll fix this shortly.____
>
> http://issues.hibernatingrhinos.com/issue/RavenDB-4073____
>
>
> ____
>
> */Hibernating Rhinos
> Ltd /*cid:image0...@01CF95E2.8ED1B7D0____
>
> Grisha Kotler* l** **RavenDB Core Team
> Developer **l *Mobile: +972-54-586-8647
> <tel:%2B972-54-586-8647>____
>
> Office: +972-4-622-7811
> <tel:%2B972-4-622-7811>* l** *Fax: +972-153-4-622-7811____
>
> */RavenDB/*/ paving the way to "Data
> Made Simple" http://ravendb.net//____
>
> ____

>
> On 30 November 2015 at 12:48, Dennis Doomen
> <dennis...@gmail.com

> <mailto:dennis...@gmail.com>> wrote:____
>
> Nope, using something like
> http://localhost:38080/studio/index.html#databases/status/debug/config?&database=IntegrationHub____
>
> ____
>
> I’m adding the settings for Esent like this:____
>
> ____
>
>
> documentStore.Configuration.Settings.Add("Raven/Esent/CacheSizeMax",
> "256");____
>
>
> documentStore.Configuration.Settings.Add("Raven/Esent/MaxVerPages",
> "32");____
>
> ____
>
> And for Voron, like this:____
>
> ____
>
> Configuration =____
>
> {____
>
> MemoryCacheLimitMegabytes =
> 256,____
>
> DefaultStorageTypeName =
> "Voron",____

>
> Storage = { Voron = {
> MaxBufferPoolSize = 2, MaxScratchBufferSize = 1024

> }}____
>
> },____
>
> ____
>
> Neither of the settings seem to work____
>
> ____
>
> *From:*rav...@googlegroups.com
> <mailto:rav...@googlegroups.com>
> [mailto:rav...@googlegroups.com
> <mailto:rav...@googlegroups.com>] *On Behalf Of
> *Grisha Kotler
> *Sent:* Monday, 30 November, 2015 10:31
> *To:* rav...@googlegroups.com
> <mailto:rav...@googlegroups.com>
> *Subject:* Re: [RavenDB] Event Sourcing, Embedded
> RavenDB and memory consumption using 3.0.3800____
>
> ____

>
> Are you seeing the debug page using the System

> Database?____
>
> ____
>
> */Hibernating Rhinos
> Ltd /*cid:image0...@01CF95E2.8ED1B7D0____
>
> Grisha Kotler* l** **RavenDB Core Team
> Developer **l *Mobile: +972-54-586-8647
> <tel:%2B972-54-586-8647>____
>
> Office: +972-4-622-7811
> <tel:%2B972-4-622-7811>* l** *Fax:
> +972-153-4-622-7811____
>
> */RavenDB/*/ paving the way to "Data
> Made Simple" http://ravendb.net//____
>
> ____

>
> On 30 November 2015 at 10:09, Dennis Doomen
> <dennis...@gmail.com

> <mailto:dennis...@gmail.com>> wrote:____

>
> Thank you Michael. This information is been
> VERY helpful and allowed me to trace the

> problem to a bug in our denormalization code. ____
>
> ____

>
> The only thing that still worries me is that
> the configuration listed through the Debug
> page still shows the defaults for the memory
> settings.
>
> On Saturday, November 28, 2015 at 2:46:25 PM

> UTC+1, Michael Yarichuk wrote:____

>
> 1) I looked on a dump that I took when the
> database used lots of memory
> 2) I loaded your database in standalone
> RavenDB instance and waited for some time.
> The studio is working in this case
>
> Take a look at this document for example :
> WorkItemProjection/c04e4557-70d4-593c-be5f-b961e52e8747
> -> it is huge
>
> About looking at the contents, the
> simplest way is using a debug endpoint -
> the following endpoint should do it:
> /databases/{databaseName}/debug/raw-doc?id={id}
>
> (list of Raven's debug endpoints

> http://ravendb.net/docs/article-page/3.0/csharp/server/troubleshooting/debug-endpoints)____
>
> ____

>
> On Sat, Nov 28, 2015 at 10:32 AM, Dennis
> Doomen <dennis...@gmail.com

> <mailto:dennis...@gmail.com>> wrote:____

>
> Oh, and is there some way I can see
> the contents of that large document
> myself. That’ll help me find the

> design error in our solution…____
>
> -- ____

>
> You received this message because you are
> subscribed to the Google Groups "RavenDB - 2nd
> generation document database" group.
> To unsubscribe from this group and stop
> receiving emails from it, send an email to
> ravendb+u...@googlegroups.com

> <mailto:ravendb+u...@googlegroups.com>.
> For more options, visit
> https://groups.google.com/d/optout.____
>
> ____

>
> --
> You received this message because you are
> subscribed to a topic in the Google Groups
> "RavenDB - 2nd generation document database" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ravendb/ZCKsa7gNc-I/unsubscribe.
> To unsubscribe from this group and all its topics,
> send an email to
> ravendb+u...@googlegroups.com

> <mailto:ravendb+u...@googlegroups.com>.
> For more options, visit
> https://groups.google.com/d/optout.____

>
> --
> You received this message because you are
> subscribed to the Google Groups "RavenDB - 2nd
> generation document database" group.
> To unsubscribe from this group and stop receiving
> emails from it, send an email to
> ravendb+u...@googlegroups.com

> <mailto:ravendb+u...@googlegroups.com>.
> For more options, visit
> https://groups.google.com/d/optout.____
>
> ____

>
> --
> You received this message because you are subscribed
> to a topic in the Google Groups "RavenDB - 2nd
> generation document database" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ravendb/ZCKsa7gNc-I/unsubscribe.
> To unsubscribe from this group and all its topics,
> send an email to ravendb+u...@googlegroups.com

> <mailto:ravendb+u...@googlegroups.com>.
> For more options, visit
> https://groups.google.com/d/optout.____

>
> --
> You received this message because you are subscribed
> to the Google Groups "RavenDB - 2nd generation
> document database" group.
> To unsubscribe from this group and stop receiving
> emails from it, send an email to
> ravendb+u...@googlegroups.com

> <mailto:ravendb+u...@googlegroups.com>.
> For more options, visit
> https://groups.google.com/d/optout.____
>
> ____

>
> --
> You received this message because you are subscribed to a
> topic in the Google Groups "RavenDB - 2nd generation
> document database" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ravendb/ZCKsa7gNc-I/unsubscribe.
> To unsubscribe from this group and all its topics, send an
> email to ravendb+u...@googlegroups.com

> <mailto:ravendb+u...@googlegroups.com>.
> For more options, visit
> https://groups.google.com/d/optout.____

>
> --
> You received this message because you are subscribed to
> the Google Groups "RavenDB - 2nd generation document
> database" group.
> To unsubscribe from this group and stop receiving emails
> from it, send an email to
> ravendb+u...@googlegroups.com

> <mailto:ravendb+u...@googlegroups.com>.
> For more options, visit
> https://groups.google.com/d/optout.____
>
> __ __

>
> --
> You received this message because you are subscribed to a
> topic in the Google Groups "RavenDB - 2nd generation document
> database" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ravendb/ZCKsa7gNc-I/unsubscribe.
> To unsubscribe from this group and all its topics, send an
> email to ravendb+u...@googlegroups.com

> <mailto:ravendb+u...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.____

>
> --
> You received this message because you are subscribed to the
> Google Groups "RavenDB - 2nd generation document database" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to ravendb+u...@googlegroups.com

> <mailto:ravendb+u...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "RavenDB - 2nd generation document database" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ravendb/ZCKsa7gNc-I/unsubscribe.
> To unsubscribe from this group and all its topics, send an email
> to ravendb+u...@googlegroups.com

> <mailto:ravendb+u...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "RavenDB - 2nd generation document database" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to ravendb+u...@googlegroups.com

> <mailto:ravendb+u...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "RavenDB - 2nd generation document database" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ravendb+u...@googlegroups.com

> <mailto:ravendb+u...@googlegroups.com>.

[dennis...@gmail.com]

Tobi, thank you for sharing this…

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Yes, those, as well as Raven/MemoryCacheLimitMegabyte

Note that memory usage in ravendb is automatically managed, and we'll by default try to use as much memory as we can, to provide faster response times.