Replication problem - EsentVersionStoreOutOfMemoryException

[Lawrence Short]

Hi guys,

We are receiving the following error when attempting to replicate between RavenDB servers.

Error: Microsoft.Isam.Esent.Interop.EsentVersionStoreOutOfMemoryException: Version store out of memory (cleanup already attempted) at Microsoft.Isam.Esent.Interop.Api.Check(Int32 err) at Microsoft.Isam.Esent.Interop.Api.JetEscrowUpdate(JET_SESID sesid, JET_TABLEID tableid, JET_COLUMNID columnid, Byte[] delta, Int32 deltaSize, Byte[] previousValue, Int32 previousValueLength, Int32& actualPreviousValueLength, EscrowUpdateGrbit grbit) at Microsoft.Isam.Esent.Interop.Api.EscrowUpdate(JET_SESID sesid, JET_TABLEID tableid, JET_COLUMNID columnid, Int32 delta) at Raven.Database.Storage.Esent.StorageActions.DocumentStorageActions.AddDocument(String key, Etag etag, RavenJObject data, RavenJObject metadata) at Raven.Database.Actions.DocumentActions.<>c__DisplayClass20_0.<Put>b__0(IStorageActionsAccessor actions) at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action) at Raven.Database.Actions.DocumentActions.Put(String key, Etag etag, RavenJObject document, RavenJObject metadata, TransactionInformation transactionInformation, String[] participatingIds) at Raven.Bundles.Replication.Responders.DocumentReplicationBehavior.AddWithoutConflict(String id, Etag etag, RavenJObject metadata, RavenJObject incoming) at Raven.Database.Bundles.Replication.Responders.Behaviors.SingleItemReplicationBehavior`2.Replicate(String id, RavenJObject metadata, TExternal incoming) at Raven.Database.Bundles.Replication.Controllers.ReplicationController.<>c__DisplayClass15_2.<DocReplicatePost>b__0(IStorageActionsAccessor actions) at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action`1 action, EsentTransactionContext transactionContext) at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action) at Raven.Database.Bundles.Replication.Controllers.ReplicationController.<DocReplicatePost>d__15.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Threading.Tasks.TaskHelpersExtensions.<CastToObject>d__3`1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()

Our server build is #35180

The number of records in the database is 18,378,296

We have tried disabling and re-enabling the Replication configuration. This has not fixed the problem. There are a number of other similar posts that have suggestions from other users of RavenDB to increase the MaxVerPages from 512 to a higher number.

Can you help with this?

[Oren Eini (Ayende Rahien)]

The configuration value is Raven/Esent/MaxVerPages

See: https://ravendb.net/docs/article-page/3.5/Csharp/server/configuration/configuration-options

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lawrence Short]

Hi Oren,

Thanks for the quick reply. Ideally we would like to avoid configuration changes and having to restart the RavenDB service. Is the change to the configuration value the recommended way to fix this type of issue?

Best Regards

Lawrence

[Oren Eini (Ayende Rahien)]

Yes

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

[Lawrence Short]

If we delete the replication configuration and then re-create it, will this reset the Esent version store and allow replication to continue?

[Oren Eini (Ayende Rahien)]

Yes, on the other side, but you'll still need to update the esent config.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Wed, Aug 9, 2017 at 11:45 AM, Lawrence Short <siml...@gmail.com> wrote:

If we delete the replication configuration and then re-create it, will this reset the Esent version store and allow replication to continue?

[Lawrence Short]

Is there a recommended method for calculating how large the Raven/Esent/MaxVerPages should be?

[Oren Eini (Ayende Rahien)]

Set it to half your memory, it will not use more then it needs.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Thu, Aug 10, 2017 at 4:09 AM, Lawrence Short <siml...@gmail.com> wrote:

Is there a recommended method for calculating how large the Raven/Esent/MaxVerPages should be?

[Lawrence Short]

By "half your memory" do you mean half the memory installed on the machine hosting the RavenDB server or half the memory available to the RavenDB server based on the License in use?

[Oren Eini (Ayende Rahien)]

Half the license memory or half the machine memory, the smallest of the values

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Thu, Aug 10, 2017 at 7:52 AM, Lawrence Short <siml...@gmail.com> wrote:

By "half your memory" do you mean half the memory installed on the machine hosting the RavenDB server or half the memory available to the RavenDB server based on the License in use?