Raven DB Server constantly restarting

[Johan Börjesson]

We have a test environment with a Raven DB server which has been running for over a month without any issues. One day we experienced some stability issues and found out that the RavenDB Server was crashing and restarting every minute. We are using Raven DB server 3.5.4 build 35215 installed as a windows service.

We looked into the windows event log and found the following stack trace for every crash:

Application: Raven.Server.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: Raven.Imports.Newtonsoft.Json.JsonReaderException

   at Raven.Imports.Newtonsoft.Json.JsonTextReader.ParseValue()

   at Raven.Imports.Newtonsoft.Json.JsonTextReader.Read()

   at Raven.Json.Linq.RavenJObject.Load(Raven.Imports.Newtonsoft.Json.JsonReader)

   at Raven.Abstractions.Extensions.JsonExtensions.ToJObject(System.IO.Stream)

   at Raven.Database.Storage.Esent.StorageActions.DocumentStorageActions.GetDocumentFromStorage(Microsoft.Isam.Esent.Interop.Table, System.String, Raven.Json.Linq.RavenJObject, Raven.Abstractions.Data.Etag, Raven.Json.Linq.RavenJObject ByRef)

   at Raven.Database.Storage.Esent.StorageActions.DocumentStorageActions.ReadDocumentData(Microsoft.Isam.Esent.Interop.Table, System.String, Raven.Abstractions.Data.Etag, Raven.Json.Linq.RavenJObject, Raven.Abstractions.Extensions.Reference`1<Int32>)

Exception Info: System.IO.InvalidDataException

   at Raven.Database.Storage.Esent.StorageActions.DocumentStorageActions.ReadDocumentData(Microsoft.Isam.Esent.Interop.Table, System.String, Raven.Abstractions.Data.Etag, Raven.Json.Linq.RavenJObject, Raven.Abstractions.Extensions.Reference`1<Int32>)

   at Raven.Database.Storage.Esent.StorageActions.DocumentStorageActions+<>c__DisplayClass3_0.<DocumentByKey>b__0(Raven.Abstractions.Data.JsonDocumentMetadata, Int32, System.Func`6<Microsoft.Isam.Esent.Interop.Table,System.String,Raven.Abstractions.Data.Etag,Raven.Json.Linq.RavenJObject,Raven.Abstractions.Extensions.Reference`1<Int32>,Raven.Json.Linq.RavenJObject>)

   at Raven.Database.Storage.Esent.StorageActions.DocumentStorageActions.DocumentByKeyInternal[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](Microsoft.Isam.Esent.Interop.Table, System.String, System.Func`4<Raven.Abstractions.Data.JsonDocumentMetadata,Int32,System.Func`6<Microsoft.Isam.Esent.Interop.Table,System.String,Raven.Abstractions.Data.Etag,Raven.Json.Linq.RavenJObject,Raven.Abstractions.Extensions.Reference`1<Int32>,Raven.Json.Linq.RavenJObject>,System.__Canon>)

   at Raven.Database.Storage.Esent.StorageActions.DocumentStorageActions.DocumentByKey(System.String, Boolean)

   at Raven.Database.Actions.DocumentActions+<>c__DisplayClass15_0.<Get>b__0(Raven.Database.Storage.IStorageActionsAccessor)

   at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(System.Action`1<Raven.Database.Storage.IStorageActionsAccessor>, Raven.Database.Impl.DTC.EsentTransactionContext)

   at Raven.Storage.Esent.TransactionalStorage.Batch(System.Action`1<Raven.Database.Storage.IStorageActionsAccessor>)

   at Raven.Database.Actions.DocumentActions.Get(System.String, Raven.Abstractions.Data.TransactionInformation)

   at Raven.Database.Extensions.DocDbExtensions.AddAlert(Raven.Database.DocumentDatabase, Raven.Abstractions.Data.Alert)

   at Raven.Database.Plugins.Builtins.CheckFreeDiskSpace.ExecuteCheck(System.Object)

   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

   at System.Threading.TimerQueueTimer.CallCallback()

   at System.Threading.TimerQueueTimer.Fire()

   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()

   at System.Threading.ThreadPoolWorkQueue.Dispatch()

A brief scan through the stacktrace (with an untrained RavenDB stack trace reading eye) lead me to believe that something may have happened when checking for free disk space. Therefore I'm adding some information here regarding our test server: 

- Windows Server 2012 

- Intel Core i7-2660 @ 3.40 GHz

- 16 GB RAM

-  C Drive 500 Gb rotating disk

-  E Drive 1 TB SSD (Samsung 850 Evo) <-- This is were RavenDB is installed and also where the database (data + indexes) is located. Still 133 GB free (out of ~1 TB)

Tests performed so far:

 - Manually stopping and starting the service. --> Same error.

 - Tried installing the latest stable version 3.5.5. --> Same error.

Any help would be much appreciated!

BR

Johan

[Oren Eini (Ayende Rahien)]

This looks like the Raven/Alerts document is in a funny shape.

Can you try to delete that from the system database and see if this helps?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johan Börjesson]

I tried to enter the system database and tried to click on the System Documents, but no documents are displayed. Only a constant spinner is visible (showing that it is currently working). Then a few seconds later the Raven DB server crashes again. So I'm not able to delete the document you suggested. 

Is it possible to check anything through the file system instead? 

BR

Johan

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Go to the database directory, there is a system directory there, rename it to system.bak

Start the server, it should start as empty.

You'll then have to point it to the existing databases.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Johan Börjesson]

That seem to have solved to problem. Since we have around 50 databases it was a tedious work to manually create new system database documents to point out all our databases, but at least it solved the problem. So thank you for that!

I'm still a little worried though. How could this happen and what if this would happen in production? 

The system has been up and running with no sudden power failures or unclean shutdowns or similar that might cause corrupt data. 

Thanks again for support!

BR

Johan

[Oren Eini (Ayende Rahien)]

The usual reason for this to happen is if there was some hard disk failure of some kind