Production server failure on indexes

[Floris Robbemont]

Hi,

We're having some very serious problems with our RavenDB production server. Yesterday, we upgraded to the latest 2.5 stable version (2956).

Today, we're getting weird errors where indexes are not created (they should be there, but Raven doesn't see them).

This is currently causing multiple applications to stop responding.

We're hosting RavenDB as a service, so no IIS is involved.

Some log entries

2015-04-23 11:52:10.5425;Raven.Storage.Esent.StorageActions.DocumentStorageActions;Warn;;Error when trying to open a new DocumentStorageActions;"Microsoft.Isam.Esent.Interop.EsentInvalidInstanceException: Invalid instance handle

   at Microsoft.Isam.Esent.Interop.Session..ctor(JET_INSTANCE instance) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\Session.cs:line 30

   at Raven.Storage.Esent.StorageActions.DocumentStorageActions..ctor(JET_INSTANCE instance, String database, TableColumnsCache tableColumnsCache, OrderedPartCollection`1 documentCodecs, IUuidGenerator uuidGenerator, IDocumentCacher cacher, EsentTransactionContext transactionContext, TransactionalStorage transactionalStorage) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Esent\StorageActions\General.cs:line 76"

2015-04-23 11:52:10.5425;Raven.Storage.Esent.StorageActions.DocumentStorageActions;Warn;;Error when trying to open a new DocumentStorageActions;"Microsoft.Isam.Esent.Interop.EsentInvalidInstanceException: Invalid instance handle

   at Microsoft.Isam.Esent.Interop.Session..ctor(JET_INSTANCE instance) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\Session.cs:line 30

   at Raven.Storage.Esent.StorageActions.DocumentStorageActions..ctor(JET_INSTANCE instance, String database, TableColumnsCache tableColumnsCache, OrderedPartCollection`1 documentCodecs, IUuidGenerator uuidGenerator, IDocumentCacher cacher, EsentTransactionContext transactionContext, TransactionalStorage transactionalStorage) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Esent\StorageActions\General.cs:line 76"

2015-04-23 11:52:10.5425;Raven.Storage.Esent.StorageActions.DocumentStorageActions;Warn;;Error when trying to open a new DocumentStorageActions;"Microsoft.Isam.Esent.Interop.EsentInvalidInstanceException: Invalid instance handle

   at Microsoft.Isam.Esent.Interop.Session..ctor(JET_INSTANCE instance) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\Session.cs:line 30

   at Raven.Storage.Esent.StorageActions.DocumentStorageActions..ctor(JET_INSTANCE instance, String database, TableColumnsCache tableColumnsCache, OrderedPartCollection`1 documentCodecs, IUuidGenerator uuidGenerator, IDocumentCacher cacher, EsentTransactionContext transactionContext, TransactionalStorage transactionalStorage) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Esent\StorageActions\General.cs:line 76"

2015-04-23 11:52:10.5425;Raven.Storage.Esent.StorageActions.DocumentStorageActions;Warn;;Error when trying to open a new DocumentStorageActions;"Microsoft.Isam.Esent.Interop.EsentInvalidInstanceException: Invalid instance handle

   at Microsoft.Isam.Esent.Interop.Session..ctor(JET_INSTANCE instance) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\Session.cs:line 30

   at Raven.Storage.Esent.StorageActions.DocumentStorageActions..ctor(JET_INSTANCE instance, String database, TableColumnsCache tableColumnsCache, OrderedPartCollection`1 documentCodecs, IUuidGenerator uuidGenerator, IDocumentCacher cacher, EsentTransactionContext transactionContext, TransactionalStorage transactionalStorage) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Esent\StorageActions\General.cs:line 76"

2015-04-23 11:52:10.5425;Raven.Storage.Esent.StorageActions.DocumentStorageActions;Warn;;Error when trying to open a new DocumentStorageActions;"Microsoft.Isam.Esent.Interop.EsentInvalidInstanceException: Invalid instance handle

   at Microsoft.Isam.Esent.Interop.Session..ctor(JET_INSTANCE instance) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\Session.cs:line 30

   at Raven.Storage.Esent.StorageActions.DocumentStorageActions..ctor(JET_INSTANCE instance, String database, TableColumnsCache tableColumnsCache, OrderedPartCollection`1 documentCodecs, IUuidGenerator uuidGenerator, IDocumentCacher cacher, EsentTransactionContext transactionContext, TransactionalStorage transactionalStorage) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Esent\StorageActions\General.cs:line 76"

2015-04-23 11:52:10.5739;Raven.Database.Indexing.IndexSearcherHolder;Warn;;Failed to properly pre-warm the facets cache (Steensoort,UseMargin,CalculatedRetailPrice,Maat,Color,Parelmaat,DiameterMM,Collection,Ringmaat,Schakeltype,AdviceGrossRetailPrice,SteengewichtCT,Steenmaat,Steenkleur,Artikelsoort,BreedteMM,LengteMM,ConceptBrand,Metal,DikteMM,HoogteMM,Bewerking,Slijpvorm,Parelsoort,Gewichtgr,Voorstelling,Vorm,MateriaalOverig,Stijl,Zuiverheid,AdviceNetRetailPrice,Zetwijze) for index ArticleSearch;"System.NullReferenceException: Object reference not set to an instance of an object.

   at Raven.Database.Indexing.IndexSearcherHolder.<>c__DisplayClass4.<SetIndexSearcher>b__0() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexSearcherHolder.cs:line 53"

2015-04-23 11:52:10.5739;Raven.Database.Indexing.IndexSearcherHolder;Warn;;Failed to properly pre-warm the facets cache (invoice nr,pile,invoiced to,hardcopy) for index Image/Search;"System.NullReferenceException: Object reference not set to an instance of an object.

   at Raven.Database.Indexing.IndexSearcherHolder.<>c__DisplayClass4.<SetIndexSearcher>b__0() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexSearcherHolder.cs:line 53"

2015-04-23 11:52:10.9019;Raven.Database.Indexing.IndexSearcherHolder;Warn;;Failed to properly pre-warm the facets cache (WeeklyHours,Region,Province,VacancyGroupName,TargetGroup,FieldOfWork,JobGroupName) for index VacancyIndex;"System.NullReferenceException: Object reference not set to an instance of an object.

   at Raven.Database.Indexing.IndexSearcherHolder.<>c__DisplayClass4.<SetIndexSearcher>b__0() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexSearcherHolder.cs:line 53"

2015-04-23 11:52:11.1206;Raven.Storage.Esent.StorageActions.DocumentStorageActions;Warn;;Error when trying to open a new DocumentStorageActions;"Microsoft.Isam.Esent.Interop.EsentInvalidInstanceException: Invalid instance handle

   at Microsoft.Isam.Esent.Interop.Session..ctor(JET_INSTANCE instance) in C:\Work\ravendb\SharedLibs\Sources\managedesent-61618\EsentInterop\Session.cs:line 30

   at Raven.Storage.Esent.StorageActions.DocumentStorageActions..ctor(JET_INSTANCE instance, String database, TableColumnsCache tableColumnsCache, OrderedPartCollection`1 documentCodecs, IUuidGenerator uuidGenerator, IDocumentCacher cacher, EsentTransactionContext transactionContext, TransactionalStorage transactionalStorage) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Esent\StorageActions\General.cs:line 76"

If needed, I can provide process dumps and/or more logs.

Please help

[Floris Robbemont]

Update: Exporting the database (Smuggler) and importing it again on the same server seems to fix this issue.

Can anyone shine some light on this issue? 

Op donderdag 23 april 2015 12:02:20 UTC+2 schreef Floris Robbemont:

[Michael Yarichuk]

Hi,

Can you provide some server logs and stats endpoint output?

This exception usually means that a thread is running on a database _after_ it was closed.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best regards,

 

Michael Yarichuk

RavenDB Core Team

Tel: 972-4-6227811

Fax:972-153-4-6227811

Email : michael....@hibernatingrhinos.com

 

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

[Floris Robbemont]

Hi Michael,

Unfortunately, we have restored the databases from our backup (this was a production server). I do have a log (attached) from start to the first error.

These errors occured out of nowhere and seems to hit only a couple of databases. The strange thing was that 1 index (out of 36) did not show up in de admin studio.

Also, the application connecting to that DB could not query the index (exception was telling us that the index didn't exist).

Recreating the index didn't help.

The index itself is faily simple:

docs.UserFavorites.Select(userFavorite => new {

    UserId = userFavorite.UserId

})

We did make a backup of that database (using Smuggler) and restored it onto our development machine, we saw nothing wrong (the index was there and the application could query it).

Hopefully this makes any sence. 

The production server is running on a Hyper-V cluster with Cluster Shared Storage (Nexanta ZFS). We did upgrade the server to the latest 2.5 version yesterday.

Like I said, everything is up and running now, and we're planning to migrate to 3.0 within a few weeks, so the priority for this is way less then this morning.

Op donderdag 23 april 2015 15:57:55 UTC+2 schreef Michael Yarichuk:

[Floris Robbemont]

It does seem however that the server (Ravendb.Server.Exe) is busier then normal. It also seems that some other databases still seem to be getting those invalid handle exception, although I cannot see from the logs which databases they are.

Attached is a full log of yesterday.

Op donderdag 23 april 2015 15:57:55 UTC+2 schreef Michael Yarichuk:

Hi,

[Oren Eini (Ayende Rahien)]

There is a known issue with a single very long request.

What will happen is that the database tracked the requests based on their start time, and when we have a very long request (streaming or bulk insert) and no other requests, we'll unload the database while the request was running.

That would cause such errors to show up in the logs.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811