indexing.crash-marker in use question

[Kyle Levien]

We've started to see a sharp increase in reports of the following error with our ravendb 2.5 deploys:

System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The process cannot access the file '<path-to-db>\\indexing\\indexing.crash-marker' because it is being used by another process.

This is occurring in IndexingStorage.cs line 111.

I was wondering if anyone has insight on what could be locking that file. We're are investigating various hunches involving multiple document stores pointing to the same directory (does not seem the case, we only create one) or issues with servers going down (also unlikely as we've testing our soft server reboot scripts and see the crash-marker properly delete and recreate).

Anybody else see reports of this error?

Thanks,

Kyle

[Oren Eini (Ayende Rahien)]

Are you using embedded?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kyle Levien]

No, these are servers running through IIS. We have the db data, assemblies, and indexes in a folder outside of the virtual directory that raven runs out of with the following app settings:

<add key="Raven/DataDir" value="..\DB\System" />

<add key="Raven/CompiledIndexCacheDirectory" value="..\DB\RavenDB25.CompiledIndexCache" />

<add key="Raven/AssembliesDirectory" value="..\DB\RavenDB25.Assemblies" />

[Oren Eini (Ayende Rahien)]

Check to see if you have defined mulitple databases to the same path

[Chris Marisic]

Have you disabled overlapped recycle?

[Kyle Levien]

Oren - I'm gathering more information on the environments we're getting reports from but as far as I can tell we do not.

Chris - Yes

[Kyle Levien]

We did some further investigation in the ravendb debug logs when the issue reproduced itself again and here are our findings (includes log snippets):

1) The db is not in use for a couple hours (overnight) and shuts down --

2016-04-27 02:29:04.7572,Raven.Database.Server.HttpServer,Info,"Database IGXContentStore, had no incoming requests idle for 00:15:01.6999686, trying to shut it down",
2016-04-27 02:29:04.7572,Raven.Database.Server.HttpServer,Info,"Shutting down database IGXContentStore. Last work time: 04/27/2016 09:14:03, skipIfActive: True",
2016-04-27 02:29:04.7572,Raven.Database.DocumentDatabase,Debug,Start shutdown the following database: IGXContentStore,
2016-04-27 02:29:04.7572,Raven.Database.Indexing.WorkContext,Debug,Stopping background workers,

2) When it is hit again our static index definitions are located and loaded (we have ~45 indexes of mixed simple and map/reduce), but one has the following error:

2016-04-27 06:39:04.7283,Raven.Database.Indexing.IndexStorage.Startup,Warn,Could not open index AllowedSecurityDescriptorsByGroup. Trying to recover index,"System.ComponentModel.Composition.CompositionException: The composition produced a single composition error. The root cause is provided below. Review the CompositionException.Errors property for more detailed information.


1) Internal error occurred. Additional information: ''.


Resulting in: Cannot populate the collection 'Raven.Database.Indexing.Index.AnalyzerGenerators' because an exception occurred while calling the Add method on the type 'Microsoft.Internal.Collections.CollectionServices+CollectionOfObject`1[[System.Lazy`2[[Raven.Database.Plugins.AbstractAnalyzerGenerator, Raven.Database, Version=2.5.0.0, Culture=neutral, PublicKeyToken=37f41c7f99471593],[Raven.Abstractions.MEF.IPartMetadata, Raven.Abstractions, Version=2.5.0.0, Culture=neutral, PublicKeyToken=37f41c7f99471593]], System.ComponentModel.Composition, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]'.


Resulting in: Cannot activate part 'Raven.Database.Indexing.MapReduceIndex'.
Element: Raven.Database.Indexing.MapReduceIndex -->  Raven.Database.Indexing.MapReduceIndex


   at System.ComponentModel.Composition.CompositionResult.ThrowOnErrors(AtomicComposition atomicComposition)
   at System.ComponentModel.Composition.Hosting.ImportEngine.SatisfyImportsOnce(ComposablePart part)
   at System.ComponentModel.Composition.Hosting.CompositionContainer.SatisfyImportsOnce(ComposablePart part)
   at System.ComponentModel.Composition.AttributedModelServices.SatisfyImportsOnce(ICompositionService compositionService, Object attributedPart)
   at Raven.Database.Indexing.IndexStorage.CreateIndexImplementation(String directoryPath, IndexDefinition indexDefinition, Directory directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 786
   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 139"

This index is a rather simple map/reduce with no defined analyzers.

3) This is followed by the following IO exceptions and a database shutdown:

2016-04-27 06:39:07.1189,Raven.Database.Indexing.IndexStorage,Warn,Could not create index storage,"System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)

   at Raven.Database.Indexing.DefaultBackgroundTaskExecuter.ExecuteAllInterleaved[T](WorkContext context, IList`1 result, Action`1 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\DefaultBackgroundTaskExecuter.cs:line 182

   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111

   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 237

---> (Inner Exception #0) System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()<---

"

2016-04-27 06:39:07.1346,Raven.Database.Indexing.IndexStorage,Fatal,Failed to disposed when already getting an error during ctor,"System.NullReferenceException: Object reference not set to an instance of an object.

   at Raven.Database.Indexing.IndexStorage.FlushMapIndexes() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 1236

   at Raven.Database.Indexing.IndexStorage.Dispose() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 803

   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 105"

2016-04-27 06:39:07.1346,Raven.Database.DocumentDatabase,Error,Could not create database,"System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)

   at Raven.Database.Indexing.DefaultBackgroundTaskExecuter.ExecuteAllInterleaved[T](WorkContext context, IList`1 result, Action`1 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\DefaultBackgroundTaskExecuter.cs:line 182

   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111

   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 237

---> (Inner Exception #0) System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()<---

"

2016-04-27 06:39:07.1346,Raven.Database.DocumentDatabase,Debug,Start shutdown the following database: IGXContentStore,

2016-04-27 06:39:07.1346,Raven.Database.Indexing.WorkContext,Debug,Stopping background workers,

2016-04-27 06:39:07.3689,Raven.Database.DocumentDatabase,Debug,Finished shutdown the following database: IGXContentStore,

2016-04-27 06:39:07.3689,Raven.Database.Server.HttpServer,Warn,Failed to create database IGXContentStore,"System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)

   at Raven.Database.Indexing.DefaultBackgroundTaskExecuter.ExecuteAllInterleaved[T](WorkContext context, IList`1 result, Action`1 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\DefaultBackgroundTaskExecuter.cs:line 182

   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111

   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 259

   at Raven.Database.Server.HttpServer.<>c__DisplayClass43.<TryGetOrCreateResourceStore>b__41() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 1112

   at System.Threading.Tasks.Task`1.InnerInvoke()

   at System.Threading.Tasks.Task.Execute()

   --- End of inner exception stack trace ---

---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)

   at Raven.Database.Indexing.DefaultBackgroundTaskExecuter.ExecuteAllInterleaved[T](WorkContext context, IList`1 result, Action`1 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\DefaultBackgroundTaskExecuter.cs:line 182

   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111

   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 259

   at Raven.Database.Server.HttpServer.<>c__DisplayClass43.<TryGetOrCreateResourceStore>b__41() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 1112

   at System.Threading.Tasks.Task`1.InnerInvoke()

   at System.Threading.Tasks.Task.Execute()

---> (Inner Exception #0) System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()<---

<---

"

2016-04-27 06:39:07.3846,Raven.Database.Server.HttpServer,Warn,Could not open database named: IGXContentStore,"System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)

   at Raven.Database.Indexing.DefaultBackgroundTaskExecuter.ExecuteAllInterleaved[T](WorkContext context, IList`1 result, Action`1 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\DefaultBackgroundTaskExecuter.cs:line 182

   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111

   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 259

   at Raven.Database.Server.HttpServer.<>c__DisplayClass43.<TryGetOrCreateResourceStore>b__41() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 1112

   at System.Threading.Tasks.Task`1.InnerInvoke()

   at System.Threading.Tasks.Task.Execute()

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)

   at System.Threading.Tasks.Task.Wait(TimeSpan timeout)

   at Raven.Database.Server.HttpServer.SetupRequestToProperDatabase(IHttpContext ctx) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 977

---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)

   at Raven.Database.Indexing.DefaultBackgroundTaskExecuter.ExecuteAllInterleaved[T](WorkContext context, IList`1 result, Action`1 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\DefaultBackgroundTaskExecuter.cs:line 182

   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111

   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 259

   at Raven.Database.Server.HttpServer.<>c__DisplayClass43.<TryGetOrCreateResourceStore>b__41() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 1112

   at System.Threading.Tasks.Task`1.InnerInvoke()

   at System.Threading.Tasks.Task.Execute()

---> (Inner Exception #0) System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()<---

<---

"

2016-04-27 06:39:07.4002,Raven.Database.Server.HttpServer,Debug,"Request #194,462: GET     - 4,177 ms -            - 503 - /site/ContentStore/databases/IGXContentStore/docs/Raven/Replication/Destinations",

2016-04-27 06:39:10.4783,Raven.Database.Server.HttpServer,Warn,Error on request,"System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)

   at Raven.Database.Indexing.DefaultBackgroundTaskExecuter.ExecuteAllInterleaved[T](WorkContext context, IList`1 result, Action`1 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\DefaultBackgroundTaskExecuter.cs:line 182

   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111

   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 259

   at Raven.Database.Server.HttpServer.<>c__DisplayClass43.<TryGetOrCreateResourceStore>b__41() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 1112

   at System.Threading.Tasks.Task`1.InnerInvoke()

   at System.Threading.Tasks.Task.Execute()

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)

   at Raven.Database.Server.HttpServer.SetupRequestToProperDatabase(IHttpContext ctx) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 1014

   at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 783

   at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 678

---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)

   at Raven.Database.Indexing.DefaultBackgroundTaskExecuter.ExecuteAllInterleaved[T](WorkContext context, IList`1 result, Action`1 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\DefaultBackgroundTaskExecuter.cs:line 182

   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111

   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 259

   at Raven.Database.Server.HttpServer.<>c__DisplayClass43.<TryGetOrCreateResourceStore>b__41() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 1112

   at System.Threading.Tasks.Task`1.InnerInvoke()

   at System.Threading.Tasks.Task.Execute()

---> (Inner Exception #0) System.IO.IOException: The following processes are locking C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\AllowedSecurityDescriptorsByGroup\_0.cfs

 w3wp 7172

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.RegenerateMapReduceIndex(Directory directory, String indexName, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 414

   at Raven.Database.Indexing.IndexStorage.TryRecoveringIndex(String indexName, IndexDefinition indexDefinition, Directory luceneDirectory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 301

   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 190

   at System.Threading.Tasks.Task.Execute()<---

<---"

4) The database then starts up again:

2016-04-27 06:39:29.1974,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,"Document with key 'Raven/Databases/IGXContentStore' was found, etag: 01000000-0000-0003-0000-000000000003",
2016-04-27 06:39:29.3380,Raven.Database.DocumentDatabase,Debug,Start loading the following database: IGXContentStore,
2016-04-27 06:39:29.3536,Raven.Database.Impl.DocumentCacher,Info,"MemoryCache Settings:
  PhysicalMemoryLimit = 99
  CacheMemoryLimit    = 16105078784
  PollingInterval     = 00:02:00",
2016-04-27 06:39:29.3536,Raven.Storage.Esent.TransactionalStorage,Info,"Esent Settings:
  MaxVerPages      = 65536
  CacheSizeMax     = 262144
  DatabasePageSize = 4096",

5) After index definitions are repeated in the log we then start to get the familiar .crash-marker in use IO exception repeatedly untilan IIS app pool recycle is performed:

2016-04-27 06:39:30.4943,Raven.Database.Indexing.IndexStorage,Warn,Could not create index storage,"System.IO.IOException: The process cannot access the file 'C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\indexing.crash-marker' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111
   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 237"
2016-04-27 06:39:30.4943,Raven.Database.DocumentDatabase,Error,Could not create database,"System.IO.IOException: The process cannot access the file 'C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\indexing.crash-marker' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111
   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 237"
2016-04-27 06:39:30.4943,Raven.Database.DocumentDatabase,Debug,Start shutdown the following database: IGXContentStore,
2016-04-27 06:39:30.4943,Raven.Database.Indexing.WorkContext,Debug,Stopping background workers,
2016-04-27 06:39:30.7756,Raven.Database.DocumentDatabase,Debug,Finished shutdown the following database: IGXContentStore,
2016-04-27 06:39:30.7756,Raven.Database.Server.HttpServer,Warn,Failed to create database IGXContentStore,"System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The process cannot access the file 'C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\indexing.crash-marker' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111
   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 259
   at Raven.Database.Server.HttpServer.<>c__DisplayClass43.<TryGetOrCreateResourceStore>b__41() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 1112
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.IO.IOException: The process cannot access the file 'C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\indexing.crash-marker' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111
   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 259
   at Raven.Database.Server.HttpServer.<>c__DisplayClass43.<TryGetOrCreateResourceStore>b__41() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 1112
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()<---
"
2016-04-27 06:39:30.7756,Raven.Database.Server.HttpServer,Warn,Could not open database named: IGXContentStore,"System.AggregateException: One or more errors occurred. ---> System.IO.IOException: The process cannot access the file 'C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\indexing.crash-marker' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111
   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 259
   at Raven.Database.Server.HttpServer.<>c__DisplayClass43.<TryGetOrCreateResourceStore>b__41() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 1112
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait(TimeSpan timeout)
   at Raven.Database.Server.HttpServer.SetupRequestToProperDatabase(IHttpContext ctx) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 977
---> (Inner Exception #0) System.IO.IOException: The process cannot access the file 'C:\igxsites\v90\site\DB\Databases\IgxContentStore\Indexes\indexing.crash-marker' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at Raven.Database.Indexing.IndexStorage..ctor(IndexDefinitionStorage indexDefinitionStorage, InMemoryRavenConfiguration configuration, DocumentDatabase documentDatabase) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 111
   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 259
   at Raven.Database.Server.HttpServer.<>c__DisplayClass43.<TryGetOrCreateResourceStore>b__41() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 1112
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()<---
"

Note: each time this .crash-marker in use exception occurs it dumps the IGXContentStore database and attempts to restart it on next hit, compiling the index definitions and throwing the same exception.

6) After an IIS app pool recycle the index definitions are compiled and we just get this exception:

2016-04-27 06:55:37.7065,Raven.Database.Indexing.IndexStorage.Startup,Warn,"Could not open index AllowedSecurityDescriptorsByGroup. Recovery operation failed, forcibly resetting index","System.InvalidOperationException: Index does not exists: AllowedSecurityDescriptorsByGroup
   at Raven.Database.Indexing.IndexStorage.OpenOrCreateLuceneDirectory(IndexDefinition indexDefinition, String indexName, Boolean createIfMissing) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 395
   at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 138"

Which creates and indexes the index and operation resumes as normal.

tl;dr is there appears to be a System.ComponentModel.Composition.CompositionException in Raven.Database.Indexing.IndexStorage.CreateIndexImplementation that causes the DB to dump, but which does not clean up file locks that are in use for indexing.

I can send the whole log if needed. 

We'll watch for another instance to see if it's the same index that crashes the db.

Any insight as to what might be happening would be greatly appreciated.

Thanks,

Kyle

[Oren Eini (Ayende Rahien)]

Do you have a an AnalyzerGenerator (or any plugins there) ?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

[Kyle Levien]

Looking into it, we do appear to have two plugins. A Lucene.Net.Analysis.Analyzer and a Raven.Database.Plugins.AbstractAnalyzerGenerator.

The analyzerGenerator doesn't look like anything particularly special:

        public class FullTextSearchAnalyzerGenerator : AbstractAnalyzerGenerator

{

public override Analyzer GenerateAnalyzerForIndexing(string indexName, Lucene.Net.Documents.Document document, Analyzer previousAnalyzer)

{

Analyzer specialAnalyzer = getSpecialAnalyzer(indexName, previousAnalyzer);

return specialAnalyzer;

}

public override Analyzer GenerateAnalyzerForQuerying(string indexName, string query, Analyzer previousAnalyzer)

{

Analyzer specialAnalyzer = getSpecialAnalyzer(indexName, previousAnalyzer);

return specialAnalyzer;

}

private static Analyzer getSpecialAnalyzer(string indexName, Analyzer previousAnalyzer)

{

Analyzer specialAnalyzer = previousAnalyzer;

if (indexName == "PageEntityFullTextSearchIndex" || indexName == "PageVersionsFullTextSearchIndex")

specialAnalyzer = new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30);

else if (indexName == "PageFieldsTextValueIndex")

specialAnalyzer = new KeywordAnalyzer();

return specialAnalyzer;

}

}

And the lucene Analyzer is an nGramAnalyzer that doesn't appear to be referenced.

...

[Oren Eini (Ayende Rahien)]

From the error, something there caused an error when creating it.

I suggest removing the analyzer generator at all, since you don't appear to be using it for something that cannot be done in the index directly.

--