Indexing issues after crash

[Vlad Kosarev]

So Azure crapped out today and once it got back up Raven started throwing this error nonstop, what can be done?

2015-03-17 21:22:10.7419,Raven.Database.Indexing.AbstractIndexingExecuter,Error,Failed to execute indexing,"System.IndexOutOfRangeException: Index was outside the bounds of the array.

   at Lucene.Net.Index.SegmentTermDocs.Next() in c:\Work\lucene.net\src\core\Index\SegmentTermDocs.cs:line 180

   at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(IndexReader reader, Int32 docIDStart) in c:\Work\lucene.net\src\core\Index\DocumentsWriter.cs:line 1319

   at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(SegmentInfos infos) in c:\Work\lucene.net\src\core\Index\DocumentsWriter.cs:line 1268

   at Lucene.Net.Index.IndexWriter.ApplyDeletes() in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 5367

   at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 4377

   at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 4197

   at Lucene.Net.Index.IndexWriter.GetReader(Int32 termInfosIndexDivisor) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 389

   at Raven.Database.Indexing.Index.RecreateSearcher() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\Index.cs:line 717

   at Raven.Database.Indexing.Index.Write(Func`4 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\Index.cs:line 472

   at Raven.Database.Indexing.MapReduceIndex.ReduceDocuments.ExecuteReduction() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\MapReduceIndex.cs:line 593

   at Raven.Database.Indexing.IndexStorage.Reduce(String index, AbstractViewGenerator viewGenerator, IEnumerable`1 mappedResults, Int32 level, WorkContext context, IStorageActionsAccessor actions, HashSet`1 reduceKeys, Int32 inputCount) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 1024

   at Raven.Database.Indexing.ReducingExecuter.<>c__DisplayClass4f.<SingleStepReduce>b__44(IStorageActionsAccessor actions) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\ReducingExecuter.cs:line 317

   at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action`1 action, EsentTransactionContext transactionContext) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 703

   at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 647

   at Raven.Database.Indexing.ReducingExecuter.SingleStepReduce(IndexToWorkOn index, String[] keysToReduce, AbstractViewGenerator viewGenerator, ConcurrentSet`1 itemsToDelete) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\ReducingExecuter.cs:line 320

   at Raven.Database.Indexing.ReducingExecuter.HandleReduceForIndex(IndexToWorkOn indexToWorkOn) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\ReducingExecuter.cs:line 51

   at Raven.Database.Indexing.DefaultBackgroundTaskExecuter.ExecuteAllInterleaved[T](WorkContext context, IList`1 result, Action`1 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\DefaultBackgroundTaskExecuter.cs:line 175

   at Raven.Database.Indexing.AbstractIndexingExecuter.ExecuteIndexing(Boolean isIdle, Boolean& onlyFoundIdleWork) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\AbstractIndexingExecuter.cs:line 237

   at Raven.Database.Indexing.AbstractIndexingExecuter.Execute() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\AbstractIndexingExecuter.cs:line 46"

2015-03-17 21:22:10.7419,Raven.Database.Indexing.AbstractIndexingExecuter,Warn,Could not flush indexes properly,"System.IndexOutOfRangeException: Index was outside the bounds of the array.

   at Lucene.Net.Index.SegmentTermDocs.Next() in c:\Work\lucene.net\src\core\Index\SegmentTermDocs.cs:line 180

   at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(IndexReader reader, Int32 docIDStart) in c:\Work\lucene.net\src\core\Index\DocumentsWriter.cs:line 1319

   at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(SegmentInfos infos) in c:\Work\lucene.net\src\core\Index\DocumentsWriter.cs:line 1268

   at Lucene.Net.Index.IndexWriter.ApplyDeletes() in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 5367

   at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 4377

   at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 4197

   at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 4032

   at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 4116

   at Raven.Database.Indexing.Index.Flush(Etag highestETag) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\Index.cs:line 266

   at Raven.Database.Indexing.IndexStorage.FlushReduceIndexes() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 1246

   at Raven.Database.Indexing.AbstractIndexingExecuter.<Execute>b__0() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\AbstractIndexingExecuter.cs:line 106"

[Oren Eini (Ayende Rahien)]

Reset the infex

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vlad Kosarev]

Tried to reset an index and this error happened and now index disappeared (getting no such index errors and number of indexes decreased) but it still shows up in UI. When I try to edit it I just get 'could not find' error.

2015-03-17 21:26:14.2996,Raven.Database.Server.HttpServer,Warn,Error on request,"System.IO.IOException: The following processes are locking S:\Databases\LEAP\Indexes\SalesIndex\write.lock

 w3wp 4216

   at Raven.Database.Extensions.IOExtensions.TryHandlingError(String directory, Int32 i, Exception e) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 115

   at Raven.Database.Extensions.IOExtensions.DeleteDirectory(String directory) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Extensions\IOExtensions.cs:line 65

   at Raven.Database.Indexing.IndexStorage.DeleteIndex(String name) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\IndexStorage.cs:line 840

   at Raven.Database.DocumentDatabase.DeleteIndex(String name) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 1543

   at Raven.Database.DocumentDatabase.ResetIndex(String index) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 2231

   at Raven.Database.Server.Responders.Index.Respond(IHttpContext context) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\Responders\Index.cs:line 63

   at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 816

   at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Server\HttpServer.cs:line 678"

[Vlad Kosarev]

Had to reboot, looks like file system was locking that file for whatever reason and Raven couldn't do anything.

[Chris Marisic]

On Tuesday, March 17, 2015 at 4:23:58 PM UTC-5, Vlad Kosarev wrote:

So Azure crapped out today 

What was this running in? A VM, cloud service, website? 

[Vlad Kosarev]

It was on VM and Azure storage failed yesterday. It messed up raven but after a bunch of reboots and hours of raven reindexing I think we are back to normal.

[Chris Marisic]

Now that's a different story. The consistency of the document store was likely impacted and resulted in raven having to do a deep recovery to prevent dataloss.

[Tobbe]

Just got this error on our staging environment after server restart and resetting index solved the issue. We're about to restart our live-servers soon, is there some new about this error or anything we might look up before to check if we'll have the same problem there?

We're using virtual machines with Windows Server 2012 R2 and RavenDb Server Build #30153.

[Oren Eini (Ayende Rahien)]

This is from over a year ago.

This issue is typically related to some sort of index corruption, sometimes after hard reset.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.