LockObtainFailedException during indexing in build 2555

[Chris van Oijen]

Just filled a newly created database with 1.5 million records (about 1.3 kB each in size). When selecting the Collections view in the Management Studio the Raven/DocumentsByEntityName index is created and indexing starts. During indexing the following error occurs multiple times:

Raven/DocumentsByEntityName System.AggregateException: One or more errors occurred. ---> Lucene.Net.Store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@F:\Indexes\Databases\test2\Raven%2fDocumentsByEntityName\write.lock

   at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout) in c:\Work\lucene.net\src\core\Store\Lock.cs:line 83

   at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean create, IndexDeletionPolicy deletionPolicy, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit commit) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 1228

   at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, IndexDeletionPolicy deletionPolicy, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit commit) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 1203

   at Lucene.Net.Index.IndexWriter..ctor(Directory d, Analyzer a, IndexDeletionPolicy deletionPolicy, MaxFieldLength mfl) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 5441

   at Raven.Database.Indexing.RavenIndexWriter.CreateIndexWriter() in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Indexing\RavenIndexWriter.cs:line 127

   at Raven.Database.Indexing.Index.AddDocumentToIndex(RavenIndexWriter currentIndexWriter, Document luceneDoc, Analyzer analyzer) in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Indexing\Index.cs:line 644

   at Raven.Database.Indexing.SimpleIndex.<>c__DisplayClass19.<>c__DisplayClass22.<IndexDocuments>b__12(IEnumerator`1 partition) in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Indexing\SimpleIndex.cs:line 128

   at System.Threading.Tasks.Task.Execute()

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)

   at Raven.Database.Indexing.DefaultBackgroundTaskExecuter.ExecuteAllInterleaved[T](WorkContext context, IList`1 result, Action`1 action) in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Indexing\DefaultBackgroundTaskExecuter.cs:line 199

   at Raven.Database.Indexing.DefaultBackgroundTaskExecuter.ExecuteAllBuffered[T](WorkContext context, IList`1 source, Action`1 action) in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Indexing\DefaultBackgroundTaskExecuter.cs:line 108

   at Raven.Database.Indexing.SimpleIndex.<>c__DisplayClass19.<IndexDocuments>b__b(RavenIndexWriter indexWriter, Analyzer analyzer, IndexingWorkStats stats) in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Indexing\SimpleIndex.cs:line 167

   at Raven.Database.Indexing.Index.Write(Func`4 action) in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Indexing\Index.cs:line 364

---> (Inner Exception #0) Lucene.Net.Store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@F:\Indexes\Databases\test2\Raven%2fDocumentsByEntityName\write.lock

   at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout) in c:\Work\lucene.net\src\core\Store\Lock.cs:line 83

   at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean create, IndexDeletionPolicy deletionPolicy, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit commit) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 1228

   at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, IndexDeletionPolicy deletionPolicy, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit commit) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 1203

   at Lucene.Net.Index.IndexWriter..ctor(Directory d, Analyzer a, IndexDeletionPolicy deletionPolicy, MaxFieldLength mfl) in c:\Work\lucene.net\src\core\Index\IndexWriter.cs:line 5441

   at Raven.Database.Indexing.RavenIndexWriter.CreateIndexWriter() in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Indexing\RavenIndexWriter.cs:line 127

   at Raven.Database.Indexing.Index.AddDocumentToIndex(RavenIndexWriter currentIndexWriter, Document luceneDoc, Analyzer analyzer) in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Indexing\Index.cs:line 644

   at Raven.Database.Indexing.SimpleIndex.<>c__DisplayClass19.<>c__DisplayClass22.<IndexDocuments>b__12(IEnumerator`1 partition) in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Indexing\SimpleIndex.cs:line 128

   at System.Threading.Tasks.Task.Execute()<---

14-apr-2013 18:15:37

Indexing still continues after the errors. When adding a second index, the number of errors increase. At a certain point indexing seems to stop, but the indexes are reported as being stale.

Raven version: unstable-2555

[Oren Eini (Ayende Rahien)]

Something is locking those files.

Do you have AV, Indexing Services, etc running there?

--
You received this message because you are subscribed to the Google Groups "ravendb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Chris van Oijen]

I have AV running (no indexing services or other scanning stuff) but that never gave me problems in the past.

To exclude the AV as possible source I have re-run the index job after AV was disabled. It resulted in the same errors.

[Oren Eini (Ayende Rahien)]

Hm...

Can you run this with the debug log enabled? Anything in the log?

[Chris van Oijen]

I have ran the indexing with Raven.Server.exe in a console: no log messages. Also nothing in the eventlogs.

[Oren Eini (Ayende Rahien)]

But still the error?

[Chris van Oijen]

Yes. Nothing in the logs but still got the error messages. It is not entirely reproducable: apart from the fact that I get an error, the number of errors depends between 2 and 5. 

On stable 2330 it runs without any problem. If you want I can provide you with a dump. 

[Oren Eini (Ayende Rahien)]

A dump would be very useful

[Tamir Shlomi]

I'm having that error too, but my scenario is a bit different:

I'm having an existing database running with ~8M documents.

there is win service that generate data and push it to the database and web site that read it. the site stopped getting the results so I found that error at the logs (50 times).

I'm running single server over IIS, the build #2241 and the index is mine. not system index.

after restarting the IIS it was back to normal.

can it be something with the machine configuration? it is the same as the "recommended configuration".

any solution/suggestion from this group thread?

[Oren Eini (Ayende Rahien)]

This usually indicate that _something_ is locking the files.

[Chris van Oijen]

Short update: currently running on build #2577 with the same data and index definitions and the issue has not appeared again.

[Oren Eini (Ayende Rahien)]

great