Indexing not working this morning

[Josh Norris]

This has happened to me at least one other time. I woke up to find that I was unable to search any documents. I left the server logging on, so now I have a stack trace.

At any rate it seems to be at random. No searching available, but I can insert/update documents and can retrieve them by ID. Raven Studio is unresponsive this morning as well. Build#2375.

Keep in mind that the disk is not (remotely) full and no access rights have changed. Recycling the app pool in IIS has no affect. This is the first few lines in the log after the app pool recycle:

time,logger,level,message,exception

2013-07-10 09:26:38.1836,Raven.Web.ForwardToRavenRespondersFactory,Info,Setting up RavenDB Http Integration to the ASP.Net Pipeline,

2013-07-10 09:26:38.6672,Raven.Database.Impl.DocumentCacher,Info,"MemoryCache Settings:

  PhysicalMemoryLimit = 99

  CacheMemoryLimit    = 2146435072

  PollingInterval     = 00:02:00",

2013-07-10 09:26:38.6828,Raven.Storage.Esent.TransactionalStorage,Info,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 262144

  DatabasePageSize = 4096",

2013-07-10 09:26:41.2668,Raven.Database.Storage.IndexDefinitionStorage,Warn,"Could not compile index C:\Raven\Web\Database\System\IndexDefinitions\Raven%2fDocumentsByEntityName.index, skipping bad index","System.IO.FileNotFoundException: Could not find file 'C:\Raven\Web\Database\System\IndexDefinitions\TemporaryIndexDefinitionsAsSource\02bnxdtc.dll'.

File name: 'C:\Raven\Web\Database\System\IndexDefinitions\TemporaryIndexDefinitionsAsSource\02bnxdtc.dll'

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath)

   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)

   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)

   at Microsoft.CSharp.CSharpCodeGenerator.FromFileBatch(CompilerParameters options, String[] fileNames)

   at Microsoft.CSharp.CSharpCodeGenerator.FromSourceBatch(CompilerParameters options, String[] sources)

   at Microsoft.CSharp.CSharpCodeGenerator.System.CodeDom.Compiler.ICodeCompiler.CompileAssemblyFromSourceBatch(CompilerParameters options, String[] sources)

   at Raven.Database.Linq.QueryParsingUtils.Compile(String source, String name, String queryText, OrderedPartCollection`1 extensions, String basePath) in c:\Builds\RavenDB-Stable\Raven.Database\Linq\QueryParsingUtils.cs:line 312

   at Raven.Database.Linq.DynamicViewCompiler.GenerateInstance() in c:\Builds\RavenDB-Stable\Raven.Database\Linq\DynamicViewCompiler.cs:line 568

   at Raven.Database.Storage.IndexDefinitionStorage.AddAndCompileIndex(IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\IndexDefinitionStorage.cs:line 153

   at Raven.Database.Storage.IndexDefinitionStorage.ReadIndexesFromDisk() in c:\Builds\RavenDB-Stable\Raven.Database\Storage\IndexDefinitionStorage.cs:line 111"

2013-07-10 09:26:44.9088,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Backup/Status' was not found,

2013-07-10 09:26:44.9588,Raven.Database.Indexing.WorkContext,Debug,"No work was found, workerWorkCounter: 0, for: ReducingExecuter, will wait for additional work",

2013-07-10 09:26:44.9618,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Authorization/WindowsSettings' was not found,

2013-07-10 09:26:44.9618,Raven.Database.Indexing.WorkContext,Debug,"No work was found, workerWorkCounter: 0, for: IndexingExecuter, will wait for additional work",

2013-07-10 09:26:45.0718,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Databases/Email' was found,

2013-07-10 09:26:45.0718,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Databases/Email' was found,

2013-07-10 09:26:45.0718,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Databases/Email' was found,

2013-07-10 09:26:45.0718,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Databases/Email' was found,

2013-07-10 09:26:45.0718,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Databases/Email' was found,

2013-07-10 09:26:45.0718,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Databases/Email' was found,

2013-07-10 09:26:45.0718,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Databases/Email' was found,

2013-07-10 09:26:45.0718,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Databases/Email' was found,

2013-07-10 09:26:45.0718,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Databases/Email' was found,

2013-07-10 09:26:45.0718,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Databases/Email' was found,

2013-07-10 09:26:45.0908,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Databases/Email' was found,

2013-07-10 09:26:46.4298,Raven.Database.Impl.DocumentCacher,Info,"MemoryCache Settings:

  PhysicalMemoryLimit = 99

  CacheMemoryLimit    = 2146435072

  PollingInterval     = 00:02:00",

2013-07-10 09:26:46.4298,Raven.Storage.Esent.TransactionalStorage,Info,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 262144

  DatabasePageSize = 4096",

2013-07-10 09:26:47.4008,Raven.Database.Storage.IndexDefinitionStorage,Warn,"Could not compile index C:\Raven\Data\Email\Data\IndexDefinitions\EmailByContent.index, skipping bad index","System.IO.FileNotFoundException: Could not find file 'C:\Raven\Data\Email\Data\IndexDefinitions\TemporaryIndexDefinitionsAsSource\cge4mpri.dll'.

File name: 'C:\Raven\Data\Email\Data\IndexDefinitions\TemporaryIndexDefinitionsAsSource\cge4mpri.dll'

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath)

   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)

   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)

   at Microsoft.CSharp.CSharpCodeGenerator.FromFileBatch(CompilerParameters options, String[] fileNames)

   at Microsoft.CSharp.CSharpCodeGenerator.FromSourceBatch(CompilerParameters options, String[] sources)

   at Microsoft.CSharp.CSharpCodeGenerator.System.CodeDom.Compiler.ICodeCompiler.CompileAssemblyFromSourceBatch(CompilerParameters options, String[] sources)

   at Raven.Database.Linq.QueryParsingUtils.Compile(String source, String name, String queryText, OrderedPartCollection`1 extensions, String basePath) in c:\Builds\RavenDB-Stable\Raven.Database\Linq\QueryParsingUtils.cs:line 312

   at Raven.Database.Linq.DynamicViewCompiler.GenerateInstance() in c:\Builds\RavenDB-Stable\Raven.Database\Linq\DynamicViewCompiler.cs:line 568

   at Raven.Database.Storage.IndexDefinitionStorage.AddAndCompileIndex(IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\IndexDefinitionStorage.cs:line 153

   at Raven.Database.Storage.IndexDefinitionStorage.ReadIndexesFromDisk() in c:\Builds\RavenDB-Stable\Raven.Database\Storage\IndexDefinitionStorage.cs:line 111"

2013-07-10 09:26:47.4088,Raven.Database.Storage.IndexDefinitionStorage,Warn,"Could not compile index C:\Raven\Data\Email\Data\IndexDefinitions\Raven%2fDocumentsByEntityName.index, skipping bad index","System.IO.FileNotFoundException: Could not find file 'C:\Raven\Data\Email\Data\IndexDefinitions\TemporaryIndexDefinitionsAsSource\xzxb0t1s.dll'.

File name: 'C:\Raven\Data\Email\Data\IndexDefinitions\TemporaryIndexDefinitionsAsSource\xzxb0t1s.dll'

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath)

   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)

   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)

   at Microsoft.CSharp.CSharpCodeGenerator.FromFileBatch(CompilerParameters options, String[] fileNames)

   at Microsoft.CSharp.CSharpCodeGenerator.FromSourceBatch(CompilerParameters options, String[] sources)

   at Microsoft.CSharp.CSharpCodeGenerator.System.CodeDom.Compiler.ICodeCompiler.CompileAssemblyFromSourceBatch(CompilerParameters options, String[] sources)

   at Raven.Database.Linq.QueryParsingUtils.Compile(String source, String name, String queryText, OrderedPartCollection`1 extensions, String basePath) in c:\Builds\RavenDB-Stable\Raven.Database\Linq\QueryParsingUtils.cs:line 312

   at Raven.Database.Linq.DynamicViewCompiler.GenerateInstance() in c:\Builds\RavenDB-Stable\Raven.Database\Linq\DynamicViewCompiler.cs:line 568

   at Raven.Database.Storage.IndexDefinitionStorage.AddAndCompileIndex(IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\IndexDefinitionStorage.cs:line 153

   at Raven.Database.Storage.IndexDefinitionStorage.ReadIndexesFromDisk() in c:\Builds\RavenDB-Stable\Raven.Database\Storage\IndexDefinitionStorage.cs:line 111"

2013-07-10 09:26:47.4498,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Backup/Periodic/Setup' was found,

2013-07-10 09:26:47.4578,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Backup/Periodic/Status' was not found,

2013-07-10 09:26:47.4578,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'Raven/Backup/Status' was not found,

2013-07-10 09:26:47.4578,Raven.Database.Indexing.WorkContext,Debug,"No work was found, workerWorkCounter: 0, for: ReducingExecuter, will wait for additional work",

2013-07-10 09:26:47.4578,Raven.Database.Indexing.WorkContext,Debug,"No work was found, workerWorkCounter: 0, for: IndexingExecuter, will wait for additional work",

2013-07-10 09:26:47.7218,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'PRODWEB01DZlLkzg0nh000f7' was not found,

2013-07-10 09:26:47.7218,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'D83478453B9485448D98DA761F58EEB3' was not found,

2013-07-10 09:26:47.7218,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'D83478453B9485448D98DA761F58EEB3' was not found,

2013-07-10 09:26:47.7218,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key '36B4B96AEAADB24691212D83D69C033E' was not found,

2013-07-10 09:26:47.7218,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key '0AD62506F838A244B234C03B6DC92B01' was not found,

2013-07-10 09:26:47.7218,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'A42CD5F9F43DFE4B98BC75F4F1EBE867' was found,

[Oren Eini (Ayende Rahien)]

What build are you running?

Can you delete the IndexDefinitions\TemporaryIndexDefinitionsAsSource manually?

It looks like something is yanking the files while we are still using it.

--
You received this message because you are subscribed to the Google Groups "ravendb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Josh Norris]

2375.Yes, I can delete that folder manually (it is empty)

[Josh Norris]

Deleting that directory and restarting the app pool: same issue.

[Oren Eini (Ayende Rahien)]

Do you have any anti virus? indexing software?

Can you setup filemon and watch that directory?

[Josh Norris]

No indexing, no antivirus. Output from procmon is attached 

[Oren Eini (Ayende Rahien)]

Look here:

SetDispositionInformationFile

C:\Raven\Data\Email\Data\IndexDefinitions\TemporaryIndexDefinitionsAsSource\sp21wmrw.err

SUCCESS

Delete: True

This is marked as deleted on close, but I am not sure why.

[Josh Norris]

I've seen this same basic issue all over the internet, and the resolutions range from fixing security problems and cleaning out temp directories to rebooting the server. I suspect that a server reboot is my only option at this point. 

I would like to ask since this has happened to me at least twice in 4 months: Is it possible to not dynamically generate and load the assembly that the index definition is stored in? Would it be possible to simply tell raven to load the index definition from a precompiled assembly?

[Josh Norris]

As an FYI, the server reboot did bring back the server. It suddenly started up. However, once it was back up, the indexes were available to view in the studio, querying them gave me a "server not found" (basically a 404) when running a query against the index. Resetting the index solved the problem.

Alas, I'm afraid that I'm done with RavenDB. It's a great tool when it works. Going to check out ElasticSearch next.

Thanks all!

[Oren Eini (Ayende Rahien)]

Hi,

Yes, in 2.5 we have moved to a method that cache them on disk, and doesn't use temp files. Instead, it use persistent caching.

This should resolve your problem, and also give a nice boost.

[Oren Eini (Ayende Rahien)]

Josh,

I understand your frustration, but the probable reason for that is that the index was created & corrupted, leading to funkiness inside.

[Josh Norris]

Yep, it might be a corrupted index, but looking at the stack trace and the code and procmon it's much more likely that csc.exe is just unable to properly compile the index definition stored on disk when windows gets in this funky state. Even if it is a corrupted index, I can't have the index getting corrupted every 4 weeks. I ran SharePoint for a few years with a few million documents in it and got a corrupted index once? At any rate I'd like slow down the recent progression in my aging.

I really do like the product. It's super cool and super easy, but it just doesn't work in my environment. 

[Oren Eini (Ayende Rahien)]

Josh,

The likely reason for this is that you are running RavenDB in a mode that has "Allow service to interact with desktop". That means that logon / logoffs can trigger this.

See this article for details: http://blogs.msdn.com/b/andreal/archive/2009/04/18/could-not-find-file-c-windows-temp-dll.aspx

[Oren Eini (Ayende Rahien)]

Also, as I mentioned, 2.5 would resolve the problem for you.