Another Temp path already used error, suddenly started

[James Tan]

On a dev server, it has about 10 instances and all running well, a week ago, all start thrown error: Temp path already used error, on the /System/Data file. After security patch and server restart.

If copied the instances to another server, all work fine, also with latest patch.

Windows 2012R2, installed latest security patch, but could not figure out where is the issue.

Enabled the debug log, found it is started cleanup and shutdown for no reason and then got problem. See attached file.

Where is the issue?

Thanks

James

[James Tan]

This is a really strange issue that I tried all kind of way and log. The only thing I can see, from the log, RavenDB was trying to initial the instance twice. 

The instance works fine for 1- couple minutes and then crash like this. Build 2.5.2956

DTC problem? Bug? Concurrency issue?

2015-03-08 13:38:08.5868,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,<system>,Performing Transactions Cleanup Sequence for db <system>,

2015-03-08 13:38:18.3898,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,eLtss,Performing Transactions Cleanup Sequence for db eLtss,

2015-03-08 13:38:38.5973,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,<system>,Performing Transactions Cleanup Sequence for db <system>,

2015-03-08 13:38:48.4000,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,eLtss,Performing Transactions Cleanup Sequence for db eLtss,

time,logger,level,database,message,exception

2015-03-08 13:39:00.5694,Raven.Web.ForwardToRavenRespondersFactory,Info,,Setting up RavenDB Http Integration to the ASP.Net Pipeline,

2015-03-08 13:39:00.9288,Raven.Database.DocumentDatabase,Debug,<system>,Start loading the following database: <system>,

2015-03-08 13:39:01.1163,Raven.Database.Impl.DocumentCacher,Info,<system>,"MemoryCache Settings:

  PhysicalMemoryLimit = 98

  CacheMemoryLimit    = 1071644672

  PollingInterval     = 00:02:00",

2015-03-08 13:39:01.1319,Raven.Storage.Esent.TransactionalStorage,Info,<system>,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 261888

  DatabasePageSize = 4096",

time,logger,level,database,message,exception

2015-03-08 13:39:04.8460,Raven.Web.ForwardToRavenRespondersFactory,Info,,Setting up RavenDB Http Integration to the ASP.Net Pipeline,

2015-03-08 13:39:05.4094,Raven.Database.DocumentDatabase,Debug,<system>,Start loading the following database: <system>,

2015-03-08 13:39:05.4875,Raven.Database.Impl.DocumentCacher,Info,<system>,"MemoryCache Settings:

  PhysicalMemoryLimit = 98

  CacheMemoryLimit    = 1071644672

  PollingInterval     = 00:02:00",

2015-03-08 13:39:05.5188,Raven.Storage.Esent.TransactionalStorage,Info,<system>,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 261888

  DatabasePageSize = 4096",

2015-03-08 13:39:08.7372,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,<system>,Performing Transactions Cleanup Sequence for db <system>,

2015-03-08 13:39:18.4097,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,eLtss,Performing Transactions Cleanup Sequence for db eLtss,

2015-03-08 13:39:38.7508,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,<system>,Performing Transactions Cleanup Sequence for db <system>,

2015-03-08 13:39:48.4079,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,eLtss,Performing Transactions Cleanup Sequence for db eLtss,

2015-03-08 13:39:49.2673,Raven.Web.ForwardToRavenRespondersFactory,Info,,Disposing of RavenDB Http Integration to the ASP.Net Pipeline,

2015-03-08 13:39:49.2829,Raven.Database.Server.HttpServer,Info,,Shutting down database eLtss because we are shutting down the server,

2015-03-08 13:39:49.2829,Raven.Database.DocumentDatabase,Debug,,Start shutdown the following database: eLtss,

2015-03-08 13:39:49.2829,Raven.Database.Indexing.WorkContext,Debug,,Stopping background workers,

2015-03-08 13:39:49.3767,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Warn,,Error when trying to open a new DocumentStorageActions,"Microsoft.Isam.Esent.Interop.EsentInvalidInstanceException: Invalid instance handle

[James Tan]

Seemed duplicated at: https://groups.google.com/forum/#!topic/ravendb/ARGR5GjgLiE

On Sunday, March 8, 2015 at 2:52:21 PM UTC-4, James Tan wrote:

[Maxim Buryak]

James,

From the information you have given me, I see three different issues:

The first issue you were talking about is the EsentTempPathAlreadyUsed exception, in order to investigate it we will need more data, from the machine in which it happens at. I did not see any traces for that exception in last two logs you sent.

The second issue you were talking about, that was documented in the log you attached is about AD inner exception, does it still happen, can you check the event viewer for possible details?

The third issue is with the EsentInvalidInstanceException, which happens when we turn off the db in the middle of idexing. Interestingly, right before it, we see the log line "Disposing of RavenDB Http Integration to the ASP.Net Pipeline" which means that the app is being shut down by the iis. - It is important to understand why the IIS is shutting us down, we need more information about that too.

Best Regards,

Hibernating Rhinos Ltd  

Maxim Buryak l Core Team Developer l Mobile:+972-54-217-7751

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Tan]

I have a dump file for the failing instance, about 120MB compressed. Will this help if I can upload it to you?

Thanks

James

[Oren Eini (Ayende Rahien)]

Yes

[James Tan]

I shared the dump file through dropbox, 80MB .7z file.

Thanks

James

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/4tk7oqAnODk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[James Tan]

This issue is happening on another 2012R2 server. Now it is a serious problem.  Please let me know if you need more information.

Thanks

James

James

Yes

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/4tk7oqAnODk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

[Maxim Buryak]

Hi James,

I've been investigating the dump you've sent me, but did not find anything suspicious.

Can you please share the unedited log files from the time the dump was taken?

Best Regards,

Hibernating Rhinos Ltd  

Maxim Buryak l Core Team Developer l Mobile:+972-54-217-7751

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

[jta...@gmail.com]

Maxim,

I shared the log file and debugdiag output (dmp analysis report) through dropbox.

Thanks

James

Sent from Windows Mail

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[James Tan]

More on this. I managed to reproduce this issue under configuration below:

MSDN downloaded: en_windows_server_2012_r2_with_update_x64_dvd_6052708.iso

which Includes November 2014 update roll up, please review KB3000850 for more information. 

Setup new VM under virtualbox, with web server and app server, .NET 4.5 features enabled.

Copied and tested the RavenDB instance that had issue, got the same problem.

Then setup a brand new empty 2.5b2956 instance and also got the same error, just wait for couple minutes. The log file is attached.

Thanks

James

[Nick Champion]

Hi All,

I am also seeing this problem. I installed windows update KB3000850 on my development environment which is Windows Server 2012 R2 on 13th march and since then have been getting frequent temp path errors for the system database, often triggered by simply writing / updating a document.

I have had detailed logging on since this started, here are the logs from the time this last occurred, very similar to James'.

2015-03-17 12:22:40.1442,Raven.Database.Indexing.Index.Indexing,Debug,Marketplace,Scheduled to touch documents: users/11:01000000-0000-0008-0000-000000000001,

2015-03-17 12:22:40.1712,Raven.Database.Indexing.Index.Indexing,Debug,Marketplace,Indexed 1 documents for 22,

2015-03-17 12:22:40.2082,Raven.Database.Indexing.AbstractIndexingExecuter,Debug,Marketplace,"After indexing 1 documents, the new last etag for is: 01000000-0000-0008-0000-000000000003 for Refunds",

2015-03-17 12:22:40.3472,Raven.Database.Indexing.Index.Indexing,Debug,Marketplace,Indexed 3 documents for 1,

2015-03-17 12:22:40.3472,Raven.Database.Indexing.AbstractIndexingExecuter,Debug,Marketplace,"After indexing 3 documents, the new last etag for is: 01000000-0000-0008-0000-000000000003 for Raven/DocumentsByEntityName",

2015-03-17 12:22:40.3472,Raven.Database.Indexing.AbstractIndexingExecuter,Debug,,"Executing task: Index: 22, References: users/11",

2015-03-17 12:22:40.3472,Raven.Database.Tasks.TouchReferenceDocumentIfChangedTask,Debug,,"Going to touch the following documents (LoadDocument references, need to check for concurrent transactions): [users/11, 01000000-0000-0008-0000-000000000001]",

2015-03-17 12:22:40.3472,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,,"Document with key 'users/11' was found, etag: 01000000-0000-0008-0000-000000000001",

2015-03-17 12:22:40.3472,Raven.Database.Tasks.TouchReferenceDocumentIfChangedTask,Debug,,"Don't need to touch users/11, etag 01000000-0000-0008-0000-000000000001 is the same as when we last saw it",

2015-03-17 12:22:40.3472,Raven.Database.Indexing.WorkContext,Debug,,Incremented work counter to 6 because: WORK BY IndexingExecuter,

2015-03-17 12:22:40.3472,Raven.Database.Indexing.WorkContext,Debug,Marketplace,"No work was found, workerWorkCounter: 6, for: ReducingExecuter, will wait for additional work",

2015-03-17 12:22:40.4232,Raven.Database.Indexing.WorkContext,Debug,,"No work was found, workerWorkCounter: 6, for: IndexingExecuter, will wait for additional work",

2015-03-17 12:22:42.4213,Raven.Database.DocumentDatabase,Debug,<system>,Start loading the following database: <system>,

2015-03-17 12:22:44.5454,Raven.Database.Impl.DocumentCacher,Info,<system>,"MemoryCache Settings:

  PhysicalMemoryLimit = 99

  CacheMemoryLimit    = 4371513344

  PollingInterval     = 00:02:00",

2015-03-17 12:22:44.5734,Raven.Database.Storage.Esent.StorageConfigurator,Info,<system>,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 262144

  DatabasePageSize = 4096",

2015-03-17 12:22:44.5734,Raven.Storage.Esent.TransactionalStorage,Info,<system>,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 262144

  DatabasePageSize = 4096",

2015-03-17 12:22:45.1914,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,Marketplace,Performing Transactions Cleanup Sequence for db Marketplace,

2015-03-17 12:22:46.3774,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,<system>,Performing Transactions Cleanup Sequence for db <system>,

2015-03-17 12:23:15.1917,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,Marketplace,Performing Transactions Cleanup Sequence for db Marketplace,

2015-03-17 12:23:16.3778,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,<system>,Performing Transactions Cleanup Sequence for db <system>,

2015-03-17 12:23:45.1921,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,Marketplace,Performing Transactions Cleanup Sequence for db Marketplace,

2015-03-17 12:23:46.3941,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,<system>,Performing Transactions Cleanup Sequence for db <system>,

2015-03-17 12:24:11.8751,Raven.Database.DocumentDatabase,Debug,<system>,Start loading the following database: <system>,

2015-03-17 12:24:12.6941,Raven.Database.Impl.DocumentCacher,Info,<system>,"MemoryCache Settings:

  PhysicalMemoryLimit = 99

  CacheMemoryLimit    = 4371513344

  PollingInterval     = 00:02:00",

2015-03-17 12:24:12.7051,Raven.Database.Storage.Esent.StorageConfigurator,Info,<system>,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 262144

  DatabasePageSize = 4096",

2015-03-17 12:24:12.7051,Raven.Storage.Esent.TransactionalStorage,Info,<system>,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 262144

  DatabasePageSize = 4096",

2015-03-17 12:24:15.2042,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,Marketplace,Performing Transactions Cleanup Sequence for db Marketplace,

2015-03-17 12:24:16.3943,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,<system>,Performing Transactions Cleanup Sequence for db <system>,

2015-03-17 12:24:39.1502,Raven.Database.Indexing.WorkContext,Debug,<system>,"No work was found, workerWorkCounter: 0, for: ReducingExecuter, will wait for additional work",

2015-03-17 12:24:39.2112,Raven.Database.Indexing.WorkContext,Debug,<system>,"No work was found, workerWorkCounter: 0, for: IndexingExecuter, will wait for additional work",

2015-03-17 12:24:45.2054,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,Marketplace,Performing Transactions Cleanup Sequence for db Marketplace,

2015-03-17 12:24:46.3954,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,<system>,Performing Transactions Cleanup Sequence for db <system>,

2015-03-17 12:25:00.6450,Raven.Database.DocumentDatabase,Debug,<system>,Start loading the following database: <system>,

2015-03-17 12:25:01.1681,Raven.Database.Impl.DocumentCacher,Info,<system>,"MemoryCache Settings:

  PhysicalMemoryLimit = 99

  CacheMemoryLimit    = 4371513344

  PollingInterval     = 00:02:00",

2015-03-17 12:25:01.1740,Raven.Database.Storage.Esent.StorageConfigurator,Info,<system>,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 262144

  DatabasePageSize = 4096",

2015-03-17 12:25:01.1740,Raven.Storage.Esent.TransactionalStorage,Info,<system>,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 262144

  DatabasePageSize = 4096",

Thanks

[Nick Champion]

Bit more info.

This is happening every few minutes for me after IIS resetting. I am not seeing any temp path errors in the raven logs but am seeing them in the event log. I am on build 3628.

This seems like a pretty serious problem, fortunately I have not patched my production servers recently so its only happening on my development environment, but its rendered the database pretty useless.

Here is another section of my log file you can clearly see the system database being loaded twice within seconds of each other.

2015-03-17 12:50:56.8509,Raven.Database.Indexing.WorkContext,Debug,,Incremented work counter to 11 because: WORK BY IndexingExecuter,

2015-03-17 12:50:56.8509,Raven.Database.Indexing.WorkContext,Debug,Audit,"No work was found, workerWorkCounter: 11, for: ReducingExecuter, will wait for additional work",

2015-03-17 12:50:56.8509,Raven.Database.Indexing.WorkContext,Debug,,"No work was found, workerWorkCounter: 11, for: IndexingExecuter, will wait for additional work",

2015-03-17 12:51:01.6920,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,<system>,Performing Transactions Cleanup Sequence for db <system>,

2015-03-17 12:51:03.4971,Raven.Database.DocumentDatabase,Debug,<system>,Start loading the following database: <system>,

2015-03-17 12:51:03.9601,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,Marketplace,Performing Transactions Cleanup Sequence for db Marketplace,

2015-03-17 12:51:06.1132,Raven.Database.Impl.DocumentCacher,Info,<system>,"MemoryCache Settings:

  PhysicalMemoryLimit = 99

  CacheMemoryLimit    = 4371513344

  PollingInterval     = 00:02:00",

2015-03-17 12:51:06.1562,Raven.Database.Storage.Esent.StorageConfigurator,Info,<system>,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 262144

  DatabasePageSize = 4096",

2015-03-17 12:51:06.1562,Raven.Storage.Esent.TransactionalStorage,Info,<system>,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 262144

  DatabasePageSize = 4096",

2015-03-17 12:51:21.5217,Raven.Database.DocumentDatabase,Debug,<system>,Start loading the following database: <system>,

2015-03-17 12:51:21.9627,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,Audit,Performing Transactions Cleanup Sequence for db Audit,

2015-03-17 12:51:22.5017,Raven.Database.Impl.DocumentCacher,Info,<system>,"MemoryCache Settings:

  PhysicalMemoryLimit = 99

  CacheMemoryLimit    = 4371513344

  PollingInterval     = 00:02:00",

2015-03-17 12:51:22.5188,Raven.Database.Storage.Esent.StorageConfigurator,Info,<system>,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 262144

  DatabasePageSize = 4096",

2015-03-17 12:51:22.5188,Raven.Storage.Esent.TransactionalStorage,Info,<system>,"Esent Settings:

  MaxVerPages      = 16384

  CacheSizeMax     = 262144

  DatabasePageSize = 4096",

2015-03-17 12:51:31.7062,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,<system>,Performing Transactions Cleanup Sequence for db <system>,

2015-03-17 12:51:33.9722,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,Marketplace,Performing Transactions Cleanup Sequence for db Marketplace,

2015-03-17 12:51:51.9708,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,Audit,Performing Transactions Cleanup Sequence for db Audit,

2015-03-17 12:52:01.7222,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,<system>,Performing Transactions Cleanup Sequence for db <system>,

2015-03-17 12:52:03.9732,Raven.Database.Impl.DTC.InFlightTransactionalState,Info,Marketplace,Performing Transactions Cleanup Sequence for db Marketplace,

And here is the error I am seeing in the event log

Event code: 3005 

Event message: An unhandled exception has occurred. 

Event time: 3/17/2015 12:51:06 PM 

Event time (UTC): 3/17/2015 12:51:06 PM 

Event ID: d73fe7a058df446cac9079e2c6b6e6ee 

Event sequence: 2 

Event occurrence: 1 

Event detail code: 0 

 

Application information: 

    Application domain: /LM/W3SVC/8/ROOT-2-130710702622550832 

    Trust level: Full 

    Application Virtual Path: / 

    Application Path: C:\Dev\Repositories\Reebonz\raven\ 

    Machine name: DEV-2012 

 

Process information: 

    Process ID: 5484 

    Process name: w3wp.exe 

    Account name: IIS APPPOOL\dev-raven.reebonz.com 

 

Exception information: 

    Exception type: TargetInvocationException 

    Exception message: Exception has been thrown by the target of an invocation.

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)

   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)

   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)

   at Owin.Loader.DefaultLoader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder)

   at Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup)

   at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint()

   at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)

   at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context)

   at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)

   at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)

   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)

   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

Could not open transactional storage: C:\Dev\Repositories\Reebonz\raven\Database\System\Data

   at Raven.Storage.Esent.TransactionalStorage.Initialize(IUuidGenerator uuidGenerator, OrderedPartCollection`1 documentCodecs) in c:\Builds\RavenDB-3.0-Unstable\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 505

   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState recievedTransportState) in c:\Builds\RavenDB-3.0-Unstable\Raven.Database\DocumentDatabase.cs:line 191

   at Raven.Database.Server.RavenDBOptions..ctor(InMemoryRavenConfiguration configuration, DocumentDatabase db) in c:\Builds\RavenDB-3.0-Unstable\Raven.Database\Server\RavenDBOptions.cs:line 54

   at Owin.AppBuilderExtensions.UseRavenDB(IAppBuilder app, InMemoryRavenConfiguration configuration) in c:\Builds\RavenDB-3.0-Unstable\Raven.Database\Server\AppBuilderExtensions.cs:line 45

Temp path already used by another database instance

   at Raven.Storage.Esent.TransactionalStorage.Initialize(IUuidGenerator uuidGenerator, OrderedPartCollection`1 documentCodecs) in c:\Builds\RavenDB-3.0-Unstable\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 493

 

 

Request information: 

    Request URL: http://dev-raven.reebonz.com/databases/Marketplace/docs?id=SystemInformation 

    Request path: /databases/Marketplace/docs 

    User host address: 127.0.0.1 

    User:  

    Is authenticated: False 

    Authentication Type:  

    Thread account name: IIS APPPOOL\dev-raven.reebonz.com 

 

Thread information: 

    Thread ID: 15 

    Thread account name: IIS APPPOOL\dev-raven.reebonz.com 

    Is impersonating: False 

    Stack trace:    at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)

   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)

   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)

   at Owin.Loader.DefaultLoader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder)

   at Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup)

   at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint()

   at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)

   at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context)

   at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)

   at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)

   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)

   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

 

 

Custom event details: 

[Oren Eini (Ayende Rahien)]

Can you change the index temp path?

--

[Nick Champion]

you mean the compiled index cache path? this was set to C:\temp\Raven\CompiledIndexCache

Nick Champion

Co-Founder

Code Trip

Online Marketplaces

www.codetrip.co.uk

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/4tk7oqAnODk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Paweł Pekról]

Nick, can you move Assemblies directory out of IIS application folder (maybe to C:\temp\Assemblies)? You can do it by changing the 'Raven/AssembliesDirectory' in web.config.

P.

[Nick Champion]

I wonder if this is somehow related to virtualization - the machine I have seen this on is a virtual machine running on VMWare Fusion and James mentioned he's seen this on VM's running on virtualbox.

Has anyone experienced it running in a non virtualized environment?

On Tuesday, March 17, 2015 at 1:41:33 PM UTC, Nick Champion wrote:

you mean the compiled index cache path? this was set to C:\temp\Raven\CompiledIndexCache

Nick Champion

Co-Founder

On Tue, Mar 17, 2015 at 1:15 PM, Oren Eini (Ayende Rahien) <aye...@ayende.com> wrote:

Can you change the index temp path?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/4tk7oqAnODk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

[Ken Dale]

We're having this issue with an Azure VM currently hosting RavenDB 2763 with IIS. The event log has a number of EsentTempPathInUseException related entries.

Nick Champion

Co-Founder

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/4tk7oqAnODk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[James Tan]

More info:

For the DEV servers with this issue, I rebuilt the system drive (with security patch till 8/2014) and imported all IIS configuration, and everything working fine then. It seems relate to security patch between 10/2014-2/2015 of 2012R2? Can someone also try this or what is your current patch level?

Thanks

James

[Maxim Buryak]

Hi All,

I'd like to try to remove all file interaction from hosted iis site:

1) I'd like you all to try and move the DB and the Logs location outside the hosted path.

2) James and Ken: can you try to set the web.config configuration to ravendb:

<add key="Raven/CompiledIndexCacheDirectory" value="[some path outside hosted site]" />

3) Nick, can you update us about your status? do you still get the Temp Path.. error? do you keep getting it when you move all files listed in (1) from your hosted path?

thanks for the cooperation, we're doing our best efforts to find the root of the problem and to find a workaround.

Best Regards,

Hibernating Rhinos Ltd  

Maxim Buryak l Core Team Developer l Mobile:+972-54-217-7751

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

[Mike Ruhlin]

I tried moving file locations, not sure if I got all the ones I need.  Here's what I have in web.config:

<appSettings>
    <add key="Raven/DataDir" value="~\..\RavenBuildDatabase\System" />
    <add key="Raven/AnonymousAccess" value="Admin" />
    <add key="Raven/Host" value="usw1build04.aws.pluck.com" />
    <add key="Raven/Port" value="8082" />
    <add key="Raven/CompiledIndexCacheDirectory" value="~\..\RavenCompiledIndexCache" />
    <add key="Raven/AssembliesDirectory" value="~\..\RavenAssemblies" />
  </appSettings>

This may be a coincidence, but behavior of the bug seems to have changed slightly.
Before, I could make read requests just fine until the first write request.  After that, all reads and writes would fail.  Didn't matter whether I was writing via my application or raven studio.  First write would always kill everything.  All read or write requests after that fail.
After moving file locations, my app makes one write on startup (creating the database, since it's not there anymore), then gets this error on all subsequent read or write requests.  Funny thing is I can read AND write via raven studio, even while my app is still getting the esent error.

[Mike Ruhlin]

Hold on.  Saying something publicly is the best way to realize how stupid you're being.
My app is failing for other reasons (some documents that our installer puts in the database aren't there because I thought I didn't need to go through that part... app fails when they're not there) and I'm misreading old log messages as if they were new.  Yeah, all indicators are that moving the file locations fixed the problem.

[Chris Marisic]

http://en.wikipedia.org/wiki/Rubber_duck_debugging strikes

[James Tan]

Maxim,

I have one question, why the debug log did not have this error? I saw it has log statement but never been called, or that thread already dead?

I am still getting temp path error after the file moved and configuration changed. (I did it for all 3 instances on the test VM to different folders)

<add key="Raven/DataDir" value="~\..\RavenDB25.Database\System"/>

<add key="Raven/CompiledIndexCacheDirectory" value="~\..\RavenDB25.CompiledIndexCache" />

The error message showed the location of DB was moved. Also attached the log file.

Temp path already used by another database instance

[InvalidOperationException: Could not open transactional storage: D:\projects\ltssdebug\InstanceEmpty\RavenDB25\..\RavenDB25.Database\System\Data]

Thanks

James

[Oren Eini (Ayende Rahien)]

Make sure to also have Raven/Assemblies in the config too

And the issue is that RavenDB it being killed by IIS, so we don't have the _time_ to output something.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

[Nick]

Hi Maxim,

Sorry for the slow response, since moving all paths external to the IIS folder I have not experienced the error.

I will let you know

Thanks

Nick

[Mike Ruhlin]

Oren, Is that Raven/Assemblies instead of Raven/AssembliesDirectory? Or both of them?

[James Tan]

It works. Add this configuration and moved all these folder to totally different folder. It needs to be not under IIS at all, moving out of Web application or to upper level folder is not enough.

<add key="Raven/DataDir" value="D:\projects\RavenDBFiles\RavenDB25\RavenDB25.Database\System"/>

<add key="Raven/CompiledIndexCacheDirectory" value="D:\projects\RavenDBFiles\RavenDB25\RavenDB25.CompiledIndexCache" />

<add key="Raven/AssembliesDirectory" value="D:\projects\RavenDBFiles\RavenDB25\RavenDB25.Assemblies" />

Also, the response time seems much faster after this moving when running multiple instances.

Thanks

James

[Oren Eini (Ayende Rahien)]

Assembliesdirectory (on gmail, didn't check the full thing)

[Maxim Buryak]

Adam, did you manage to solve your problem using the next configuration changes:
1) Move the DB and the Logs location outside the hosted path, also change the web.config setting:

<add key="Raven/DataDir" value="[Some Path Outside The IIS ]"/>
2) Set the web.config configuration to ravendb:

<add key="Raven/CompiledIndexCacheDirectory" value="[some path outside hosted site]" />

Best Regards,

Hibernating Rhinos Ltd  

Maxim Buryak l Core Team Developer l Mobile:+972-54-217-7751

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

On Fri, Mar 20, 2015 at 7:37 PM, Adam Busbin <aw.b...@gmail.com> wrote:

I am also only seeing this issue in Virtualized environments, but not all of them.

[Adam Busbin]

Yup, this seems to have fixed it. Before I read this thread I also tried running it as a service and that worked as well. This was on build 2956. 

[James Tan]

Maxim,

From now, I can see this workaround works well, in various tests. What is the root case for this problem?  And is there any real fix could be done (RavenDB or .NET libraries and even OS/IIS)? Or we need to change the deployments/configuration starts now?

Thanks

James

[Oren Eini (Ayende Rahien)]

The underlying issue is that the recent patch caused IIS to be much more agressive with stealth recycle, which kill us.

The fix is to move things outside of the IIS directory

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

[James Tan]

Oren,

I tested to install and try some patches (from Oct,2014 to Feb. 2015) but this is time consuming and did not get there yet. Do you know which patch causes this issue, already?

Shall submit this as issue to MS?

Thanks

James

[Oren Eini (Ayende Rahien)]

We are pretty sure that the issue is here: KB3000850

The underlying reason is that the behavior of IIS changed when we modify dlls on the IIS directory.

It will do a forced AppDomain unload without letting us know, causing us to leak a connection

[Ken Dale]

It was OK for awhile, but now we are having the issue again with the following set:

<add key="Raven/DataDir" value="C:\RavenDB\Database\System"/>

<add key="Raven/CompiledIndexCacheDirectory" value="C:\RavenDB\CompiledIndexCache" />

[Michael Yarichuk]

has something changed in your hosting environment? anything in the logs?

Best regards,

 

Michael Yarichuk

RavenDB Core Team

Tel: 972-4-6227811

Fax:972-153-4-6227811

Email : michael....@hibernatingrhinos.com

 

[Oren Eini (Ayende Rahien)]

You also need to set Raven/Assemblies config value.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Wed, Mar 25, 2015 at 2:19 PM, Ken Dale <k...@kendaleiv.com> wrote:

[Ken Dale]

I don't see Raven/Assemblies on http://ravendb.net/docs/article-page/2.5/csharp/server/administration/configuration. Is there something special we need to do on the IIS side to get this to work (since, all the binaries will be outside of wwwroot)?

[Oren Eini (Ayende Rahien)]

No, once this happens, this is okay.

There are some config options that aren't documented, because they aren't generally usable except for very specific scenarios

[Adam Busbin]

This seems related.
http://issues.umbraco.org/issue/U4-6338

[James Tan]

I tried it. The attached hotfix (KB3052480) seems fixed the issue.

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/4tk7oqAnODk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Mark Kabana]

I also ran into this issue over the weekend after installing updates and can confirm that the hotfix KB3052480 i found here: https://support.microsoft.com/en-us/kb/3052480 seems to have fixed the issue.

Oren,

Moving forward, do you think that it is a good idea (and possibly a recommended practice) to move the DataDir, CompiledIndexCacheDirectory, and AssembliesDirectory outside of the root website folder?

Although the hotfix seems to have worked, I'm just thinking that if there are lots of file changes going on in these directories and that IIS is monitoring all these changes, that nothing good could ever come from having these files in the root website folder - it may be best just to move them outside of the folder for good to avoid any future issues and not have to worry about IIS getting in the way of the file changes raven makes to these folders.

What do you think and what would you suggest / recommend?

Let me know, thanks!

[Oren Eini (Ayende Rahien)]

Mark,

We already recommend that, and the installer and default behavior will enfore that.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--