EsentTempPathInUseException - How to avoid?

[BAM]

Hello,

I recently started to move a web app I’ve been developing locally into
our network development environment. As part of that process I’ve set
up Raven as an IIS application. When running against the IIS instance
I'm intermittently seeing the exception mentioned in the subject line.
One of my unit tests consistently sees the error.

The test is fairly straightforward:
1. Open session
a. Query for specific object
b. Assert object is null

2. Call an action on a controller which creates and persists the
object.

3. Open a new session
a. Query for specific object
b. Assert object is not null

Sessions for 1 and 3 are each in a using block. Session for 2 is
managed within the application's controller.

The test passes when connecting to my local instance of Raven.

If I run the test connecting to the IIS Raven it errors out at 3.a
with the following error.

[EsentTempPathInUseException: Temp path already used by another
database instance]
Microsoft.Isam.Esent.Interop.Api.Check(Int32 err) in C:\Work\ravendb
\SharedLibs\Sources\managedesent-61618\EsentInterop\Api.cs:2739
Microsoft.Isam.Esent.Interop.Api.JetInit(JET_INSTANCE&
instance) in C:\Work\ravendb\SharedLibs\Sources
\managedesent-61618\EsentInterop\Api.cs:131
Raven.Storage.Esent.TransactionalStorage.Initialize(IUuidGenerator
uuidGenerator) in c:\Builds\raven\Raven.Storage.Esent
\TransactionalStorage.cs:207

[InvalidOperationException: Could not open transactional storage: C:
\inetpub\wwwroot\Raven\Data\Data]
Raven.Storage.Esent.TransactionalStorage.Initialize(IUuidGenerator
uuidGenerator) in c:\Builds\raven\Raven.Storage.Esent
\TransactionalStorage.cs:220
Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration
configuration) in c:\Builds\raven\Raven.Database\DocumentDatabase.cs:
152
Raven.Web.ForwardToRavenRespondersFactory.Init() in c:\Builds\raven
\Raven.Web\ForwardToRavenRespondersFactory.cs:79
Raven.Web.RavenDbStartupAndShutdownModule.<Init>b__0(Object
sender, EventArgs args) in c:\Builds\raven\Raven.Web\BootStrapper.cs:
13

System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
+148
System.Web.HttpApplication.ExecuteStep(IExecutionStep step,
Boolean& completedSynchronously) +75


If I debug/step through the test connecting to IIS Raven it passes.

I'm using build 616.

Am I doing something wrong?

Thanks in advance.

B.

[Oren Eini (Ayende Rahien)]

Do you have any way of reproducing it?

Are you calling to a tenant database or to the master one?

[BAM]

Running the unit test I described above reliably reproduces it. Calls
involved there are to a tenant database.

Running the application against IIS Raven I see the error
sporadically. (FYI - All calls the application makes are to tenant
DBs.)

I never saw this error while working against a local Raven instance
using Raven.Server.exe.


On Feb 27, 4:09 pm, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>
wrote:

[Oren Eini (Ayende Rahien)]

Can you actually send us the code to generate this error?

The description you had allows for a lot of variations, which can be pretty important.

[BAM]

Understood.

I don't want to send my entire application.

Will try to find some time to create a pared down version and see what
the results are.

Thanks,

B.

On Feb 27, 4:29 pm, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>
wrote:

[Oren Eini (Ayende Rahien)]

Thanks

[BAM]

https://github.com/bryanvt/EsentTempPathInUseException

You will probably need to run the test twice. There's a problem where
an invoke thinks the method is null the first time through. I'll need
to figure that one out...

On Feb 28, 9:52 am, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>

[Oren Eini (Ayende Rahien)]

Hi,

How am I supposed to be running this?

I tried running the test, but it failed ( I run it 3 - 4 times, just to be sure):

Test 'Tests.TrmAdmin_CharacterizationControllerFixture.save_characterization_test' failed: System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.

  ----> System.NullReferenceException : Object reference not set to an instance of an object.

at System.RuntimeMethodHandle._InvokeMethodFast(IRuntimeMethodInfo method, Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeType typeOwner)

at System.RuntimeMethodHandle.InvokeMethodFast(IRuntimeMethodInfo method, Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeType typeOwner)

at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)

at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)

Config.cs(155,0): at Tests.Config.RavenControllerOnActionExecuting(RavenController controller)

TrmAdmin_CharacterizationControllerFixture.cs(52,0): at Tests.TrmAdmin_CharacterizationControllerFixture.save_characterization_test()

--NullReferenceException

at Temp.TrmAppHelper.GetRequestedDb(RouteValueDictionary routeValueDictionary)

at Temp.TrmAppHelper.GetCoreData(ControllerContext context)

at Temp.Controllers.RavenController.OnActionExecuting(ActionExecutingContext filterContext)

0 passed, 1 failed, 0 skipped, took 1.48 seconds (NUnit 2.5.10).

[BAM]

Odd. I get sometimes get that error but only once at the most. Let me
see if I can tweak a workaround for you.

Sorry.


On Feb 28, 6:54 pm, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>
wrote:

[BAM]

Oren,

First my apologies. I was going through the code I'd provided you. I
realized there was just too much going on. I need to re-factor and
simplify things. Then I'll see whether the error occurs any more.

That was one conclusion I came to. I also began to be more convinced
that the Esent error is caused, at root, by something I'm doing or not
in my code. The thing is I'm not sure what sort of thing I'm looking
for. So a couple questions:

1. What is your hunch about what might cause that error to occur?
e.g. Nested OpenSession calls (not that I think I'm doing that), or
some other bad thing?
2. When I go into Raven studio there are no log entries. Do I need to
do something to enable logging?

Again sorry to have brought you along on my snipe hunt.

B.

On Feb 28, 6:54 pm, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>
wrote:

[Oren Eini (Ayende Rahien)]

The most likely cause is that you are trying to open EmbeddedDocumentStore more than once.

[BAM]

Thanks. That's not impossible in the test units. I've seen the error
when running the app as well, and it is not using embedded.

What would be the most likely cause that doesn't involve embedded?

On Feb 29, 10:36 am, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>

[Oren Eini (Ayende Rahien)]

How are you running RavenDB outside of unit tests?

[BAM]

It's a multi-tenant application that connects to an IIS instance of
RavenDB. The IIS server is a development box within our network. I
have never had a problem establishing the connection.

DocumentStore instantiating code follows:

public static IDocumentStore GetDocumentStore(IContext
context)
{
lock (typeof(MvcApplication))
{
if (_documentStore == null)
{
_documentStore = new DocumentStore
{ ConnectionStringName = "RavenDB" };
_documentStore.Conventions.IdentityPartsSeparator
= "-";
_documentStore.Initialize();
}
}

return _documentStore;
}

Does that answer your question?

The error is intermittent and not predictable.

On Feb 29, 11:42 am, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>
wrote:

[Oren Eini (Ayende Rahien)]

Did you modify the indexpath or logspath?

[BAM]

The config settings? No. This is copied straight from the server:

<configuration>
<appSettings>
<add key="Raven/DataDir" value="~\Data"/>
<add key="Raven/AnonymousAccess" value="All"/>
</appSettings>
<system.webServer>
<handlers>
<add name="All" path="*" verb="*"
type="Raven.Web.ForwardToRavenRespondersFactory, Raven.Web"/>
</handlers>
<modules runAllManagedModulesForAllRequests="true">
<remove name="WebDAVModule" />
</modules>
</system.webServer>
<runtime>
<loadFromRemoteSources enabled="true"/>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<probing privatePath="Analyzers"/>
</assemblyBinding>
</runtime>
<system.web>
<customErrors mode="Off">
</customErrors>
</system.web>
</configuration>

On Mar 2, 3:08 pm, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>
wrote:

[Oren Eini (Ayende Rahien)]

Hm,

No idea then. If you can find a way to reproduce this, I would be most grateful.

I went over the code a couple of times, and it looks fine.

[BAM]

Thanks. I'm making progress toward a more targeted test. I'll let you
know when I think it's worth sharing.

It does _seem_ to be connected with scenarios where several sessions
are rapidly open and closed. As in the test I initially described.

On Mar 4, 5:57 am, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>
wrote:

[Mike Chaliy]

Just an idea.
When IIS recycles app pool it does not kill previous worker process
before new worker process is created.
It seems that for some time there are two phisical processes that
access to the same database directory.
You can disable this by Disable Overlapped Recycle in App Pool
Advanced Settings.

On 4 Бер, 12:57, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>

[Oren Eini (Ayende Rahien)]

Haven't thought of that, yes.

That might very well be a reason for that.

[BAM]

Mike,

Thanks for the idea. I've set up an App Pool with overlap disabled. So
far so good. I hesitate to jump to certainty, but it seems like you've
put your finger on the problem.

B.

[BAM]

All,

Just wanted to close this thread. It's been a couple of days since I
implemented Mike's suggestion. Since that time I have not seen the
error once. I feel confident that pool recycling was the root cause.

On a related note, before changing the recycling setting I regularly
would see errors in Studio. As it recovered after I closed the message
and wasn't affecting my application I did not take the time to
investigate. Since disabling recycling I have seen a significant
reduction (essentially eliminated) in the number of errors.

I would suggest y'all consider adding this as a recommendation for
configuration of IIS in your instructions, at the very least for multi-
tenant situations.

Thanks for all the help.

B.

On Mar 5, 3:43 am, "Oren Eini (Ayende Rahien)" <aye...@ayende.com>
wrote:

[Oren Eini (Ayende Rahien)]

BAN, we will, thanks.