FileNotFoundException: Could not load file or assembly 'raven.metrics'

[Michael Davis]

We have our application's app pool set to recycle at midnight every night.  Twice in the last week or so, after the scheduled recycle, something happened that caused all RavenDB operations to fail on the client side because it can't load the "raven.metrics" assembly.  This will continue until we manually recycle the app pool again.

Here's an example of the exception we're getting:

System.TypeInitializationException: The type initializer for 'Raven.Abstractions.Extensions.JsonExtensions' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'raven.metrics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=37f41c7f99471593' or one of its dependencies. The system cannot find the file specified.

   at Raven.Abstractions.Extensions.JsonExtensions..cctor()

   --- End of inner exception stack trace ---

   at Raven.Json.Linq.RavenJValue.WriteTo(JsonWriter writer, JsonConverterCollection converters)

   at Raven.Json.Linq.RavenJObject.WriteTo(JsonWriter writer, JsonConverterCollection converters)

   at Raven.Json.Linq.RavenJObject.WriteTo(JsonWriter writer, JsonConverterCollection converters)

   at Raven.Json.Linq.RavenJArray.WriteTo(JsonWriter writer, JsonConverterCollection converters)

   at Raven.Abstractions.Connection.JsonContent.SerializeToStreamAsync(Stream stream, TransportContext context)

   at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context)

   at Raven.Abstractions.Connection.CompressedContent.<SerializeToStreamAsync>d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()

   at Raven.Client.Connection.Implementation.HttpJsonRequest.<>c__DisplayClasse.<<SendRequestInternal>b__d>d__10.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()

   at Raven.Client.Connection.Implementation.HttpJsonRequest.<RunWithAuthRetry>d__15`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.Async.AsyncServerClient.<>c__DisplayClass1fe.<<BatchAsync>b__1fc>d__201.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()

   at Raven.Client.Connection.ReplicationInformerBase`1.<TryOperationAsync>d__29`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()

   at Raven.Client.Connection.ReplicationInformerBase`1.<ExecuteWithReplicationAsync>d__19`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.Async.AsyncServerClient.<ExecuteWithReplication>d__2c6`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

   at Raven.Client.Document.Async.AsyncDocumentSession.<SaveChangesAsync>d__d1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at ECrashAR.Core.Database.DatabaseSession.<SaveChangesAsync>d__7.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at ECrashAR.Web.Controllers.Api.CrashReportController.<UploadCrashReportCore>d__17.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Polly.Policy.<>c__DisplayClass13`1.<<ExecuteAsync>b__12>d__15.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Polly.Retry.RetryPolicy.<ImplementationAsync>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Polly.Policy.<ExecuteAsync>d__18`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

   at ECrashAR.Web.Controllers.Api.CrashReportController.<UploadCrashReport>d__16.MoveNext()

Any ideas what might be going on here?  I might be able to come up with a workaround to automatically recycle the app pool if an exception like this occurs, but I'd prefer not to need something like that.

[Oren Eini (Ayende Rahien)]

Can you make sure that your binaries are all from the same version?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Davis]

Raven.Abstractions.dll and Raven.Client.Lightweight.dll are both 3.0.30156.0 in our application.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Can you verify that your project all reference the same version?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Michael Davis]

The references on all of our DLLs say this in dotPeek:
Raven.Abstractions, Version=3.0.0.0, Culture=neutral, PublicKeyToken=37f41c7f99471593
Raven.Client.Lightweight, Version=3.0.0.0, Culture=neutral, PublicKeyToken=37f41c7f99471593

That matches what's on the Raven DLLs themselves (the version number I posted before was the file version, not the assembly version).

[Oren Eini (Ayende Rahien)]

Is this reproducible in any manner?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Michael Davis]

I'm not sure how we would, aside from just repeatedly recycling the app pool and hoping it occurs again (which isn't something I'd want to do during normal business hours).  This application has been live for over a year, and we've only seen this twice (once last Friday and again today), so we don't really have a good feel for how likely it is for this to continue happening.  So far, I haven't seen anything out of the ordinary going on around midnight that might explain what could have caused the problem on either of those occasions.

If I remember correctly, you're using Fody/Costura to embed the metrics assembly into one of the others, right?  Are you aware of any circumstances where loading the embedded assembly might fail?

[Oren Eini (Ayende Rahien)]

Only if there are multiple competing versions.

Can you try doing:

GC.KeepAlive(typeof(Raven.Imports.metrics.Metrics))

GC.KeepAlive(typeof(Raven.Abstractions.Extensions.JsonExtensions));

In the App_Start ?

That will force it to load it upfront

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Michael Davis]

I don't have a direct reference to the raven.metrics assembly, so the compiler can't find the type Raven.Imports.metrics.Metrics.  I think I'd have to manually extract it from the Raven.Abstractions assembly to do that.

I tried just putting Assembly.Load("raven.metrics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=37f41c7f99471593") at the start of App_Start, and that caused the exception to occur every time (I'm assuming because Raven.Abstractions hadn't been loaded, so the module initialization code that hooks into assembly resolution hadn't been run yet).  Then I tried putting GC.KeepAlive(typeof(Raven.Abstractions.Extensions.JsonExtensions)) before that to get it to go ahead and load Raven.Abstractions first, and doing it in that order seems to work.

Since I'm still not sure why it fails some of the time, I'm looking at putting this in there to try to make the application restart if it fails again:

            try
            {
                GC.KeepAlive(typeof(Raven.Abstractions.Extensions.JsonExtensions));
                Assembly.Load("raven.metrics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=37f41c7f99471593");
            }
            catch (Exception)
            {
                HttpRuntime.UnloadAppDomain();
                throw;
            }

Is there any reason why that might not be a good way to try and deal with this?

[Oren Eini (Ayende Rahien)]

Can you send us a sample project that reproduces it each time?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Kjersti Berg]

Hi

We experienced a burst of these exceptions on a production system recently, for no apparent reason. It affected two out of four server instances, and an app pool recycle seemed to solve the problem, for now. We have upgraded clients to 3.0.30155 recently. Do you have any idea why this exception could occur? The dll that fails to load has a different key than the one mentioned above, see stack trace below. 

System.InvalidOperationException: Query failed. See inner exception for details. ---> System.TypeInitializationException: The type initializer for 'Raven.Abstractions.Extensions.JsonExtensions' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'metrics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=ca6c6ef570198eba' or one of its dependencies. The system cannot find the file specified. at Raven.Abstractions.Extensions.JsonExtensions..cctor() --- End of inner exception stack trace --- at Raven.Abstractions.Extensions.JsonExtensions.JsonDeserialization[T](RavenJToken self) at Raven.Client.Connection.SerializationHelper.ToQueryResult(RavenJObject json, Etag etag, String tempRequestTime, Int64 numberOfCharactersRead) in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\SerializationHelper.cs:line 136 at Raven.Client.Connection.Async.AsyncServerClient.<>c__DisplayClass1ce.<<QueryAsyncAsGet>b__1ca>d__1d3.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\Async\AsyncServerClient.cs:line 1359 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Raven.Client.Connection.ReplicationInformerBase`1.<TryOperationAsync>d__29`1.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\ReplicationInformerBase.cs:line 442 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Raven.Client.Connection.ReplicationInformerBase`1.<ExecuteWithReplicationAsync>d__19`1.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\ReplicationInformerBase.cs:line 334 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Raven.Client.Connection.Async.AsyncServerClient.<ExecuteWithReplication>d__2c6`1.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\Async\AsyncServerClient.cs:line 0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Raven.Abstractions.Util.AsyncHelpers.<>c__DisplayClassb`1.<<RunSync>b__8>d__d.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Abstractions\Util\AsyncHelpers.cs:line 75 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Raven.Abstractions.Util.AsyncHelpers.RunSync[T](Func`1 task) in c:\Builds\RavenDB-Stable-3.0\Raven.Abstractions\Util\AsyncHelpers.cs:line 90 at Raven.Client.Connection.ServerClient.Query(String index, IndexQuery query, String[] includes, Boolean metadataOnly, Boolean indexEntriesOnly) in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\ServerClient.cs:line 279 --- End of inner exception stack trace --- at Raven.Client.Connection.ServerClient.Query(String index, IndexQuery query, String[] includes, Boolean metadataOnly, Boolean indexEntriesOnly) in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\ServerClient.cs:line 286 at Raven.Client.Document.AbstractDocumentQuery`2.ExecuteActualQuery() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Document\AbstractDocumentQuery.cs:line 685 at Raven.Client.Document.AbstractDocumentQuery`2.get_QueryResult() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Document\AbstractDocumentQuery.cs:line 655 at Raven.Client.Linq.RavenQueryProviderProcessor`1.ExecuteQuery[TProjection]() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Linq\RavenQueryProviderProcessor.cs:line 1635 at Raven.Client.Linq.RavenQueryInspector`1.GetEnumerator() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Linq\RavenQueryInspector.cs:line 101 at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

Adding this line at the top of App_start consistently reproduces that exception. 

Assembly.Load("metrics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=ca6c6ef570198eba");

Adding this line after seems to fix the issue. 

GC.KeepAlive(typeof(Raven.Abstractions.Extensions.JsonExtensions));

Why would this exception only occur occasionally, and how can it hit two different servers at the same time, when this is the client dll's? 

Do you have a working theory? Could we upgrade/downgrade the client? 

Thanks for any light you can shed on the incident.

Cheers,

Kjersti Berg

[Michael Davis]

I was never able to reproduce this in a sample project reliably.  I ended up just writing a PowerShell script to extract the assembly to disk during deployment, and that seems to have solved the problem for us (at least, it hasn't occurred again since then).

Function Extract-Assembly {
    param (
        [string]$Path,
        [string]$Name
    )
    $Path = Resolve-Path $Path
    $folder = Split-Path $Path
    $asm = [System.Reflection.Assembly]::LoadFile($Path)
    $in = $asm.GetManifestResourceStream("costura.$Name.dll.zip")
    $in = New-Object -TypeName System.IO.Compression.DeflateStream -ArgumentList @($in, [System.IO.Compression.CompressionMode]::Decompress)
    $out = New-Object -TypeName System.IO.FileStream -ArgumentList @((Join-Path $folder "$Name.dll"),[System.IO.FileMode]::Create)
    $in.CopyTo($out)
    $in.Close()
    $out.Close()
}

Extract-Assembly -Path "path-to-website\bin\Raven.Abstractions.dll" -Name "raven.metrics"

-- You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/xz3sZzDMgEk/unsubscribe. To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.

[Phil Sandler]

Was this problem ever resolved?  Was a bug with this discovered resulting in a fix?  We are randomly getting the same error in one environment using version 3.0.30171.

[Oren Eini (Ayende Rahien)]

You are getting that on the client side?

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Oren Eini CEO   /   Hibernating Rhinos LTD Mobile:  +972-52-548-6969 Sales:  sa...@ravendb.net Skype:  ayenderahien Support:  sup...@ravendb.net

[Phil Sandler]

Yes, on the client side.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Oren Eini (Ayende Rahien)]

Are you running the latest client code? In what env are you running this?

Is this reproducible for you?

[Phil Sandler]

The version we are running is 3.0.30171.

We are running in Windows 2012 R2.  The application runs in an IIS process.  We have two dev servers running the same code base, and the problem seems to only occur in one (because it's dev it's not heavily monitored).

I don't know of any way to reproduce consistently.  As the OP suggested, it may be possible if we keep stopping and starting the app pool until it happens.

Back to my original question: was this ever logged as a bug and addressed/fixed?  Is it a known problem?

[Oren Eini (Ayende Rahien)]

This was fixed, AFAIR.

Try adding:

GC.KeepAlive(typeof(Metrics)) 

In the application startup, which should resolve it regardless 

[Phil Sandler]

Thanks, will give this a try.