System.InvalidOperationException: Custom counters file view is out of memory.

[Robert Labrie]

We run nearly 300 tenants, running Raven 2.5 as an IIS application. The app pool was restarted yesterday, and we're noticing this error for 8 of the tenants. Googling says this might do the trick:

<configuration> 
<system.diagnostics> 
<performanceCounters filemappingsize="33554432" /> 
</system.diagnostics> 
</configuration> 

And since we're not using the counters for anything, maybe this is better:
<configuration>
    <system.serviceModel>
        <diagnostics performanceCounters="Off" />
    </system.serviceModel>
</configuration>

Anyone ever experience the same or have any suggestions? We can't just reboot whenever this happens.

[WebException: The remote server returned an error: (503) Server Unavailable.]
   System.Net.HttpWebRequest.GetResponse() +6592536
   Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func`1 getResponse) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:349

[InvalidOperationException: Reason: "System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Custom counters file view is out of memory.\r\n   at System.Diagnostics.SharedPerformanceCounter.CalculateMemory(Int32 oldOffset, Int32 totalSize, Int32& alignmentAdjustment)\r\n   at System.Diagnostics.SharedPerformanceCounter.CreateInstance(CategoryEntry* categoryPointer, Int32 instanceNameHashCode, String instanceName, PerformanceCounterInstanceLifetime lifetime)\r\n   at System.Diagnostics.SharedPerformanceCounter.GetCounter(String counterName, String instanceName, Boolean enableReuse, PerformanceCounterInstanceLifetime lifetime)\r\n   at System.Diagnostics.SharedPerformanceCounter..ctor(String catName, String counterName, String instanceName, PerformanceCounterInstanceLifetime lifetime)\r\n   at System.Diagnostics.PerformanceCounter.InitializeImpl()\r\n   at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName, Boolean readOnly)\r\n   at Raven.Database.Indexing.WorkContext.SetupPerformanceCounter(String name) in c:\\Builds\\RavenDB-Stable\\Raven.Database\\Indexing\\WorkContext.cs:line 353\r\n   at Raven.Database.Indexing.WorkContext.Init(String name) in c:\\Builds\\RavenDB-Stable\\Raven.Database\\Indexing\\WorkContext.cs:line 390\r\n   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\\Builds\\RavenDB-Stable\\Raven.Database\\DocumentDatabase.cs:line 252\r\n   at Raven.Database.Server.HttpServer.<>c__DisplayClass4a.<TryGetOrCreateResourceStore>b__47() in c:\\Builds\\RavenDB-Stable\\Raven.Database\\Server\\HttpServer.cs:line 1049\r\n   at System.Threading.Tasks.Task`1.InnerInvoke()\r\n   at System.Threading.Tasks.Task.Execute()\r\n   --- End of inner exception stack trace ---\r\n   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)\r\n   at System.Threading.Tasks.Task.Wait(TimeSpan timeout)\r\n   at Raven.Database.Server.HttpServer.SetupRequestToProperDatabase(IHttpContext ctx) in c:\\Builds\\RavenDB-Stable\\Raven.Database\\Server\\HttpServer.cs:line 921\r\n---> (Inner Exception #0) System.InvalidOperationException: Custom counters file view is out of memory.\r\n   at System.Diagnostics.SharedPerformanceCounter.CalculateMemory(Int32 oldOffset, Int32 totalSize, Int32& alignmentAdjustment)\r\n   at System.Diagnostics.SharedPerformanceCounter.CreateInstance(CategoryEntry* categoryPointer, Int32 instanceNameHashCode, String instanceName, PerformanceCounterInstanceLifetime lifetime)\r\n   at System.Diagnostics.SharedPerformanceCounter.GetCounter(String counterName, String instanceName, Boolean enableReuse, PerformanceCounterInstanceLifetime lifetime)\r\n   at System.Diagnostics.SharedPerformanceCounter..ctor(String catName, String counterName, String instanceName, PerformanceCounterInstanceLifetime lifetime)\r\n   at System.Diagnostics.PerformanceCounter.InitializeImpl()\r\n   at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName, Boolean readOnly)\r\n   at Raven.Database.Indexing.WorkContext.SetupPerformanceCounter(String name) in c:\\Builds\\RavenDB-Stable\\Raven.Database\\Indexing\\WorkContext.cs:line 353\r\n   at Raven.Database.Indexing.WorkContext.Init(String name) in c:\\Builds\\RavenDB-Stable\\Raven.Database\\Indexing\\WorkContext.cs:line 390\r\n   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, TransportState transportState) in c:\\Builds\\RavenDB-Stable\\Raven.Database\\DocumentDatabase.cs:line 252\r\n   at Raven.Database.Server.HttpServer.<>c__DisplayClass4a.<TryGetOrCreateResourceStore>b__47() in c:\\Builds\\RavenDB-Stable\\Raven.Database\\Server\\HttpServer.cs:line 1049\r\n   at System.Threading.Tasks.Task`1.InnerInvoke()\r\n   at System.Threading.Tasks.Task.Execute()<---\r\n"

Could not open database named: sometenant
]
   Raven.Client.Connection.HttpJsonRequest.HandleErrors(WebException e) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:541
   Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func`1 getResponse) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:355
   Raven.Client.Connection.HttpJsonRequest.ReadResponseJson() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:239
   Raven.Client.Connection.ServerClient.DirectGet(OperationMetadata operationMetadata, String key, String transformer) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ServerClient.cs:220
   Raven.Client.Connection.<>c__DisplayClass1.<Get>b__0(OperationMetadata u) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ServerClient.cs:123
   Raven.Client.Connection.ReplicationInformer.TryOperation(Func`2 operation, OperationMetadata operationMetadata, OperationMetadata primaryOperationMetadata, Boolean avoidThrowing, T& result, Boolean& wasTimeout) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ReplicationInformer.cs:515
   Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication(String method, String primaryUrl, OperationCredentials primaryCredentials, Int32 currentRequest, Int32 currentReadStripingBase, Func`2 operation) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ReplicationInformer.cs:461
   Raven.Client.Connection.ServerClient.ExecuteWithReplication(String method, Func`2 operation) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ServerClient.cs:179
   Raven.Client.Connection.ServerClient.Get(String key) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ServerClient.cs:123
   Raven.Client.Document.DocumentSession.Load(String id) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\DocumentSession.cs:243
   Raven.Client.Document.DocumentSession.Load(ValueType id) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\DocumentSession.cs:287
   TNW.Core.TNWDocumentSession`1.Load(ValueType id) in c:\BuildAgent\work\aded08fac9df9d2d\core\TNW.Core\Connections\Raven\TNWDocumentSession.cs:58
   TNW.Core.Entities.ProductsDescription.ProductsDescription.LoadOrCreate(ITNWDocumentSession`1 documentSession) in c:\BuildAgent\work\aded08fac9df9d2d\core\TNW.Core\Entities\ProductsDescription\ProductsDescription.cs:106
   TNW.Core.Diagnostics.Probes.Products.CapaProductDiagnostic.Probe() in c:\BuildAgent\work\aded08fac9df9d2d\core\TNW.Core\Diagnostics\Probes\Products\CapaProductDiagnostic.cs:15
   TNW.Core.Diagnostics.ProbeBot.<Run>d__0.MoveNext() in c:\BuildAgent\work\aded08fac9df9d2d\core\TNW.Core\Diagnostics\ProbeBot\ProbeBot.cs:20
   System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) +381
   System.Linq.Enumerable.ToList(IEnumerable`1 source) +58
   Diagnostics.Controllers.DiagnosticsController.ExecuteTenantProbesForTenant(ConnectionBundle bundle) in c:\BuildAgent\work\aded08fac9df9d2d\platform\diagnostics\Site\Controllers\DiagnosticsController.cs:72
   Diagnostics.Controllers.<>c__DisplayClass10.<GetAndExecuteTenantProbes>b__e(ConnectionBundle bundle) in c:\BuildAgent\work\aded08fac9df9d2d\platform\diagnostics\Site\Controllers\DiagnosticsController.cs:83
   System.Action`1.Invoke(T obj) +0
   TNW.Core.<>c__DisplayClass19.<ConstructNamedTenantInvoker>b__15(String tenantName, Action`1 doThis) in c:\BuildAgent\work\aded08fac9df9d2d\core\TNW.Core\Connections\Tenants.cs:112
   TNW.Core.CommandLine.<>c__DisplayClass10.<GetTenantIterator>b__f(Action`1 action) in c:\BuildAgent\work\aded08fac9df9d2d\core\TNW.Core\CommandLine\MultitenancyHelper.cs:59
   Diagnostics.Controllers.DiagnosticsController.GetAndExecuteTenantProbes(String client) in c:\BuildAgent\work\aded08fac9df9d2d\platform\diagnostics\Site\Controllers\DiagnosticsController.cs:83
   Diagnostics.Controllers.DiagnosticsController.Tenant(String client) in c:\BuildAgent\work\aded08fac9df9d2d\platform\diagnostics\Site\Controllers\DiagnosticsController.cs:47
   lambda_method(Closure , ControllerBase , Object[] ) +104
   System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters) +14
   System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters) +214
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters) +27
   System.Web.Mvc.<>c__DisplayClass15.<InvokeActionMethodWithFilters>b__12() +55
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation) +253
   System.Web.Mvc.<>c__DisplayClass17.<InvokeActionMethodWithFilters>b__14() +21
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodWithFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor, IDictionary`2 parameters) +191
   System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName) +324
   System.Web.Mvc.Controller.ExecuteCore() +106
   System.Web.Mvc.ControllerBase.Execute(RequestContext requestContext) +91
   System.Web.Mvc.ControllerBase.System.Web.Mvc.IController.Execute(RequestContext requestContext) +10
   System.Web.Mvc.<>c__DisplayClassb.<BeginProcessRequest>b__5() +34
   System.Web.Mvc.Async.<>c__DisplayClass1.<MakeVoidDelegate>b__0() +19
   System.Web.Mvc.Async.<>c__DisplayClass8`1.<BeginSynchronous>b__7(IAsyncResult _) +10
   System.Web.Mvc.Async.WrappedAsyncResult`1.End() +62
   System.Web.Mvc.<>c__DisplayClasse.<EndProcessRequest>b__d() +48
   System.Web.Mvc.SecurityUtil.<GetCallInAppTrustThunk>b__0(Action f) +7
   System.Web.Mvc.SecurityUtil.ProcessInApplicationTrust(Action action) +22
   System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult) +60
   System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result) +9
   System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +9657896
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +155

[Ricardo Diaz]

Hey folks, I work with Robert, and we're not finding a lot of info on this.

It seems to somehow be connected to WCF? Is this because Raven uses Silverlight for the Studio?

Also, is there any way to "bounce" an individual tenant when something like this happens?

Thanks in advance,

Ricardo

[Oren Eini (Ayende Rahien)]

You can disable performance counters in RavenDB via Raven/DisablePerformanceCounters = false;

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Labrie]

What if we want to keep the performance counters enabled? I might want to use them in the future, without running out of memory.

[Chris Marisic]

Buy more memory?

[Robert Labrie]

Really? The box has 384 GB of ram, the overall memory utilization is under 50%. The links I found above suggested that the max memory for the performance counters is 512K, not MB or GB, but KB. What I'm looking for is a way to prevent an application specific out of memory error, not an offhand remark to just buy more memory.

Custom counters file view is out of memory.

Thanks,
Rob

[Oren Eini (Ayende Rahien)]

What build are you using?

Also, did you see the docs here:

http://msdn.microsoft.com/en-us/library/ms229387.aspx

You might need to change the machine.config

Note that by default, the size is 512Kb, which make sense if you are using 300 dbs.

[Robert Labrie]

Hi Oren,

Sorry for the delay and thanks for your reply. We upped it to 2MB and the problem went away. It's Raven 2.5.2750. The server shares with SQL, so maybe that's why we hit the max.

How can I find the performance counters anyway? I went into perfmon and looked for RavenDB but I couldn't find it.

Thanks,
Rob