Ninject failures under high load with ASP.NET MVC?

[Kendall Bennett]

Hi Guys,

We have been battling some really odd bugs in our system when under high load for some time, but never could track it down. Then recently we made some changes that caused Ninject to be used more heavily and earlier in the request pipeline on our web site, and immediately started to see some odd crashes, which when we examined the issue looked clearly like threading issues. Two threads were executing on the same Ninject resolved instance, which should never happen. We searched high and low, but never could find any issues in our code, and it finally led me to start to suspect Ninject. No matter what we do, we cannot reproduce the issue in a debug environment, and even on our staging server we had to heavily load the machine using Telerik Test Studio before we finally managed to have it fail in the same way that our live server does. We have been able to work around the crash issues in the code by putting threading locks in place, but it is only a stop gap measure, not a proper solution. And on our live server the issue only appears to happen once every couple of hours, or when the application pool is recycled. It seems to happen a lot more regularly when the load on the server is going up or down, indicating that new scope objects are getting created or destroyed.

We also updated Ninject from the 2.2 release we were using, to the latest stable release in NuGet (3.0.2 or something). It made no difference.

Near as I can figure out, when under load, Ninject sometimes serves up the WRONG instance to us, like it has scoped it wrong or something. We do not currently use request scope, but rather use application scope, which is a custom scope we made ourselves to bind the Ninject instances to instances of HttpApplication under ASP.NET. Since ASP.NET ensures that only one request at a time can be handled by a single application instance, it means we can avoid having to deal with threading issues within our code, and code it liked it is single threaded, as each application instance should end up getting their own set of objects, scoped to that application instance. More importantly, the instances remain across requests so we can cache things internally as necessary between requests for better performance. In our code we put some checks in to see if the application instance stored in the object Ninject returns to us is DIFFERENT to the current application instance, and then we log it. This happens every couple of hours, at the same time that we were normally seeing the crashes. So clearly Ninject at some point is giving us objects for the WRONG application instance, and it only happens under load. Which would indicate there is probably some kind of threading issue within the Ninject kernel (since it is a singleton instance used throughout the entire app).

Has anyone noticed anything similar before? I spoke briefly with the folks at Mindscape about the issues, as we have been using their new RayGun.IO project to capture the runtime errors and log information about the issues, and they mentioned they gave up using Ninject as they had seen some similar issues in the past and never worked it out. This is what he said:

"I can't help explicitly with your issue here but I will say that we've used Ninject several times and have run into the problem that it never disposes things properly -- even when using the InstancePerRequest model (or something like that -- it was a long time ago). What would happen is that you'd expect it to therefore give you clean items per request but in fact it wouldn't -- it would re-use them globally. This caused us a lot of issues until we figured out that it wasn't doing what you'd think.

It's a random comment but perhaps that's what is happening for you also? It's the sort of issue that only appeared for us once under load as well - on a dev machine or similar low load machine it was difficult to create weirdness."

Not sure if that was related to this or not, but now I am 99.9% sure something is wrong with Ninject. I plan to try using a different IoC container and see if it makes any difference, but I would really like to know if anyone else has seen anything similar to this, and if you have any suggestions on where I can look in the code to see where it might be going wrong?

Unfortunately because we cannot reproduce this problem in a debug environment, and only in a non-debug environment under load, it is really difficult to actually setup up a debug version of Ninject and try to follow the code through and figure out where it might be going wrong. But if someone can point out where in the code it does the scoping stuff, I could follow it through and see if I can spot anything.

BTW, the machine is running Windows Server 2008 R2, 64-bit, ASP.NET 4.5, .NET 4.5, ASP.NET MVC 4.

- Kendall

[Remo Gloor]

Hi Kendall

 

There are three things that have an influence on the instance returned to you:

1.       The same binding must be used to resolve an instance. If you have conditional bindings for a type they will return different instances even when having the same scope.

2.       The scope. Since you have a custom scope but didn’t provide any code, it is impossible to tell if you have implemented them correctly. You have to know that the instance that you return here decides which cached object will be returned. And that they are kept in the cache until the scope object is GC’d. In a high load environment is might be necessary to actively tell Ninject to remove objects cached by some scope. E.g. when using request scope we actively tell Ninject to remove the objects in this request after the request is completed. Otherwise you can get out of memory exceptions because the GC loop takes too long. Also be aware that your scope implementation must be thread safe. Ninject will access it concurrently from all threads that you use to resolve objects.

3.       The object must be resolved from the same kernel instance. Depending on your implementation you are either using one shared kernel or one for each app pool instance. In the second case the resolved objects are per app pool instance and you cannot expect to reuse them between app pools

 

Probably you should add code to show how you implemented your scope. I think there is a high change that the problem is in there.

 

Btw, caching is done in the cache: https://github.com/ninject/ninject/blob/master/src/Ninject/Activation/Caching/Cache.cs

 

Remo

--
You received this message because you are subscribed to the Google Groups "ninject" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninject+u...@googlegroups.com.
To post to this group, send email to nin...@googlegroups.com.
Visit this group at http://groups.google.com/group/ninject.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Sergii Sakharov]

Could be a bit offtopic, but you can try sth like http://blogs.microsoft.co.il/blogs/sasha/archive/2008/05/28/programmatically-generating-a-dump-file.aspx on live and then inspect the dump using windbg to figure out what has actually gone wrong. At some point we had a concurrency issue around rebinds(which are not thread safe) and it was a pain to identify the problem. Now, looking back, I think that having auto dump creation around specific exceptions might have saved us a bit of time as opposed to logging more and more info (although in the end we actually managed to reproduce it on dev machine).

Good luck,

Sergii

[Kendall Bennett]

Hi Remo,

There are three things that have an influence on the instance returned to you:

1.       The same binding must be used to resolve an instance. If you have conditional bindings for a type they will return different instances even when having the same scope.

When you say the same binding must be used to resolve an instance, what exactly do you mean? We have a single, static kernel instance that is used throughout the code to resolve bindings, and it is used by the ASP.NET MVC code and also by our own code. All the bindings are set up in one place, once for the single Ninject kernel instance. We use Kernel.Get<IBlah>() to get the instances we need manually, and from what I can work out the same essentially happens inside the ASP.NET MVC NinjectDependencyResolver (although it goes through the Bootstrapper class, but all it does when you create it is return the same single static instance; not any different to a static global variable?). So I think everything goes through Kernel.Get<>() for all the instances we need, and they should all be bound in only one place. I have posted our binding code below.

2.       The scope. Since you have a custom scope but didn’t provide any code, it is impossible to tell if you have implemented them correctly. You have to know that the instance that you return here decides which cached object will be returned.

Yes, I am aware of that. I have my own custom scopes for request scope and application scope, since the original method in Ninject 2.2 of passing in request scope is gone (using callbacks). So the code we use to define the scopes are:

namespace GoodTimeHobbies.WebScaffold.Scaffold

{

    public class ScopeCallbacks

    {

        /// <summary>

        /// Gets the callback for HttpApplication scope.

        /// </summary>

        public static readonly Func<IContext, object> HttpApplication = ctx => HttpContext.Current.ApplicationInstance;

        /// <summary>

        /// Gets the callback for Http request scope.

        /// </summary>

        public static readonly Func<IContext, object> Request = ctx => HttpContext.Current;

    }

}

Our request scope should pretty much resolve to the same code used by Ninject when you bind stuff to request scope with .InRequestScope(). Since I am posting code, here is the way we bind stuff:

namespace GoodTimeHobbies.DataAccess.Scaffold

{

    /// <summary>

    /// Class to configures how abstract service types are mapped to concrete implementations 

    /// when using a real database.

    /// </summary>

    public class DataAccessModule : NinjectModule

    {

        // Scope callback function to return the scope object used for our bindings

        private readonly Func<IContext, object> _depotScopeCallback;

        private readonly Func<IContext, object> _couchBaseScopeCallback;

        private readonly string _appConfigFile;

        private readonly string _siteConfigFile;

        /// <summary>

        /// Constructor to create the module

        /// </summary>

        /// <param name="depotScopeCallback">Scope to use for all depot bindings</param>

        /// <param name="couchBaseScopeCallback">Scope to use for CouchBase binding</param>

        public DataAccessModule(

            Func<IContext, object> depotScopeCallback,

            Func<IContext, object> couchBaseScopeCallback,

            string appConfigFile,

            string siteConfigFile)

        {

            _depotScopeCallback = depotScopeCallback;

            _couchBaseScopeCallback = couchBaseScopeCallback;

            _appConfigFile = appConfigFile;

            _siteConfigFile = siteConfigFile;

        }

        /// <summary>

        /// Overloaded function to set up all the bindings

        /// </summary>

        public override void Load()

        {

            // Bind our system shims to insulate classes from system functions

            Bind<IDateTimeShim>().To<DateTimeShim>().InSingletonScope();

            Bind<IRandomShim>().To<RandomShim>().InSingletonScope();

            // Make sure the Couchbase factory is bound to application instance scope

            Bind<ICouchbaseFactory>().To<CouchbaseFactory>().InScope(_couchBaseScopeCallback);

            // Bind the data access class

            Bind<IAccessNamesDepot>().To<AccessNamesDepot>().InScope(_depotScopeCallback);

            Bind<IOrderStatusDepot>().To<OrderStatusDepot>().InScope(_depotScopeCallback);

            // Blah blah ...

            // Bind some utility classes

            Bind<ISeoUtils>().To<SeoUtils>().InSingletonScope();

            Bind<IFtpUtils>().To<FtpUtils>().InSingletonScope();

            // Bind the configuration classes to the appropriate XML configuration files for this module

            Bind<IAppConfig>().To<AppConfig>().InSingletonScope().WithConstructorArgument("filename", _appConfigFile);

            Bind<ISiteConfig>().To<SiteConfig>().InSingletonScope().WithConstructorArgument("filename", _siteConfigFile);

        }

    }

}

Now it is clear from this example that we have some objects are are request scope and some application scope. We normally use only application scope for both on our live server, as presently when I tried to use request scope it totally hangs up on me in the caching code, getting stuck in a lock() which would indicate we have a deadlock somehow. It gets stuck on this lock in Resolve() in context.cs:

        public object Resolve()

        {

            lock (Binding)

            {

Which indicates a deadlock. 

You mentioned above we have to use the same bindings. Does that mean we cannot mix scopes? Some items we want to be singleton scope, some request scope and some application scope (actually we really only need application scope but we coded up request scope to see if it behaves any differently, and it does). Some of the items which are request scope will reference the Couchbase factory class, which is application scope. If it then has references to request scope, perhaps that is the root cause of that issue? It won't be a cause of the scope being wrong when all application scope, but maybe that will shed some light on this.

And that they are kept in the cache until the scope object is GC’d. In a high load environment is might be necessary to actively tell Ninject to remove objects cached by some scope. E.g. when using request scope we actively tell Ninject to remove the objects in this request after the request is completed. Otherwise you can get out of memory exceptions because the GC loop takes too long. Also be aware that your scope implementation must be thread safe. Ninject will access it concurrently from all threads that you use to resolve objects.

Right, we replicated the same feature in the Dispose() method for our HttpApplication class, to make sure they get cleaned up immediately when an application is disposed, not when it is GC'ed. 

        /// <summary>

        /// Called when the application instance is being disposed of

        /// </summary>

        public override void Dispose()

        {

            // Make sure the Couchbase clients are disposed of by clearing the scope of this application instance

            // with Ninject

            try {

                var cache = Globals.Kernel.Components.Get<ICache>();

                cache.Clear(this);

            } catch {

                // Do nothing here. It is possible during app pool shutdown that the above may well fail ...

            }

            // Call dispose on the base class

            base.Dispose();

        }

3.       The object must be resolved from the same kernel instance. Depending on your implementation you are either using one shared kernel or one for each app pool instance. In the second case the resolved objects are per app pool instance and you cannot expect to reuse them between app pools

We have a single kernel instance used for the entire application stack, created inside Application_Start(). That function only gets called once for the life of the application for that web site, when the first application instance fires up. We don't create a kernel for every application instance.

We do have multiple applications all running in the same application pool, but as I understand it apps in the same application pool are totally isolated from each other and don't share anything, global variables or application instances?

 Probably you should add code to show how you implemented your scope. I think there is a high change that the problem is in there.

 

Btw, caching is done in the cache: https://github.com/ninject/ninject/blob/master/src/Ninject/Activation/Caching/Cache.cs

 

Remo

 

From: nin...@googlegroups.com [mailto:ninject@googlegroups.com] On Behalf Of Kendall Bennett
Sent: Montag, 15. Juli 2013 23:47
To: nin...@googlegroups.com
Subject: [ninject] Ninject failures under high load with ASP.NET MVC?

 

Hi Guys,

We have been battling some really odd bugs in our system when under high load for some time, but never could track it down. Then recently we made some changes that caused Ninject to be used more heavily and earlier in the request pipeline on our web site, and immediately started to see some odd crashes, which when we examined the issue looked clearly like threading issues. Two threads were executing on the same Ninject resolved instance, which should never happen. We searched high and low, but never could find any issues in our code, and it finally led me to start to suspect Ninject. No matter what we do, we cannot reproduce the issue in a debug environment, and even on our staging server we had to heavily load the machine using Telerik Test Studio before we finally managed to have it fail in the same way that our live server does. We have been able to work around the crash issues in the code by putting threading locks in place, but it is only a stop gap measure, not a proper solution. And on our live server the issue only appears to happen once every couple of hours, or when the application pool is recycled. It seems to happen a lot more regularly when the load on the server is going up or down, indicating that new scope objects are getting created or destroyed. 

We also updated Ninject from the 2.2 release we were using, to the latest stable release in NuGet (3.0.2 or something). It made no difference.

Near as I can figure out, when under load, Ninject sometimes serves up the WRONG instance to us, like it has scoped it wrong or something. We do not currently use request scope, but rather use application scope, which is a custom scope we made ourselves to bind the Ninject instances to instances of HttpApplication under ASP.NET. Since ASP.NET ensures that only one request at a time can be handled by a single application instance, it means we can avoid having to deal with threading issues within our code, and code it liked it is single threaded, as each application instance should end up getting their own set of objects, scoped to that application instance. More importantly, the instances remain across requests so we can cache things internally as necessary between requests for better performance. In our code we put some checks in to see if the application instance stored in the object Ninject returns to us is DIFFERENT to the current application instance, and then we log it. This happens every couple of hours, at the same time that we were normally seeing the crashes. So clearly Ninject at some point is giving us objects for the WRONG application instance, and it only happens under load. Which would indicate there is probably some kind of threading issue within the Ninject kernel (since it is a singleton instance used throughout the entire app). 

Has anyone noticed anything similar before? I spoke briefly with the folks at Mindscape about the issues, as we have been using their new RayGun.IO project to capture the runtime errors and log information about the issues, and they mentioned they gave up using Ninject as they had seen some similar issues in the past and never worked it out. This is what he said:

"I can't help explicitly with your issue here but I will say that we've used Ninject several times and have run into the problem that it never disposes things properly -- even when using the InstancePerRequest model (or something like that -- it was a long time ago). What would happen is that you'dexpect it to therefore give you clean items per request but in fact it wouldn't -- it would re-use them globally. This caused us a lot of issues until we figured out that it wasn't doing what you'd think.

It's a random comment but perhaps that's what is happening for you also? It's the sort of issue that only appeared for us once under load as well - on a dev machine or similar low load machine it was difficult to create weirdness."

Not sure if that was related to this or not, but now I am 99.9% sure something is wrong with Ninject. I plan to try using a different IoC container and see if it makes any difference, but I would really like to know if anyone else has seen anything similar to this, and if you have any suggestions on where I can look in the code to see where it might be going wrong?

Unfortunately because we cannot reproduce this problem in a debug environment, and only in a non-debug environment under load, it is really difficult to actually setup up a debug version of Ninject and try to follow the code through and figure out where it might be going wrong. But if someone can point out where in the code it does the scoping stuff, I could follow it through and see if I can spot anything.

BTW, the machine is running Windows Server 2008 R2, 64-bit, ASP.NET 4.5, .NET 4.5, ASP.NET MVC 4.

- Kendall

-- 
You received this message because you are subscribed to the Google Groups "ninject" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninject+u...@googlegroups.com.
To post to this group, send email to nin...@googlegroups.com.
Visit this group at http://groups.google.com/group/ninject.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to a topic in the Google Groups "ninject" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ninject/SEVZ8SUnAOo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ninject+u...@googlegroups.com.

To post to this group, send email to nin...@googlegroups.com.
Visit this group at http://groups.google.com/group/ninject.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Regards,

Kendall Bennett, CEO 
http://www.AMainHobbies.com
1-800-705-2215 (Toll-Free)
1-530-894-0797 (Local)

Facebook | Twitter | Youtube

[Sean Chambers]

I'm not sure I have a lot to contribute but perhaps I can throw some relevant questions in that may help. 

Out of curiosity, is there a major reason why you have to manage your own scoping? I've found that ninject 3 and up is pretty rock solid as far as using the scope management that's baked in, but perhaps you need that level of wiring for your application. 

When you mentioned that using request scope on your live server completely looks everything up, We use a mix of request/application scope in all our production apps and have never experienced anything like this. 

As far as iis config, you mentioned you have multiple applications use the same app pool. I would star by separating these out because it proves for an easy way to determine in exactly which app the problem resides when you can easily pinpoint production problems. 

I'm curious as to what the problem is here as I haven't experienced behavior out of ninject like this before. If you find the problem can you update the thread to help my curiosity?

Hopefully that helps

Sean

[Remo Gloor]

1.

By the same binding I mean that if there are two conditional bindings they will not return the same instance e.g.

 

Bind<IFoo>().To<Foo>().When(c => DateTime.Now.Hour % 2 == 0).InSingletonScope()

Bind<IFoo>().To<Foo>().When(c => DateTime.Now.Hour % 2 == 1).InSingletonScope()

 

Will create two instances of Foo.

 

2.

I don’t’ know when IIS creates a new HttpContext.Current.ApplicationInstance Instance and when it reuses an existing one. Probably you should verify that it has the lifetime that you expect. E.g. is it the same instance after an App Pool recycling or does it just contain the same data but it is a different instance?

 

You can mix scope. But you have to make sure that no object uses another with shorter lifetime. E.g. it is a bad idea to inject a request scoped object instance into a singleton/application scoped object instance.

 

BTW request Scope still exists it just has been moved into the Web extensions so that the core doesn’t depend on System.Web anymore.

[Kendall Bennett]

Hi Sean,

I'm not sure I have a lot to contribute but perhaps I can throw some relevant questions in that may help. 

Out of curiosity, is there a major reason why you have to manage your own scoping? I've found that ninject 3 and up is pretty rock solid as far as using the scope management that's baked in, but perhaps you need that level of wiring for your application. 

As far as I am aware there is no support for application instance scoping in Ninject. Also since the changed to Ninject scoping in 3.0 over 2.2, the callbacks no longer appear to be included anywhere, and I need scope callbacks so I can pass in different scoping to my modules. The reason we need this is because we use the same module init code to wire stuff up for the web site, unit tests and back end services. The web site runs application scope, the unit tests run singleton scope (since they are single threaded) and the back end services need thread scope since they are multi-threaded but do not have any web context. We just need to make sure every background thread we start up to process background services gets it's own instances so it is all thread safe.

If there were callbacks for application and request scoping in Ninject proper, I would just use that. I also had to wire up my own disposal code to clean out the cache when application instances are disposed of, since that I know is not in Ninject.

When you mentioned that using request scope on your live server completely looks everything up, We use a mix of request/application scope in all our production apps and have never experienced anything like this. 

I fixed this easily enough. Turns out when I had hacked in support for request scope we wanted to keep the couchbase factory in application scope, I had references to a request scoped object. It never really needed the request scoped object so with a bit of refactoring I was able to move it out and clean up the scope. Now request scope works fine and we are load testing that on a test system to see if we can replicate the same issues, hopefully quicker and maybe in a way that will shed light on the issue more quickly.

If you are wondering why the couchbase factory needs to be application scoped, it is long story. But in short we needed to make sure only one connection is created and re-used per application instance for performance; the built in HTTP cache does not work properly under heavy load and we are still looking into that one :).

BTW, you mentioned you are also using application scope. How did you wire up application scope in your web apps?

As far as iis config, you mentioned you have multiple applications use the same app pool. I would star by separating these out because it proves for an easy way to determine in exactly which app the problem resides when you can easily pinpoint production problems. 

I can certainly move it to it's own app pool, but from my experience there is no cross contamination between the app pools with apps running within them, so I doubt it would make any difference.

I'm curious as to what the problem is here as I haven't experienced behavior out of ninject like this before. If you find the problem can you update the thread to help my curiosity?

Sure, I hope I figure it out!

- Kendall

[Sean Chambers]

I misspoke about the application scope. I meant singleton scope. 

Sean

[Kendall Bennett]

Hi Remo,

1.

By the same binding I mean that if there are two conditional bindings they will not return the same instance e.g.

 

Bind<IFoo>().To<Foo>().When(c => DateTime.Now.Hour % 2 == 0).InSingletonScope()

Bind<IFoo>().To<Foo>().When(c => DateTime.Now.Hour % 2 == 1).InSingletonScope()

 

Will create two instances of Foo.

Ahh, OK. We don't use any conditional bindings like that. The most complex we have is constructor injection :)

2.

I don’t’ know when IIS creates a new HttpContext.Current.ApplicationInstance Instance and when it reuses an existing one. Probably you should verify that it has the lifetime that you expect. E.g. is it the same instance after an App Pool recycling or does it just contain the same data but it is a different instance?

I have debugged that extensively so I know how it works. Basically IIS creates new application instances whenever it needs to handle more load. The more load it needs to handle, the more instances it fires up. It is the way it handles parallelism in ASP.NET applications without the application code needing to worry about threading, because only ONE request will ever be serviced at a time on each application instance in parallel. When IIS decides the load has gone down and it does not need any instances, it disposes of them. I have debugged that and watched instances come up and then later after a couple of minutes of no activity start to get killed off. And we clean up our scope when the instance gets disposed of so I know our scope is being cleaned up.

One thing really confusing about the life cycle of applications in IIS is the Application_Start() method. It is only called ONE time when the application pool is fired up, even though it is an instance function. It gets called on the first instance of HttpApplication that is created, not any other one. So the mistake it so think this happens for every application instance and to store stuff in class variables in the HttpApplication class, as that will never work. New instances will not have those variables be set. You need to do that in the constructor if you want class variables in application scope.

Likewise Application_End() is called when it kills off the last application instance when it is shutting down the application pool, or if the pool is recycling. It does not re-use application instances across recycles. Our applications only recycle twice a day now, but the errors we see can happen once every couple of hours, long before the app pool recycles. 

BTW, I did quickly discover some time ago that using thread scope on a web server is a really bad idea. I never did track it down, but IIS seems to randomly pickup threads from the thread pool and I *think* if a process blocks on an IO completion operation, it may well end up on a *different* thread when it comes back to life later on. Which clearly can never work with an IoC container where your objects are scoped to threads. Just an FYI for anyone reading this :)

 You can mix scope. But you have to make sure that no object uses another with shorter lifetime. E.g. it is a bad idea to inject a request scoped object instance into a singleton/application scoped object instance.

Yep. Figured that one out. Referencing something in request scope from an application scoped item is a bad idea :) Reference singleton scope from any other scope is fine.

BTW request Scope still exists it just has been moved into the Web extensions so that the core doesn’t depend on System.Web anymore.

Yes, but the callbacks don't appear to be there anymore?

- Kendall

[Kendall Bennett]

Hi Remo,

We ran into another nasty scoping bug that took a while to figure out. I started using ninject to inject interfaces we need into MVC Filter attributes, which works fine but I have now discovered that MVC creates the filter attributes ONCE and reuses the instances over and over again for new requests. I assume they are application instance scoped, and not request scoped like controller instances are. I had assumed they were request scoped and the instances I am injecting are indeed request scoped. Which of course led me to an almost identical lockup to the one from last time.

I have worked around it for the moment by calling Kernel.Get() to get my instances when I need them which works, but is not clean and certainly slower. So my question is, is there any way to control the life cycle of MVC filter attributes so it will toss them away at the end of a request and create new ones for each request, so I can inject request scoped interfaces into it? Otherwise injecting filter attributes is pointless unless we only inject singleton instance classes (which mine are not and it would not be easy to make them singleton).

Regards,

Kendall Bennett, CEO 
http://www.AMainHobbies.com
1-800-705-2215 (Toll-Free)
1-530-894-0797 (Local)

Facebook | Twitter | Youtube

[Kendall Bennett]

Also come to think of it this keeps happening, where we have scoping issues and it causes a deadlock in the Ninject activation code under high load. It is not clear what causes the deadlock but it occurred to me that perhaps we can add a feature in debug mode (activated in Web.config or something) that would have Ninject check the scope of objets when it is creating new ones, to make sure their lifetime is <= the lifetime of the new object being created? Do you think that is possible?

Regards,

Kendall Bennett, CEO
http://www.AMainHobbies.com
1-800-705-2215 (Toll-Free)
1-530-894-0797 (Local)

Facebook | Twitter | Youtube

On Oct 31, 2013, at 12:52 AM, Kendall Bennett <kend...@amainhobbies.com> wrote:

> Hi Remo,
>
> We ran into another nasty scoping bug that took a while to figure out. I started using ninject to inject interfaces we need into MVC Filter attributes, which works fine but I have now discovered that MVC creates the filter attributes ONCE and reuses the instances over and over again for new requests. I assume they are application instance scoped, and not request scoped like controller instances are. I had assumed they were request scoped and the instances I am injecting are indeed request scoped. Which of course led me to an almost identical lockup to the one from last time.
>
> I have worked around it for the moment by calling Kernel.Get() to get my instances when I need them which works, but is not clean and certainly slower. So my question is, is there any way to control the life cycle of MVC filter attributes so it will toss them away at the end of a request and create new ones for each request, so I can inject request scoped interfaces into it? Otherwise injecting filter attributes is pointless unless we only inject singleton instance classes (which mine are not and it would not be easy to make them singleton).

> ________________________________________
> From: nin...@googlegroups.com [nin...@googlegroups.com] on behalf of Remo Gloor [remo....@bbv.ch]
> Sent: Wednesday, July 17, 2013 1:07 AM
> To: nin...@googlegroups.com
> Subject: RE: [ninject] Ninject failures under high load with ASP.NET MVC?

>
> 1.
> By the same binding I mean that if there are two conditional bindings they will not return the same instance e.g.
>
> Bind<IFoo>().To<Foo>().When(c => DateTime.Now.Hour % 2 == 0).InSingletonScope()
> Bind<IFoo>().To<Foo>().When(c => DateTime.Now.Hour % 2 == 1).InSingletonScope()
>
> Will create two instances of Foo.
>
> 2.
> I don’t’ know when IIS creates a new HttpContext.Current.ApplicationInstance Instance and when it reuses an existing one. Probably you should verify that it has the lifetime that you expect. E.g. is it the same instance after an App Pool recycling or does it just contain the same data but it is a different instance?
>
> You can mix scope. But you have to make sure that no object uses another with shorter lifetime. E.g. it is a bad idea to inject a request scoped object instance into a singleton/application scoped object instance.
>
> BTW request Scope still exists it just has been moved into the Web extensions so that the core doesn’t depend on System.Web anymore.
>
>
>
>
>
> From: nin...@googlegroups.com [mailto:nin...@googlegroups.com] On Behalf Of Kendall Bennett
> Sent: Mittwoch, 17. Juli 2013 01:09
> To: nin...@googlegroups.com
> Subject: Re: [ninject] Ninject failures under high load with ASP.NET MVC?
>
> Hi Remo,
>
>
> There are three things that have an influence on the instance returned to you:
> 1. The same binding must be used to resolve an instance. If you have conditional bindings for a type they will return different instances even when having the same scope.
>

> When you say the same binding must be used to resolve an instance, what exactly do you mean? We have a single, static kernel instance that is used throughout the code to resolve bindings, and it is used by the ASP.NET<http://ASP.NET> MVC code and also by our own code. All the bindings are set up in one place, once for the single Ninject kernel instance. We use Kernel.Get<IBlah>() to get the instances we need manually, and from what I can work out the same essentially happens inside the ASP.NET<http://ASP.NET> MVC NinjectDependencyResolver (although it goes through the Bootstrapper class, but all it does when you create it is return the same single static instance; not any different to a static global variable?). So I think everything goes through Kernel.Get<>() for all the instances we need, and they should all be bound in only one place. I have posted our binding code below.

> From: nin...@googlegroups.com<mailto:nin...@googlegroups.com> [mailto:nin...@googlegroups.com<http://googlegroups.com>] On Behalf Of Kendall Bennett
> Sent: Montag, 15. Juli 2013 23:47
> To: nin...@googlegroups.com<mailto:nin...@googlegroups.com>
> Subject: [ninject] Ninject failures under high load with ASP.NET<http://ASP.NET> MVC?
>
> Hi Guys,
>
> We have been battling some really odd bugs in our system when under high load for some time, but never could track it down. Then recently we made some changes that caused Ninject to be used more heavily and earlier in the request pipeline on our web site, and immediately started to see some odd crashes, which when we examined the issue looked clearly like threading issues. Two threads were executing on the same Ninject resolved instance, which should never happen. We searched high and low, but never could find any issues in our code, and it finally led me to start to suspect Ninject. No matter what we do, we cannot reproduce the issue in a debug environment, and even on our staging server we had to heavily load the machine using Telerik Test Studio before we finally managed to have it fail in the same way that our live server does. We have been able to work around the crash issues in the code by putting threading locks in place, but it is only a stop gap measure, not a proper solution. And on our live server the issue only appears to happen once every couple of hours, or when the application pool is recycled. It seems to happen a lot more regularly when the load on the server is going up or down, indicating that new scope objects are getting created or destroyed.
>
> We also updated Ninject from the 2.2 release we were using, to the latest stable release in NuGet (3.0.2 or something). It made no difference.
>

> Near as I can figure out, when under load, Ninject sometimes serves up the WRONG instance to us, like it has scoped it wrong or something. We do not currently use request scope, but rather use application scope, which is a custom scope we made ourselves to bind the Ninject instances to instances of HttpApplication under ASP.NET<http://ASP.NET>. Since ASP.NET<http://ASP.NET> ensures that only one request at a time can be handled by a single application instance, it means we can avoid having to deal with threading issues within our code, and code it liked it is single threaded, as each application instance should end up getting their own set of objects, scoped to that application instance. More importantly, the instances remain across requests so we can cache things internally as necessary between requests for better performance. In our code we put some checks in to see if the application instance stored in the object Ninject returns to us is DIFFERENT to the current application instance, and then we log it. This happens every couple of hours, at the same time that we were normally seeing the crashes. So clearly Ninject at some point is giving us objects for the WRONG application instance, and it only happens under load. Which would indicate there is probably some kind of threading issue within the Ninject kernel (since it is a singleton instance used throughout the entire app).

>
> Has anyone noticed anything similar before? I spoke briefly with the folks at Mindscape about the issues, as we have been using their new RayGun.IO project to capture the runtime errors and log information about the issues, and they mentioned they gave up using Ninject as they had seen some similar issues in the past and never worked it out. This is what he said:
>
> "I can't help explicitly with your issue here but I will say that we've used Ninject several times and have run into the problem that it never disposes things properly -- even when using the InstancePerRequest model (or something like that -- it was a long time ago). What would happen is that you'dexpect it to therefore give you clean items per request but in fact it wouldn't -- it would re-use them globally. This caused us a lot of issues until we figured out that it wasn't doing what you'd think.
>
> It's a random comment but perhaps that's what is happening for you also? It's the sort of issue that only appeared for us once under load as well - on a dev machine or similar low load machine it was difficult to create weirdness."
>
> Not sure if that was related to this or not, but now I am 99.9% sure something is wrong with Ninject. I plan to try using a different IoC container and see if it makes any difference, but I would really like to know if anyone else has seen anything similar to this, and if you have any suggestions on where I can look in the code to see where it might be going wrong?
>
> Unfortunately because we cannot reproduce this problem in a debug environment, and only in a non-debug environment under load, it is really difficult to actually setup up a debug version of Ninject and try to follow the code through and figure out where it might be going wrong. But if someone can point out where in the code it does the scoping stuff, I could follow it through and see if I can spot anything.
>

> BTW, the machine is running Windows Server 2008 R2, 64-bit, ASP.NET<http://ASP.NET> 4.5, .NET 4.5, ASP.NET<http://ASP.NET> MVC 4.

>
> - Kendall
> --
> You received this message because you are subscribed to the Google Groups "ninject" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to ninject+u...@googlegroups.com<mailto:ninject+u...@googlegroups.com>.
> To post to this group, send email to nin...@googlegroups.com<mailto:nin...@googlegroups.com>.

> Visit this group at http://groups.google.com/group/ninject.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "ninject" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/ninject/SEVZ8SUnAOo/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to ninject+u...@googlegroups.com<mailto:ninject+u...@googlegroups.com>.
> To post to this group, send email to nin...@googlegroups.com<mailto:nin...@googlegroups.com>.

> Visit this group at http://groups.google.com/group/ninject.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
> Regards,
>
> Kendall Bennett, CEO

> http://www.AMainHobbies.com<http://www.amainhobbies.com/>
> 1-800-705-2215 (Toll-Free)
> 1-530-894-0797 (Local)
>
> Facebook<http://www.facebook.com/amainhobbies> | Twitter<http://www.twitter.com/AMainHobbies> | Youtube<http://www.youtube.com/AMainHobbiesVideos>

>
> --
> You received this message because you are subscribed to the Google Groups "ninject" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to ninject+u...@googlegroups.com<mailto:ninject+u...@googlegroups.com>.
> To post to this group, send email to nin...@googlegroups.com<mailto:nin...@googlegroups.com>.