simultaneous calls throws "Collection was modified after the enumerator was instantiated"

[Choco Smith]

Hi

Running Ninject (3.0.1.10), ninject mvc3(3.0.0.6), and some other ninject extensions (logging and conventions)

Our project works fine but after a couple of hours on a heavily loaded server we start getting an issue similar to https://groups.google.com/forum/#!topic/ninject/f2HgHwTRFwY where the application throws an exception with "Collection was modified after ... " and then every call afterwards results in a "A cyclical dependency was detected..."  until the application pool is restarted.

The particular call that fails usually takes about 15seconds to complete.

After a few days of digging I was able to reproduce and found a solution by simply changing the constructor argument from an IEnumerable<ITypeAdder> to List<ITypeAdder> and the problem goes away (nonworking class below) 

 public class CommandParameterAdder : ICommandParameterAdder

    {

        private readonly IEnumerable<ITypeAdder> _typeAdders;

       //change this to list and it works

        public CommandParameterAdder(IEnumerable<ITypeAdder> typeAdders)

        {

            _typeAdders = typeAdders;

        }

  public void AddParameter(object value, PropertyInfo propInfo, iDB2Command command, IParameterDefinition parameterDefinition)

        {

           //throws exception after a while

            var typeAdder = _typeAdders.FirstOrDefault(a => a.Handles(propInfo.PropertyType));

        }

}

There are roughly 15 concrete class for ITypeAdder 

  this.Bind(config => config

                                    .FromThisAssembly()

                                    .SelectAllClasses()

                                    .InheritedFrom<ITypeAdder>()

                                    .BindSingleInterface());

The exception stack traces:

2014-02-01 13:15:55.3597 | Error | An error occurred. Url: 'http://AProject:91/api/agreements/SOLID/910', Method: POST  | (System.Web.Http.Tracing.ITraceWriterExtensions.TraceBeginEnd) 

| (TaskHelpersExtensions.CatchImpl => <>c__DisplayClass4`1.<Catch>b__3 => <>c__DisplayClass8.<InvokeActionWithExceptionFilters>b__5 => TaskHelpers.Iterate 

=> TaskHelpers.IterateImpl => WhereSelectEnumerableIterator`2.MoveNext => <>c__DisplayClassa.<InvokeActionWithExceptionFilters>b__6 

=> ExceptionFilterAttribute.System.Web.Http.Filters.IExceptionFilter.ExecuteExceptionFilterAsync 

=> ExceptionFilterAttributeTracer.OnException => ITraceWriterExtensions.TraceBeginEnd) System.InvalidOperationException: 

Collection was modified after the enumerator was instantiated.

   at System.Collections.Generic.Stack`1.Enumerator.MoveNext()

   at System.Collections.Generic.Stack`1..ctor(IEnumerable`1 collection)

   at Ninject.Activation.Request..ctor(IContext parentContext, Type service, ITarget target, Func`1 scopeCallback)

   at Ninject.Activation.Request.CreateChild(Type service, IContext parentContext, ITarget target)

   at Ninject.Planning.Targets.Target`1.GetValue(Type service, IContext parent)

   at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()

   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)

   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)

   at Ninject.Activation.Providers.StandardProvider.Create(IContext context)

   at Ninject.Activation.Context.Resolve()

   at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()

   at System.Linq.Enumerable.<CastIterator>d__b1`1.MoveNext()

   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)

   at AProject.Data.DB2.CommandParameterAdder.AddParameter(Object value, PropertyInfo propInfo, iDB2Command command, IParameterDefinition parameterDefinition) in c:\Builds\3cd0427f6943da7f\AProject.Data\DB2\CommandParameterAdder.cs:line 22

   at AProject.Data.DB2.StoredProcedureCaller.AddParameters(IStoredProcedureDefinition storedProcedureDefinition, Object obj, Type type, iDB2Command command, String executingUserName, Int32 inOutStatusValue) in c:\Builds\3cd0427f6943da7f\AProject.Data\DB2\StoredProcedureCaller.cs:line 116

   at AProject.Data.DB2.StoredProcedureCaller.CallProcedureInternal(String commandExecutingUser, Object obj, Int32 inOutStatusValue, Type type) in c:\Builds\3cd0427f6943da7f\AProject.Data\DB2\StoredProcedureCaller.cs:line 70

   at AProject.Data.DB2.StoredProcedureCaller.CallProcedure(String commandExecutingUser, Object obj, Int32 inOutStatusValue) in c:\Builds\3cd0427f6943da7f\AProject.Data\DB2\StoredProcedureCaller.cs:line 48

   at AProject.Data.DB2.DataAccessService.ExecuteInternal(Object obj, String executingUserName, DataAccessMode statusValue) in c:\Builds\3cd0427f6943da7f\AProject.Data\DB2\DataAccessService.cs:line 83

   at AProject.Data.DB2.DataAccessService.Execute[T](T obj, String executingUserName, DataAccessMode statusValue) in c:\Builds\3cd0427f6943da7f\AProject.Data\DB2\DataAccessService.cs:line 35

   at AProject.Agreement.Data.AgreementWriteDataAccess.CalculatePremiumForProducts(Agreement agreement, String executingUser) in c:\Builds\3cd0427f6943da7f\AProject.Agreement\Data\AgreementWriteDataAccess.cs:line 194

   at AProject.Agreement.Data.AgreementWriteDataAccess.Access(Agreement agreement, String executingUser) in c:\Builds\3cd0427f6943da7f\AProject.Agreement\Data\AgreementWriteDataAccess.cs:line 71

   at AProject.Agreement.Handlers.CreateAgreementCommandHandler.Handle(CreateAgreementCommand command) in c:\Builds\3cd0427f6943da7f\AProject.Agreement\Handlers\CreateAgreementCommandHandler.cs:line 52

   at AProject.Api.Agreement.Controllers.AgreementController.Create(String companyId, Int64 agentId, CreateAgreementCommand command) in c:\Builds\3cd0427f6943da7f\AProject.Api.Agreement\Controllers\AgreementController.cs:line 57

   at lambda_method(Closure , Object , Object[] )

   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass13.<GetExecutor>b__c(Object instance, Object[] methodParameters)

   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)

   at System.Threading.Tasks.TaskHelpers.RunSynchronously[TResult](Func`1 func, CancellationToken cancellationToken)

   

 

From now on Every call to this api will fails with the message:

2014-02-01 13:15:55.3597 | Error | An error occurred. Url: 'http://AProject:91/api/agreements/SOLID/910', Method: POST  | (System.Web.Http.Tracing.ITraceWriterExtensions.TraceBeginEnd) | (TaskHelpersExtensions.CatchImpl => <>c__DisplayClass4`1.<Catch>b__3 => <>c__DisplayClass8.<InvokeActionWithExceptionFilters>b__5 => TaskHelpers.Iterate => TaskHelpers.IterateImpl => WhereSelectEnumerableIterator`2.MoveNext => <>c__DisplayClassa.<InvokeActionWithExceptionFilters>b__6 => ExceptionFilterAttribute.System.Web.Http.Filters.IExceptionFilter.ExecuteExceptionFilterAsync => ExceptionFilterAttributeTracer.OnException => ITraceWriterExtensions.TraceBeginEnd) Ninject.ActivationException: Error activating ITypeAdder using binding from ITypeAdder to Int32TypeAdder

A cyclical dependency was detected between the constructors of two services.

Activation path:

  5) Injection of dependency ITypeAdder into parameter typeAdders of constructor of type CommandParameterAdder

  4) Injection of dependency ICommandParameterAdder into parameter commandParameterAdder of constructor of type StoredProcedureCaller

  3) Injection of dependency IStoredProcedureCaller into parameter storedProcedureCaller of constructor of type DataAccessService

  2) Injection of dependency IDataAccessService into parameter dataAccessService of constructor of type AS400PremiumDiscountCalculator

  1) Request for IPremiumDiscountCalculator

Suggestions:

  1) Ensure that you have not declared a dependency for ITypeAdder on any implementations of the service.

  2) Consider combining the services into a single one to remove the cycle.

  3) Use property injection instead of constructor injection, and implement IInitializable

     if you need initialization logic to be run after property values have been injected.

   at Ninject.Activation.Context.Resolve()

   at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()

   at System.Linq.Enumerable.<CastIterator>d__b1`1.MoveNext()

   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)

   at AProject.Data.DB2.CommandParameterAdder.AddParameter(Object value, PropertyInfo propInfo, iDB2Command command, IParameterDefinition parameterDefinition) in c:\Builds\3cd0427f6943da7f\AProject.Data\DB2\CommandParameterAdder.cs:line 22

 

If you look at the time stamp you can see the cause seems to be two request at the same time (2014-02-01 13:15:55.3597), after that it doesn't matter each call will fail.

Unfortunately, while I can easily reproduce the error in our business project by simply sending in simultaneous calls, I have not been able to create a minimal solution and wcaf to reproduce the exception.

So I am not sure its a Ninject bug or a we're using it wrong bug. 

Is there anything I can run locally to help/trace the issue?

[Choco Smith]

Update:

While stress testing the above we had the same failure but this time slightly higher in the stack. The interesting part is that it failed at the next constructor with an IEnumerable<T> (different T to before). 

The solution we resorted to was too eagerly evaluated all IEnumerable<T> in the constructor call, this seems to have resolved the problem and passed the weekends stress test.

Cheers

Choco

[Choco Smith]

update prod release:

ahh signs of a mad man, self dialog... anyway.

We released to prod this hotfix, prod is stressed 20 hours of the day so its a good test environment.

Since release we have had only 2 of the cyclic errors return (almost same timestamp). BUT!!! this time they ddi not start with "Collection was modified after the enumerator was instantiated" and the next call into the method worked. 

As a recap from the above a call into a method would work fine for a few days and then it would fail with collection modified error. Then EVERY call after the method failed with cyclic error. Solution so far has been to eagerly evaluate IEnumerable<T> (we used an extension method to make it clear when we call ToList on everything.

[Remo Gloor]

You have to materialize IEnumerables in the constructor. Otherwise it will be done outside of the Ninject internal locks which will lead to unpredictable behaviors. So using ToList, injecting a IList instead of an IEnumerable or loop over the enumerable only in the constructor is exactly what you need to do.

 

When you still have problems after that then there might be an unknown problem within Ninject. Please give more information in that case.

--
You received this message because you are subscribed to the Google Groups "ninject" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninject+u...@googlegroups.com.
To post to this group, send email to nin...@googlegroups.com.
Visit this group at http://groups.google.com/group/ninject.
For more options, visit https://groups.google.com/groups/opt_out.

[Choco Smith]

Hi Remo, 

Thanks for the response.

I'd thought I'd post an update as a feed back to anyone in the future (including myself :D) 

We have now been in prod for 3 days and no failures (thank god my service contract was about to be breached).

Turns out we had two issues.

The first was as stated above, we were using IEnumberable<T> in our constructors and under enough load these would break. What is written above by Remo makes sense and explains it well.

The second was due to a hack by a previous programmer, turns out that the this guy had had issues with "A cyclical dependency was detected between the constructors of two services." and "solved" it by adding a Func<IEnumberable<MyClass> into the constructor. 

This bypass worked but only until the load got heavy.

Again many thanks for the help above. 

Cheers,

Choco