Performance testing Ninject

[simone.c...@gmail.com]

Hi all,
I'm doing a little bit of a performance testing with various IoC
framework, and from a quick test I found out that Ninject is 4 time
slower than StructureMap.
So maybe I'm doing something wrong that is hitting the performances:

Here is the objects of my "model"

public class Repository : IRepository
{
private IDataAccess _dal;

public IDataAccess Dal
{
get { return _dal; }
set { _dal = value; }
}

public Repository(IDataAccess dal)
{
_dal = dal;
}

public Repository()
{
_dal = new DataAccess();
}
}

public interface IRepository
{
IDataAccess Dal { get; set; }
}

public class DataAccess: IDataAccess
{
}

public interface IDataAccess
{
}


Here the Loader Module

public class SingleDepModule : StandardModule
{
public override void Load()
{
Bind<IRepository>().To<Repository>();
Bind<IDataAccess>().To<DataAccess>();
}
}

And this the way I load it:

IKernel c = new StandardKernel(new
SingleDependency.SingleDepModule());
for (ulong i2 = 0; i2 < loopsize; i2++)
{
SingleDependency.IRepository classe =
c.Get<SingleDependency.IRepository>();
}

It seems to me everything is correct... but it's the first time I'm
using Ninject, so I might be totally wrong.

Simo

[Nate Kohari]

Simo:

I'm not sure if it makes a big difference in your benchmark, but you should use the [Inject] attribute on one of your constructors of your Repository class. If you leave the [Inject] attribute off, Ninject will call the parameterless one (or, if there's only a single constructor defined, it will call that one). Since you have two constructors, Ninject won't know which to select, and will call the incorrect one in this case.

Torkel Ödegaard has already done some other work done on benchmarking the different dependency injection frameworks, which you might find interesting:

http://www.codinginstinct.com/2008/05/ioc-container-benchmark-rerevisted.html

Essentially, the conclusion is while any dependency injection framework adds overhead to your application, it isn't significant enough to matter. Bear in mind that you will never be creating millions of objects with the DI framework, and if you are, you might want to rethink your design!

That being said, feel free to run your benchmark, and post the results. I'm always interested in improving the framework.


Thanks,
Nate

[David]

Those results dont match my findings. I have found Unity 1.1 to be
4-6x faster than Ninject for resolution, so far. I am working to clean
up my test, and try with other IOC's.

David

On Jul 3, 1:46 pm, "Nate Kohari" <nkoh...@gmail.com> wrote:
> Simo:
>
> I'm not sure if it makes a big difference in your benchmark, but you should
> use the [Inject] attribute on one of your constructors of your Repository
> class. If you leave the [Inject] attribute off, Ninject will call the
> parameterless one (or, if there's only a single constructor defined, it will
> call that one). Since you have two constructors, Ninject won't know which to
> select, and will call the incorrect one in this case.
>
> Torkel Ödegaard has already done some other work done on benchmarking the
> different dependency injection frameworks, which you might find interesting:
>

> http://www.codinginstinct.com/2008/05/ioc-container-benchmark-rerevis...

>
> Essentially, the conclusion is while any dependency injection framework adds
> overhead to your application, it isn't significant enough to matter. Bear in
> mind that you will never be creating millions of objects with the DI
> framework, and if you are, you might want to rethink your design!
>
> That being said, feel free to run your benchmark, and post the results. I'm
> always interested in improving the framework.
>
> Thanks,
> Nate
>

> On Thu, Jul 3, 2008 at 1:18 PM, simone.chiare...@gmail.com <

[Simone Chiaretta]

Hi Nate,
thank you for the answer...
I didn't find the link when I searched for other bencharmks...
In my baseline benchmark I noticed that constructon Injection is twice as fast as setter injection, so maybe that's where the problem lies.

The 10M object instances is because just one instance will be to fast to get an accurate timing.

So far I've implemented benchmarks for SM and Ninject (plus the baseline ones with poor's man DI).
Tomorrow I'll do Unity, spring and castle.

I'd be happy if I could help in same way to Ninject

Simo

--
Simone Chiaretta
codeclimber.net.nz
Any sufficiently advanced technology is indistinguishable from magic
"Life is short, play hard"

[Simone Chiaretta]

Well... from the graph it is:
Ninject 24 sec
Unity 8 sec
ok, not 4 times, but at least 3 :)
I think the "singleton" test is misleading (and actually I don't understand what singleton means in the context of DI)
Simo

[David]

I downloaded the perf test and will try to recreate.

David

On Jul 3, 5:55 pm, "Simone Chiaretta" <simone.chiare...@gmail.com>
wrote:

[Nate Kohari]

Unity is definitely faster than Ninject, which itself is faster than Windsor and StructureMap. However, bear in mind that the difference is insignificant in practice, and only becomes apparent in contrived scenarios like benchmarks.

Simone actually said it very well -- when activating a small number of instances, the difference is too insignificant to measure. A typical application will have dozens of services, and a very large system might have hundreds, but none will have millions. Further, if you use a database, transmit information across a network, display a user interface, or save information to disk -- which nearly every useful application does -- the cost of using a DI framework will pale in comparison.

I've used benchmark results to discover performance issues before, but otherwise, while it can be an interesting exercise, it's largely irrelevant to actual performance.


-Nate

[Matt Burton]

Right - with a small number of components / short container lifetime,
Ninject is rockin' - beats Windsor and just under Autofac. The
scenario I'm looking at, however, is more of a long-running process
such as a service host environment for WCF services. There the
container is going to live for the lifetime of the application
instance, so a benchmark that measures repeated component resolutions
(say 1,000,000) is more valid. What I've seen thus far, unfortunately,
is that the the performance of Ninject goes downhill under such a
simulated high-load - some numbers from a set of benchmarks I've been
creating for a project at work:

*** 100,000 iterations:

ManualBenchmarkSuite starting... (this is DI by hand using a generic dictionary)
CounterLogger called 100000 times.
Elapsed Time: 332 milliseconds

AutofacBenchmarkSuite starting...
CounterLogger called 100000 times.
Elapsed Time: 613 milliseconds

NinjectBenchmarkSuite starting...
CounterLogger called 100000 times.
Elapsed Time: 1683 milliseconds

WindsorBenchmarkSuite starting...
CounterLogger called 100000 times.
Elapsed Time: 979 milliseconds

*** 1,000,000 iterations:

ManualBenchmarkSuite starting...
CounterLogger called 1000000 times..
Elapsed Time: 4417 milliseconds

AutofacBenchmarkSuite starting...
CounterLogger called 1000000 times.
Elapsed Time: 7434 milliseconds

NinjectBenchmarkSuite starting...
CounterLogger called 1000000 times.
Elapsed Time: 20653 milliseconds

WindsorBenchmarkSuite starting...
CounterLogger called 1000000 times.
Elapsed Time: 10773 milliseconds

You can see here that there is a dramatic degradation in performance
with Ninject under sustained load...this is using a singleton
lifecycle - I would expect under this scenario that the measurements
would be linear as the container should be returning the same instance
repeatedly, right? (and no - I don't have any of the interception
stuff wired in - never could get that working...) I'd be happy to send
you the code I'm using for the benchmarks, but it's pretty much by the
book, straightforward binding calls and then resolves. I also have a
warm up cycle that throws away the first request of the container, so
it's not that I'm including the container startup time in my
measurements...

Any ideas to eke some performance out of Ninject?

Thanks,
Matt

[Heinrich Breedt]

Just wondering if I understand correctly: you are saying that you expect +- 1 million component resolutions every 2 or 3 seconds? And thats why you need to eke performance? Or 1 million per day? Or what?

Heinrich

[Nicholas Blumhardt]

Hi Matt,

That still looks like pretty good performance to me.

I know it is old advice, but, before you tune - measure the actual
app. Within web, networked or database application, the differences
we're seeing here will be thousandths (millionths?) of a percentage
point of actual execution time.

Of course it makes sense to do some initial exploration before you
adopt a framework, but beyond the kind of smoke test you've already
done (in which all of those containers perform fine) I think you're in
for diminishing returns from this kind of profiling.

Cheers

Nick

On Jul 4, 2:53 pm, "Heinrich Breedt" <heinrichbre...@gmail.com> wrote:
> Just wondering if I understand correctly: you are saying that you expect +-
> 1 million component resolutions every 2 or 3 seconds? And thats why you need
> to eke performance? Or 1 million per day? Or what?
>
> Heinrich
>

> > On Thu, Jul 3, 2008 at 4:06 PM, Nate Kohari <nkoh...@gmail.com> wrote:
> > > Unity is definitely faster than Ninject, which itself is faster than
> > Windsor
> > > and StructureMap. However, bear in mind that the difference is
> > insignificant
> > > in practice, and only becomes apparent in contrived scenarios like
> > > benchmarks.
>
> > > Simone actually said it very well -- when activating a small number of
> > > instances, the difference is too insignificant to measure. A typical
> > > application will have dozens of services, and a very large system might
> > have
> > > hundreds, but none will have millions. Further, if you use a database,
> > > transmit information across a network, display a user interface, or save
> > > information to disk -- which nearly every useful application does -- the
> > > cost of using a DI framework will pale in comparison.
>
> > > I've used benchmark results to discover performance issues before, but
> > > otherwise, while it can be an interesting exercise, it's largely
> > irrelevant
> > > to actual performance.
>
> > > -Nate
>

[Simone Chiaretta]

Today I'll be finishing my testing... I'll make them available to everyone to play with when the holiday ends :)
Simo

[Matt Burton]

Of course I recognize these aren't real world numbers. I've got
applications using Windsor that are holding up fine and, as you say,
when all the other factors come into play the overhead of the DI
container pales in comparison. I'm considering Ninject and Autofac as
alternatives to Windsor for their features, not performance. The only
reason I put my numbers out there was my concern about the performance
of Ninject under this particular scenario. Granted, it's not realistic
but the dramatic difference in numbers was a bit alarming, for me at
least.

In any event, I'm regretting jumping on the "me, too" bandwagon here
and posting my findings - I didn't mean any ill will, I have the
deepest respect for what Nate's been able to accomplish with Ninject,
you, Nick, with Autofac, etc... far beyond anything I could dream of
pulling off.

Arbitrary performance benchmarks have no place here and should not be
used as the basis for picking one container over another - the
underlying mechanics are largely the same these days - choose the
container that has the features you need for your applications.

[David]

I am in a similar scenario, where I will be having very long running
services in which I expect many millions of instantiation over the
day. My nightmare scenario is burst speed requirements in which a 10's
and possibly 100's of thousands in a second is possible. If the system
slows down significantly during the burst, it will impact for a short
while after. These bursts come a few times a day (market open/close
for example for each major market), and can last for upwards of 1/2
hour.

David

> On Thu, Jul 3, 2008 at 4:06 PM, Nate Kohari <nkoh...@gmail.com> wrote:
> > Unity is definitely faster than Ninject, which itself is faster than Windsor
> > and StructureMap. However, bear in mind that the difference is insignificant
> > in practice, and only becomes apparent in contrived scenarios like
> > benchmarks.
>
> > Simone actually said it very well -- when activating a small number of
> > instances, the difference is too insignificant to measure. A typical
> > application will have dozens of services, and a very large system might have
> > hundreds, but none will have millions. Further, if you use a database,
> > transmit information across a network, display a user interface, or save
> > information to disk -- which nearly every useful application does -- the
> > cost of using a DI framework will pale in comparison.
>
> > I've used benchmark results to discover performance issues before, but
> > otherwise, while it can be an interesting exercise, it's largely irrelevant
> > to actual performance.
>
> > -Nate
>

[Nate Kohari]

If you are managing that many transient instances, just be sure to pass them to kernel.Release() after you're finished with them. This will tear them down in the same way they were built, and execute lifecycle actions associated with interfaces like IDisposable, IStartable, etc.

I'm not sure what you mean by Ninject slowing down under a heavy load. Can you post your benchmark?


-Nate

[David]

I simulated a heavy load by using a tight loop, and call
IKernel.Get<>. I used Stopwatch.Start before the loop, and
Stopwatch.Stop after. I tested Get<> against a singleton, a transient,
a class with explicit (attribute) constructor injection, a class with
implicit (no attribute) constructor injection, a class with property
injection, and a class with field injection. I found that classes with
any form of injection are roughly 5 times slower that just a transient
and singleton.

I will post the code for my test this weekend.

David

On Jul 4, 9:55 am, "Nate Kohari" <nkoh...@gmail.com> wrote:
> If you are managing that many transient instances, just be sure to pass them
> to kernel.Release() after you're finished with them. This will tear them
> down in the same way they were built, and execute lifecycle actions
> associated with interfaces like IDisposable, IStartable, etc.
>
> I'm not sure what you mean by Ninject slowing down under a heavy load. Can
> you post your benchmark?
>
> -Nate
>

[Simone Chiaretta]

Damn, I should have posted before...
now you are stealing my scoop :)

[David]

Sorry! I will try to be slower next time :)

On Jul 4, 2:20 pm, "Simone Chiaretta" <simone.chiare...@gmail.com>
wrote:

> Damn, I should have posted before...
> now you are stealing my scoop :)
>

> ...
>
> read more »

[Simone Chiaretta]

Just to make sure I understood...
what do you mean with "singleton"?

[Simone Chiaretta]

I mean, in the context of IoC

On Fri, Jul 4, 2008 at 10:58 PM, Simone Chiaretta <simone.c...@gmail.com> wrote:

Just to make sure I understood...
what do you mean with "singleton"?

[David]

It is an object bound in Ninject with the singleton behavior, thus 1
per container. I also test the Ninject specific 1 per thread behavior,
but there is no difference in performance between the two.

David

On Jul 4, 4:58 pm, "Simone Chiaretta" <simone.chiare...@gmail.com>
wrote:

> Just to make sure I understood...
> what do you mean with "singleton"?
>

> ...
>
> read more »

[Simone Chiaretta]

Cool.. so instead of creating one every time, ninject just injects the new dependencies every time?
and is this a thing specific to Ninject or something available also in other IoCc like SM, or Unity?
SImo

[David]

I expect that it is injected only the first time, and after that the
same instance is returned.

Every system I have seen supports this.

David

On Jul 4, 6:54 pm, "Simone Chiaretta" <simone.chiare...@gmail.com>
wrote:

> Cool.. so instead of creating one every time, ninject just injects the new
> dependencies every time?
> and is this a thing specific to Ninject or something available also in other
> IoCc like SM, or Unity?
> SImo
>

> ...
>
> read more »

[Simone Chiaretta]

I didn't know of that approach... sorry... I'm totally a newbie in the IoC world
This might work well when the object contains only methods and doesn't hold state, like the usual repository. While not work well for MVC controllers.
Simo