Memory Leak when running NUnit Tests

[MattO]

Folks,

I have about 285 unit tests in a project that is using s#harp
architecture 1.6.

The memory usage continues to grow to over a gigabyte before finally
running into a OutOfMemory Exception at which point every unit test
after that point fails.

I've tried TestDriven .NET and the DevExpress Test Runner, both
exhibit the same behavior.

I ran an Ants Memory Profiler trace and the issue *appears* to be due
to the sessionFactories dictionary within the NHibernateSession.cs
file that seems to stick around even though Reset() function within
NHibernateSession.cs is ultimately called during the TearDown process
within RepositoryTestsBase.cs.

The Reset function is suppose to dispose of the NHibernate session
object, but I don't think that is actually happening.

I tried the following:

foreach (ISession session in Storage.GetAllSessions())
{
session.Dispose();
session = null; // <--- This can not be done because its in
a foreach loop
}

Anyone have any ideas on how to fix this issue so I can actually run
all unit tests to completion?

[Seif Attar]

That is strange, I have worked on projects using sharp architecture with 1000+ tests and never had a problem :s 

A colleague of mine had an issue with an outofmemoryexception, which turned out to be caused by the fact that he was setting up windsor for each test and not disposing. Are you doing anything with windsor in your tests? sessionFactories is static, and doesn't get disposed when you dispose the sessions.

Sent from my phone

---

--
You received this message because you are subscribed to the Google Groups "S#arp Architecture" group.
To post to this group, send email to sharp-arc...@googlegroups.com.
To unsubscribe from this group, send email to sharp-architect...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sharp-architecture?hl=en.

[MattO]

Seif,

So here is a bit more details:

I've made a slight modification to DatabaseRepositoryTestsBase as
follows:

public abstract class DatabaseRepositoryTestsBaseSQL
{
[SetUp]
public virtual void SetUp()
{

RepositoryTestsHelper.InitializeNHibernateSessionSQL(Assembly.GetCallingAssembly().Location,
Assembly.GetCallingAssembly().GetName().Name + ".Validators");

NHibernateSession.CurrentFor(NHibernateSession.DefaultFactoryKey).BeginTransaction();

NHibernateSession.CurrentFor("nHibernate.AS400").BeginTransaction();
}

[TearDown]
public virtual void TearDown()
{

NHibernateSession.CurrentFor(NHibernateSession.DefaultFactoryKey).Transaction.Rollback();

NHibernateSession.CurrentFor("nHibernate.AS400").Transaction.Rollback();

NHibernateSession.CurrentFor(NHibernateSession.DefaultFactoryKey).Dispose();
NHibernateSession.CurrentFor("nHibernate.AS400").Dispose();
RepositoryTestsHelper.Shutdown();
}
}

This is so I can setup my NHibernate sesions to use two different
database servers.

This is the other part:

public static void InitializeDatabaseSQL(string
validationAssemblyPath, string validationAssemblyNamespace)
{
Configuration cfg =
InitializeNHibernateSessionSQL(validationAssemblyPath,
validationAssemblyNamespace);
}

public static Configuration
InitializeNHibernateSessionSQL(string validationAssemblyPath, string
validationAssemblyNamespace)
{
string[] mappingAssemblies = GetMappingAssemblies();
Configuration cfg = NHibernateSession.Init(new
SimpleSessionStorage(), mappingAssemblies, "hibernate.config");

//Add our AS400 configuration as well, default init will
load the AS400
NHibernateSession.AddConfiguration("nHibernate.AS400",
mappingAssemblies, null,
"hibernate.AS400.config", null, null, null);

ValidatorEngine _validatorEngine;
var provider = new NHibernateSharedEngineProvider();
Environment.SharedEngineProvider = provider;
_validatorEngine =
Environment.SharedEngineProvider.GetEngine();
var configure = new FluentConfiguration();
configure
.Register(Assembly.LoadFile(validationAssemblyPath).GetTypes().Where(t
=>

t.Namespace.Equals(validationAssemblyNamespace)).ValidationDefinitions())
.SetDefaultValidatorMode(ValidatorMode.UseExternal)
.IntegrateWithNHibernate.AvoidingListenersRegister();
_validatorEngine.Configure(configure);

NHibernateSession.ValidatorEngine = _validatorEngine;

return cfg;
}

And lastly my unit tests look like this for the Setup and Teardown:

[SetUp]
public override void SetUp()
{
base.SetUp();
managerFactory = new ManagerFactory();
manager = managerFactory.GetFormManager();
}
[TearDown]
public override void TearDown()
{
base.TearDown();
managerFactory = null;
manager = null;
}

protected IFormManager manager;
protected ManagerFactory managerFactory;

[Test]
public void Create()
{
Form entity = CreateNewForm();

object result = manager.Save(entity);
manager.DbContext.CommitChanges();

Assert.IsNotNull(result);
}

Anyone see any problems with this? I don't...

Matt

[Chris Bingham]

This is a Long shot, but Im pretty sure Nunit doesn't run your tear
down if an exception is thrown during setup. Doesn't look like this
should cause you a problem, unless you are doing more stuff in the test
setups (and you would probably have lots of failing tests too)

Sent from my Windows Phone
From: MattO
Sent: 14/12/2011 15:42
To: S#arp Architecture
Subject: [sharp-architecture] Re: Memory Leak when running NUnit Tests
Seif,

NHibernateSession.CurrentFor(NHibernateSession.DefaultFactoryKey).BeginTransaction();

NHibernateSession.CurrentFor("nHibernate.AS400").BeginTransaction();
}

NHibernateSession.CurrentFor(NHibernateSession.DefaultFactoryKey).Transaction.Rollback();

NHibernateSession.CurrentFor("nHibernate.AS400").Transaction.Rollback();

t.Namespace.Equals(validationAssemblyNamespace)).ValidationDefinitions())
.SetDefaultValidatorMode(ValidatorMode.UseExternal)
.IntegrateWithNHibernate.AvoidingListenersRegister();
_validatorEngine.Configure(configure);

NHibernateSession.ValidatorEngine = _validatorEngine;

return cfg;
}

Assert.IsNotNull(result);
}

Matt

--

[MattO]

FYI,

I don't get any exceptions or anything until I get the out of memory
exception.

Additionally, I moved all of my Unit tests back to inheriting from
DatabaseRepositoryTestsBase and the issue still occurs. (I removed
all my other unit tests that used the other database).

So now only the unit tests that interact with the one SQL 2008 R2
database remain. I still however get memory that grows forever.

From what I've read is that some object is maintaing a reference to
one of those static variables, not sure where though. The only clue I
could gather from ANTS profiler is this object in NHibernateSession.cs
seems to stick around:

private static Dictionary<string, ISessionFactory> sessionFactories =
new Dictionary<string, ISessionFactory>();

Atleast thats what I think ANTS profiler is telling me, all the memory
seems to be rooted in that dictionary so some how the sessions in the
dictionary are not getting disposed of because something else has a
reference to them (not sure what it could be though).

Matt

On Dec 14, 12:40 pm, Chris Bingham <ccb...@gmail.com> wrote:
> This is a Long shot, but Im pretty sure Nunit doesn't run your tear
> down if an exception is thrown during setup. Doesn't look like this
> should cause you a problem, unless you are doing more stuff in the test
> setups (and you would probably have lots of failing tests too)
>
> Sent from my Windows Phone
> From: MattO
> Sent: 14/12/2011 15:42
> To: S#arp Architecture
> Subject: [sharp-architecture] Re: Memory Leak when running NUnit Tests
> Seif,
>
> So here is a bit more details:
>
> I've made a slight modification to DatabaseRepositoryTestsBase as
> follows:
>
>   public abstract class DatabaseRepositoryTestsBaseSQL
>   {
>     [SetUp]
>     public virtual void SetUp()
>     {
>

> RepositoryTestsHelper.InitializeNHibernateSessionSQL(Assembly.GetCallingAss­embly().Location,
> Assembly.GetCallingAssembly().GetName().Name + ".Validators");
>
> NHibernateSession.CurrentFor(NHibernateSession.DefaultFactoryKey).BeginTran­saction();

>
> NHibernateSession.CurrentFor("nHibernate.AS400").BeginTransaction();
>     }
>
>     [TearDown]
>     public virtual void TearDown()
>     {
>

> NHibernateSession.CurrentFor(NHibernateSession.DefaultFactoryKey).Transacti­on.Rollback();

[Seif Attar]

Right, I have written a test to chk this, and I can see memory usage increasing, didn't get to investigatw further, and won't be able to until the weekend :s

Sent from my phone

---

[MattO]

Thanks for looking into this. I was about write a reproducible test
for everyone but I think you saved me some time :-)

From what I can tell the NHibernate sessions are sticking around. So
something is not getting disposed of because there are references
still around. From reading around static variables are also a big
culprit for this... I just don't know where to check, there are a lot
of static variables in the s#harp source.

ANTS profiler says there are hundreds of NHibernate sessions within
the following dictionary:

private static Dictionary<string, ISessionFactory> sessionFactories =
new Dictionary<string, ISessionFactory>();

My objects are quite large with lots of child object "relationships"
using many-to-one, etc type of bags in my .hbm files. Some tables are
70+ columns long and can contain NVARCHAR(max) data, so thats probably
why the memory pressure in my case is so extreme. It must be keeping
all that retrieved data in some of NHibernate's caches or
something...

On Dec 15, 2:12 am, "Seif Attar" <i...@seifattar.net> wrote:
> Right, I have written a test to chk this, and I can see memory usage increasing, didn't get to investigatw further, and won't be able to until the weekend :sSent from my phoneOn 14 Dec 2011 15:43, MattO <xan...@gmail.com> wrote:Seif,

[Seif Attar]

Does is say there hundreds of sessions? Or factories?

Sent from my phone

---

[MattO]

Hundreds of NHibernate.Impl.SessionFactoryImpl, which in turn has
hundreds of Dictionary<string, Entity.IEntityPersistor> objects, and
thousands of NHibernate.Persister.Entity.SingleTableEntityPersister
objects, then 30,000+ NHibernate.Loader.Entity.EntityLoader Instances
which contains 30,000+ NHibernate.SqlCommand.SqlString instances

The NHibernate.SqlCommand.SqlString instances seem to be taking the
massive amount of data, but they are all rooted in the
Dictionary<string, ISessionFactory>

Here is the object graph. http://www.mediafire.com/?946199su9bfsjq3

Note this doesn't show "gigabytes of data" that I was describing in
this snapshot session It was just a snapshot after running about 20%
of the unit tests I have thus the relatively low amount of memory
usage at the point of this memory capture (138.43 megabytes). This is
calculated by all the string storage that NHibernate is using which
doesn't seem to be getting disposed of because the session factory
doesn't seem to be getting disposed of. I just added up 78.04 + 37.52
+ 22.87 that you can find at the end of the object graph.

Assuming you can create a long enough running unit test library to
capture a baseline snapshot and a working snapshot within one testing
session you can use Ants Profiler 14-day trial on the unit test runner
(you may have to turn off concurrent garbage collection, but the
program will tell you how to do that with your test runner). From
this you should be able to see the same object graph that I am seeing
along with the memory allocations for those object graphs.

Matt

On Dec 15, 9:03 am, "Seif Attar" <i...@seifattar.net> wrote:
> Does is say there hundreds of sessions? Or factories?Sent from my phoneOn 15 Dec 2011 14:53, MattO <xan...@gmail.com> wrote:Thanks for looking into this. I was about write a reproducible test

[MattO]

One last thing, you can probably see this with even a small unit test
library. To perform the capture, just put in
System.Threading.Thread.Sleep(1500000) on the last unit test to run to
give ANTS Profiler time to capture the trace, and perhaps a Sleep for
a much short amount of time at the beginning of the trace so you can
capture the baseline memory.

> > Does is say there hundreds of sessions? Or factories?Sent from my phoneOn 15 Dec 2011 14:53, MattO <xant...@gmail.com> wrote:Thanks for looking into this. I was about write a reproducible test

> > for everyone but I think you saved me some time :-)
> > From what I can tell the NHibernate sessions are sticking around. So
> > something is not getting disposed of because there are references
> > still around. From reading around static variables are also a big
> > culprit for this... I just don't know where to check, there are a lot
> > of static variables in the s#harp source.
> > ANTS profiler says there are hundreds of NHibernate sessions within
> > the following dictionary:
> > private static Dictionary<string, ISessionFactory> sessionFactories =
> > new Dictionary<string, ISessionFactory>();
> > My objects are quite large with lots of child object "relationships"
> > using many-to-one, etc type of bags in my .hbm files. Some tables are
> > 70+ columns long and can contain NVARCHAR(max) data, so thats probably
> > why the memory pressure in my case is so extreme. It must be keeping
> > all that retrieved data in some of NHibernate's caches or
> > something...
> > On Dec 15, 2:12 am, "Seif Attar" <i...@seifattar.net> wrote:

> > > Right, I have written a test to chk this, and I can see memory usage increasing, didn't get to investigatw further, and won't be able to until the weekend :sSent from my phoneOn 14 Dec 2011 15:43, MattO <xant...@gmail.com> wrote:Seif,

[Seif Attar]

I am not around a pc at the moment, but can you try removing the code that inits nh from setup, and put in assembly setup? So that session factory is created once? That might get you going for now

Sent from my phone

---

[MattO]

Seif,

Sorry for being a bit dense here. What exactly should I try? Can you
give me an example?

1. Do I stop inheriting from DatabaseRepositoryTestsBase in my Unit
test?
2. If I do #1, are you saying to just call this instead from my
SetUp method on my Unit Tests?

NHibernateSession.Init(new SimpleSessionStorage(),
mappingAssemblies, autoPersistenceModel);

?

On Dec 15, 10:28 am, "Seif Attar" <i...@seifattar.net> wrote:
> I am not around a pc at the moment, but can you try removing the code that inits nh from setup, and put in assembly setup? So that session factory is created once? That might get you going for nowSent from my phoneOn 15 Dec 2011 16:18, MattO <xan...@gmail.com> wrote:One last thing, you can probably see this with even a small unit test

[Seif Attar]

In the code you gave in previous email, there is a setup method that calls the helper to init NH, remove that call from the setup, and move it to assembly setup.

I am pretty sure that should sort out Ur problem until we figure out what is wrong with that base class

Sent from my phone

---

[MattO]

Well I tried that, I still get the memory leak issue:

public static void Setup()
{
string[] mappingAssemblies =
RepositoryTestsHelper.GetMappingAssemblies();
SimpleSessionStorage storage = new SimpleSessionStorage();
Configuration cfg = NHibernateSession.Init(storage,
mappingAssemblies, "hibernate.config");

//Add our AS400 configuration as well, default init will load
the AS400
NHibernateSession.AddConfiguration("nHibernate.AS400",
mappingAssemblies, null,
"hibernate.AS400.config", null, null, null);

string validationAssemblyPath =
Assembly.GetExecutingAssembly().Location;
string validationAssemblyNamespace =
Assembly.GetExecutingAssembly().GetName().Name + ".Validators";

ValidatorEngine _validatorEngine;
var provider = new NHibernateSharedEngineProvider();

NHibernate.Validator.Cfg.Environment.SharedEngineProvider =
provider;
_validatorEngine =

NHibernate.Validator.Cfg.Environment.SharedEngineProvider.GetEngine();

var configure = new FluentConfiguration();
configure
.Register(Assembly.LoadFile(validationAssemblyPath).GetTypes().Where(t
=>

t.Namespace.Equals(validationAssemblyNamespace)).ValidationDefinitions())
.SetDefaultValidatorMode(ValidatorMode.UseExternal)
.IntegrateWithNHibernate.AvoidingListenersRegister();
_validatorEngine.Configure(configure);

NHibernateSession.ValidatorEngine = _validatorEngine;
}

Here is my unit tests now:

[SetUp]
public override void SetUp()
{

UnitTestSetup.Setup();

base.SetUp();
managerFactory = new ManagerFactory();

manager = managerFactory.GetAS400_CBTFORMManager();
}

Here is the SetUp method in DatabaseRepositoryTestBase.cs:

public virtual void SetUp()
{
//RepositoryTestsHelper.InitializeNHibernateSession();
NHibernateSession.Current.BeginTransaction();
}

Not really sure what else to try. Something is causing that
dictionary to not be disposed of.

On Dec 15, 2:32 pm, "Seif Attar" <i...@seifattar.net> wrote:
> In the code you gave in previous email, there is a setup method that calls the helper to init NH, remove that call from the setup, and move it to assembly setup.

> I am pretty sure that should sort out Ur problem until we figure out what is wrong with that base classSent from my phoneOn 15 Dec 2011 20:21, MattO <xan...@gmail.com> wrote:Seif,

[Seif Attar]

I guess I didn't explain the point of what needs to be done, you need to ensure that the call to UnitTestSetup.Setup is only called once for all tests, nunit site seems down, but use this as a guide:

http://frazzleddad.blogspot.com/2010/03/warming-up-sites-before-running.html

Hope that helps.

Sent from my phone

---

[MattO]

Gotcha.

I did this and it solved the problem. However, the only thing that
concerns me is that now NHibernate session only gets created once
instead of a new one for each unit test. I sort of wish I could
create a new one each time.

Thats not a deal breaker for me. However there is another issue, the
database I am using has identity columns and I've seen scenarios where
even after a rollback transaction the entity gets committed to the
database. I'm hoping using .Clear() will prevent this but I'm not so
sure according to this: http://stackoverflow.com/questions/2853117/nhibernate-session-flush-evict-vs-clear

So the end result is all 285 unit tests only use about 65MB of memory,
and it completes about 2000% faster since it only creates the
NHibernate session once.

Here is what my code ended up looking like:

1. I created a unit test setup class that gets executed only once at
the beginning of the unit tests (using the link you gave me regarding
SetUpFixture).

[SetUpFixture]
public class UnitTestSetup : DatabaseRepositoryTestsBase
{

[SetUp]
public override void SetUp()
{

string location =
System.IO.Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location)
+ "\\DataAccessLayer.dll";
RepositoryTestsHelper.InitializeDatabaseSQL(location,
"DataAccessLayer.Validators");
}

[TearDown]
public override void TearDown()
{

RepositoryTestsHelper.Shutdown();
}
}

2. The unit tests themselves still call the Setup and TearDown
procedures, but I've removed the initialization and Shutdown logic as
follows from DatabaseRepositoryTestHelper. The only thing they do now
is create a new transaction and rollback transactions and clear the
session at the end. Perhaps I should do a CLEAR before the rollback
to avoid the potential issue i've run into in the past where you
rolled back transactions:

public abstract class DatabaseRepositoryTestsBase
{
[SetUp]

public virtual void SetUp()
{
//RepositoryTestsHelper.InitializeNHibernateSession();
NHibernateSession.Current.BeginTransaction();
}

[TearDown]
public virtual void TearDown()
{
NHibernateSession.Current.Transaction.Rollback();
NHibernateSession.Current.Clear(); //Added clear just to try be
safe, hoping this really does evict and clear everything even when
using identity columns
//RepositoryTestsHelper.Shutdown();
}
}

On Dec 16, 10:52 am, "Seif Attar" <i...@seifattar.net> wrote:
> I guess I didn't explain the point of what needs to be done, you need to ensure that the call to UnitTestSetup.Setup is only called once for all tests, nunit site seems down, but use this as a guide:http://frazzleddad.blogspot.com/2010/03/warming-up-sites-before-running.html

[Seif Attar]

Glad you can run them tests now! If you want to create a new session for each test, then add

NHibernateSession.CurrentFor(NHibernateSession.DefaultFactoryKey).Dispose();
NHibernateSession.Storage.SetSessionForKey(NHibernateSession.DefaultFactoryKey, null);

to the end of the tear down that runs for each test. SharpArch will create a new session when you next try to get the session.

dont know how to fix the nh native identity generator not respecting rollback, I just tried it and it works fine.

[MattO]

Awesome thanks for all the help!!!

On Dec 16, 8:10 pm, Seif Attar <i...@seifattar.net> wrote:
> Glad you can run them tests now! If you want to create a new session for
> each test, then add
>

> NHibernateSession.CurrentFor(NHibernateSession.DefaultFactoryKey).Dispose()­;
> NHibernateSession.Storage.SetSessionForKey(NHibernateSession.DefaultFactory­Key,

> null);
>
> to the end of the tear down that runs for each test. SharpArch will create
> a new session when you next try to get the session.
>
> dont know how to fix the nh native identity generator not respecting
> rollback, I just tried it and it works fine.
>

> On 16 December 2011 19:36, MattO <xant...@gmail.com> wrote:
>
>
>
> > Gotcha.
>
> > I did this and it solved the problem.  However, the only thing that
> > concerns me is that now NHibernate session only gets created once
> > instead of a new one for each unit test.  I sort of wish I could
> > create a new one each time.
>
> > Thats not a deal breaker for me.  However there is another issue, the
> > database I am using has identity columns and I've seen scenarios where
> > even after a rollback transaction the entity gets committed to the
> > database.  I'm hoping using .Clear() will prevent this but I'm not so
> > sure according to this:

> >http://stackoverflow.com/questions/2853117/nhibernate-session-flush-e...

> >http://frazzleddad.blogspot.com/2010/03/warming-up-sites-before-runni...

> > > Hope that helps.Sent from my phoneOn 16 Dec 2011 15:19, MattO <

> > xant...@gmail.com> wrote:Well I tried that, I still get the memory leak