Session leak in NHLinq Query Plan Cache

[Lauri Kotilainen]

Hi, folks.

I've been debugging an issue with a moderately high-traffic web site that uses NHibernate extensively. Periodically, during peak load times, one of the two nodes serving the site goes unresponsive, and during that time, extremely high memory usage is observed. My hypothesis is that the unresponsiveness is related to an ongoing Full GC. 

I expected the issue to mostly be related to the NHibernate usage patterns of the site, but during debugging I found something else: it looks like whenever a session is used for the first, uncached execution of a query, the Query Plan Cache will hold a reference to that NhQueryable (and hence, that session) via the Expression Tree generated by the Linq extension methods.

Here is an example of a SOS.dll !gcroot dump:

Thread 2264:

*** WARNING: Unable to verify checksum for System.ni.dll

    0000000018e6ecb0 000007fef6638de6 System.Net.TimerThread.ThreadProc()

        r12:  (interior)

            ->  00000006fff977e8 System.Object[]

            ->  00000003fff90658 System.Web.Hosting.ObjectCacheHost

            ->  00000006800b5cd0 System.Collections.Generic.Dictionary`2[[System.Runtime.Caching.MemoryCache, System.Runtime.Caching],[System.Web.Hosting.ObjectCacheHost+MemoryCacheInfo, System.Web]]

            ->  00000006800b5df0 System.Collections.Generic.Dictionary`2+Entry[[System.Runtime.Caching.MemoryCache, System.Runtime.Caching],[System.Web.Hosting.ObjectCacheHost+MemoryCacheInfo, System.Web]][]

            ->  00000004fffc3248 System.Runtime.Caching.MemoryCache

            ->  00000004fffc3298 System.Object[]

            ->  00000004fffc4410 System.Runtime.Caching.MemoryCacheStore

            ->  00000004fffc44e0 System.Runtime.Caching.CacheExpires

[... removed repetitive cache timer instances ...]

            ->  00000003fff8da38 System.Web.RequestTimeoutManager

            ->  00000003fff8da70 System.Object[]

            ->  00000003fff8db20 System.Web.Util.DoubleLinkList

            ->  0000000403438450 System.Web.RequestTimeoutManager+RequestTimeoutEntry

            ->  0000000403436ee8 System.Web.HttpContext

            ->  0000000502f505b0 System.Collections.Hashtable

            ->  0000000502f856d8 System.Collections.Hashtable+bucket[]

            ->  0000000502f50660 Autofac.Core.Lifetime.LifetimeScope

            ->  0000000502f506e0 System.Collections.Generic.Dictionary`2[[System.Guid, mscorlib],[System.Object, mscorlib]]

            ->  0000000502f91678 System.Collections.Generic.Dictionary`2+Entry[[System.Guid, mscorlib],[System.Object, mscorlib]][]

            ->  0000000502f51e58 NHibernate.Impl.SessionImpl

            ->  00000001fff942e8 NHibernate.Impl.SessionFactoryImpl

            ->  00000001fff95588 NHibernate.Engine.Query.QueryPlanCache

            ->  00000001fff96918 NHibernate.Util.SoftLimitMRUCache

            ->  00000001fff969b0 NHibernate.Util.LRUMap

            ->  00000001fff969e0 NHibernate.Util.SequencedHashMap+Entry

            ->  00000001002beb80 NHibernate.Util.SequencedHashMap+Entry

            ->  00000001002b9930 NHibernate.Engine.Query.HQLExpressionQueryPlan

            ->  00000001002b9290 NHibernate.Linq.NhLinqExpression

            ->  00000001002b9410 System.Linq.Expressions.MethodCallExpressionN

            ->  00000001002b93f0 System.Runtime.CompilerServices.TrueReadOnlyCollection`1[[System.Linq.Expressions.Expression, System.Core]]

            ->  00000001002b93c0 System.Object[]

            ->  00000001002b91c0 System.Linq.Expressions.ConstantExpression

            ->  00000001002b9188 NHibernate.Linq.NhQueryable`1[[REDACTED]]

            ->  00000001002b91a8 NHibernate.Linq.DefaultQueryProvider

            ->  00000001002b3f08 NHibernate.Impl.SessionImpl

Here's a XUnit test that seems to corroborate my current theory:

public class Test {

public Guid Id { get; set; }

}

public class SessionLeakTest {

[Fact]

public void SessionGetsCollected() {

WeakReference reference = null;

new Action(() => {

var sessionFactory = ConfigureSessionFactory();

var session = sessionFactory.OpenSession();

// Comment this line out to make the test pass

var query = session.Query<Test>().FirstOrDefault(t => t.Id != Guid.Empty);

session.Dispose();

reference = new WeakReference(session, true);

})();

GC.Collect();

GC.WaitForPendingFinalizers();

Assert.Null(reference.Target);

}

private ISessionFactory ConfigureSessionFactory() {

var cfg = new Configuration();

cfg.Proxy(p => p.ProxyFactoryFactory<DefaultProxyFactoryFactory>())

.DataBaseIntegration(db => {

db.ConnectionString = "Data Source=:memory:";

db.ConnectionReleaseMode = ConnectionReleaseMode.OnClose;

db.Driver<SQLite20Driver>();

db.Dialect<SQLiteDialect>();

})

.SessionFactory().GenerateStatistics();

cfg.Properties.Add("hbm2ddl.keywords", "auto-quote");

return cfg.BuildSessionFactory();

}

}

The point of the test is not to actually query for instances of the Test entity, but to ensure that the Query Plan Cache is invoked. I'm aware that GC.Collect is not required to do anything and GC details are implementation-specific, but running this test on .NET 4.0 fails -- unless you comment out the part where the query is performed. 

Given that sessions will hold references the entities loaded through them, this sort of explains a large part of the problem I'm seeing. 

I'd appreciate if somebody could prove me wrong (or right!), and possibly point me in the right direction with regards to getting this fixed. :)