NHibernate - Wrong Columns on Queries

[Tom]

I'm getting an intermittant problem with NHibernate where it generates a query for an entity, but replaces one of the columns with a column from completely different (and unrelated) entity. It only ever replaces a single column, and is generally solved by restarting the application (though sometimes it takes a couple of attempts). ASP.NET application (.NET 4.0) SessionFactory created during Application_Start NHibernate 3.3.1- All mappings/configuration done via Mapping By Code Using Nhibernate Criteria Any input on this would be much appreciated! Entity public class LiquiditySourceItem : RunDataEntity, IEntity<int> { public virtual int Id { get; protected internal set; } public virtual int IdentID { get; protected internal set; } public virtual string Portfolio { get; protected internal set; } public virtual string ProfitCentre { get; protected internal set; } public virtual DateTime? MaturityDate { get; protected internal set; } public virtual string Curr1 { get; protected internal set; } public virtual string Curr2 { get; protected internal set; } public virtual decimal Reval { get; protected internal set; } public virtual string ContractType { get; protected internal set; } public virtual string ContractType2 { get; protected internal set; } public virtual string ContractCode { get; protected internal set; } public virtual decimal AmountSignedTradeUnit { get; protected internal set; } public virtual decimal Amount2Signed { get; protected internal set; } public virtual decimal SpotDelta { get; protected internal set; } public virtual string TradeRevalCurr { get; protected internal set; } } Entity Mapping public LiquiditySourceItemMap() { Id(x => x.Id, map => map.Column("RowId")); Property(x => x.IdentID, map => map.Column("IdentID")); Property(x => x.Portfolio, map => map.Column("Portfolio")); Property(x => x.ProfitCentre, map => map.Column("ProfitCentre")); Property(x => x.MaturityDate, map => map.Column("Con_Expiry")); Property(x => x.BuySell, map => map.Column("BS")); Property(x => x.Curr1, map => map.Column("Curr1")); Property(x => x.Curr2, map => map.Column("Curr2")); Property(x => x.Reval, map => map.Column("Reval")); Property(x => x.ContractType, map => map.Column("ContractType")); Property(x => x.ContractType2, map => map.Column("ContractType2")); Property(x => x.ContractCode, map => map.Column("ContractCode")); Property(x => x.AmountSignedTradeUnit, map => map.Column("AmountSignedTradeUnit")); Property(x => x.Amount2Signed, map => map.Column("Amount2Signed")); Property(x => x.ValSpot, map => map.Column("Val_Spot")); Property(x => x.SpotDelta, map => map.Column("SpotDelta")); Property(x => x.TradeRevalCurr, map => map.Column("Traderevalcurr")); Property(x => x.SourceReport, map => map.Column("SourceReport")); ManyToOne(x => x.RunContext, map => map.Column("RunContextID")); Table("Staging.vw_Liquidity"); } Report Entity public class BusinessBreakdownStandardPosition : ReportRunDataEntity, IEntity<long> { public virtual long Id { get; set; } public virtual decimal FinalNettingAmountUSD { get; set; } public virtual decimal InitialChargeAmountUSD { get; set; } public virtual BusinessBreakdownInitialPrr InitialPrr { get; set; } public virtual IEnumerable<FinalInstrumentPosition> FinalInstrumentPositions { get; set; } public virtual decimal CreditEventPaymentUSD { get; set; } public virtual decimal ValuationChangeIncreaseUSD { get; set; } public virtual decimal ValuationChangeDecreaseUSD { get; set; } public virtual string ReportKey { get; set; } public virtual decimal USDCharge { get; set; } public virtual decimal USDChargeICG { get; set; } public virtual string InstrumentType { get; set; } } Report Entity Mapping public class BusinessBreakdownStandardPositionMap : ClassMapping<BusinessBreakdownStandardPosition> { public BusinessBreakdownStandardPositionMap() { Id(x => x.Id, m => { m.Column("BusinessBreakdownStandardPositionID"); m.Generator(Generators.HighLow, g => g.Params( new { table = "dbo.HiValue", max_lo = 10000, Where = string.Format("EntityName = 'BusinessBreakdownStandardPosition'") })); }); Property(x => x.FinalNettingAmountUSD, map => map.Column("FinalNettingAmountUSD")); Property(x => x.InitialChargeAmountUSD, map => map.Column("InitialAmountUSD")); Property(x => x.CreditEventPaymentUSD); Property(x => x.ValuationChangeDecreaseUSD); Property(x => x.ValuationChangeIncreaseUSD); Property(x => x.USDCharge); Property(x => x.USDChargeICG); Property(x=>x.InstrumentType); ManyToOne(p => p.RunContext, map => map.Column("ReportRunContextID")); ManyToOne(p => p.InitialPrr, m => { m.Column("InitialPrrID"); m.Cascade(Cascade.All); }); Property(x => x.ReportKey); Bag(x => x.FinalInstrumentPositions, collectionMapping => { collectionMapping.Table("Reporting.BusinessBreakdownFinalInstrumentPositionStandardPositionMap"); collectionMapping.Cascade(Cascade.All); collectionMapping.Key(k => k.Column("StandardPositionID")); }, mapping => mapping.ManyToMany(y => y.Column("FinalInstrumentPositionID"))); Table("Reporting.BusinessBreakdownStandardPosition"); } } SQL Query, Generated By NHibernate SELECT this_.RowId AS RowId47_0_, this_.IdentID AS IdentID47_0_, this_.Portfolio AS Portfolio47_0_, this_.ProfitCentre AS ProfitCe4_47_0_, this_.Con_Expiry AS Con5_47_0_, this_.BS AS BS47_0_, this_.Curr1 AS Curr7_47_0_, this_.Curr2 AS Curr8_47_0_, this_.Reval AS Reval47_0_, this_.ContractType AS Contrac10_47_0_, this_.ContractType2 AS Contrac11_47_0_, this_.ContractCode AS Contrac12_47_0_, this_.AmountSignedTradeUnit AS AmountS13_47_0_, this_.Amount2Signed AS Amount14_47_0_, this_.Val_Spot AS Val15_47_0_, this_.SpotDelta AS SpotDelta47_0_, this_.InitialAmountUSD AS Initial17_47_0_, this_.RunContextID AS RunCont18_47_0_, this_.SourceReport AS Sou19_47_0_ FROM Staging.vw_Liquidity this_ Exception System.Data.SqlClient.SqlException (0x80131904): Invalid column name 'InitialAmountUSD'. As you can see, nhibernate has replaced the LiquiditySourceItem column 'Traderevalcurr' with 'InitialAmountUSD', which belongs to the BusinessBreakdownStandardPosition entity. These entities have no relationship whatsoever. Otherwise, the SQL is exactly as you'd expect( including column order). Observations The wrong column is always a valid column in a different mapped entity The wrong column will replace an existing one The order of columns remains intact Everything else with the SQL query is otherwise correct The issue sometimes ocurrs between other entities. Again, there's no relationship between these Any thoughts? http://stackoverflow.com/questions/21382756/nhibernate-wrong-columns-on-queries http://stackoverflow.com/questions/16991838/nhibernate-generating-wrong-columns-on-queries

[Richard Wilde]

Does this fail on both DEV and PROD servers?

 

One thought, not sure if it will help BUT if it fails on DEV you may be able to…

 

·         Generate the XML files on build session factory.

·         Look to see if all is OK with the XML files…. Check and double check

·         Comment out your configuration for the mapping-by-code `AddDeserializedMapping`

·         Embed the XML as resource

·         Use the XML mappings when building sessionfactory and test again

 

This at least (*might*) rule in/out if it is a mapping-by-code issue…

 

Rippo

--
You received this message because you are subscribed to the Google Groups "nhusers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nhusers+u...@googlegroups.com.
To post to this group, send email to nhu...@googlegroups.com.
Visit this group at http://groups.google.com/group/nhusers.
For more options, visit https://groups.google.com/groups/opt_out.

[Tom]

Yes, it fails on both DEV and PROD servers.

I've tried generating the .hbm.xml mapping files and have compared those from a sucessful run and those when it's thrown this exception. However, both sets of mappings appear fine, and they're completely identical.This suggests to me that NHibernate isn't actually using these generated files to perform the db/entity mapping, rather it's doing it programatically

The problem is that it's such an intermittant bug. I've just tried running the code over 20 times on the DEV server, but it only failed once. Currently, can't figure out a way to consistently repicatle the issue, which is making it very hard to debug!

I believe though that the issue may have something to do with missing the column names on the mapping classes. It seems that explicitly adding these columns tends to prevent the error from ocurring, i.e. changing 'Property(x => x.USDCharge);' to 'Property(x => x.USDCharge, map => map.Column("USDCharge"));'

Any thoughts on why this may be the case?

[Richard Wilde]

“This suggests to me that NHibernate isn't actually using these generated files to perform the db/entity mapping, rather it's doing it programmatically”

 

The XML or (mapping-by-code that is injected) is used by the session factory upon creation…

 

Can you post the code the builds the sessionfactory?

 

Also what does this 'Property(x => x.USDCharge);’ translate  to in XML terms? That is when you write out XML what is shown for this property?

 

I must admit I have used NH for many years and have never seen this behaviour, but it does seem odd to be sure…. I am sure if this was a bug then it would have been picked up before. Do you share multiple entities with same mapping file or vice versa?

 

In my case (note I never use map.Column unless the column names are not the same):-

 

Property(x => x.NewsDate, x => x.NotNullable(true));

Translates to

<property name="NewsDate" not-null="true" />

 

And this

Property(x => x.NewsDate);

Translates to

<property name="NewsDate" />

 

Both these XML representations are fine, I prefer the first one as I tend not to use nulls.

--

[Tom]

Sure, here's the SessionFactory:

configuration.DataBaseIntegration(dbi =>
{
    dbi.Dialect<MsSql2008Dialect>();
    dbi.Driver<SqlClientDriver>();
    dbi.ConnectionProvider<DriverConnectionProvider>();
    dbi.IsolationLevel = IsolationLevel.ReadCommitted;
    dbi.BatchSize = 200;
    dbi.Timeout = 255;
    dbi.LogFormattedSql = true;
    dbi.ConnectionString = configurationService.ConnectionString;
    dbi.OrderInserts = false;
});
configuration.Cache(x => x.UseQueryCache = true);
configuration.SessionFactory().Caching.Through<SysCacheProvider>().WithDefaultExpiration(60);
var mapperserialized = mapperFactory.Create().CompileMappingForAllExplicitlyAddedEntities();
configuration.AddDeserializedMapping(mapperserialized, "NhibConfig");

foreach (string assemblyName in GetUniqueAssemblyNames(assemblyNames))
{
    configuration.AddAssembly(assemblyName);
}

configuration.BuildSessionFactory();


In terms of XML, 'Property(x => x.USDCharge);' translates to '<property name="ReportKey" />', which is as I'd expect.

As you said, I don't usually specify a column name in the mapping file unless it's different from the property, but as I mentioned before, doing so for this entity seems to resolve the problem with the incorrect column.I'd still like to get to the bottom of the issue though, as I've seen the same problem ocurr for different entities, both of which contain explicitt column name mappings.

No, we don't share mappings between entities and no entities have multiple mappings. The only slightly unusual thing is that most/all entities extend a base class, and these base properties are mapped by each derived classes' mapping files. Other than that, can't see anything else out of the ordinary with our set-up!

[vitalya]

Hello Tom,

We've beend experiencing this issue mostly on test environments. After several rounds of investigation we did not find what is causing it. For now we're suspecting that bug is reproduced on servers with low performance and is somehow related to concurency. Have you managed to solve this problem?

...

[Tom]

Hi vitalya,

Unfortunately not. We added explicit column names to the mappings and the problem seems to have been resolved - since doing that about 6 weeks ago, we've not since had this issue (though I'm not convinced that's actually solved the issue, and it's certainly not a permanent solution).

For reference, we mainly get it on our DEV environments, and occasionally UAT and production environments too. Our dev servers (where the issue occurs the vast majority of times), is a reasonably powerful continuous integration server (using TeamCity), but the it's shared between multiple projects, running multiple builds, and also all dev & uat environments sit here too, so there may not  always be that much resources  available  for each build.

Thanks,
Tom

[Shawn Gresham]

We've been running into this for while now (a couple of years at least) in our production web garden.  The problem seemed to appear more often as we slowly switched our codebase over from XML to Mapping-By-Code so I am guessing it is probably some sort of race condition in that logic.  The production server is also pretty weak, so I am fairly certain that system stress does play a part in it.  We've had it appear on our development workstations probably 3 or 4 times only.

The last time it occured it was a mapping that had Column() specification, so that does not appear to resolve our issue here.  In fact, most of our column names are specified due to poor column naming on our legacy databases.

I've been capturing mini-dumps of the corrupt sessions and looking over them with windbg. I've found SingleTableEntityPersister's with the incorrect column so I know it goes back that far.  I can't seem to catch one before the HbmMappings get garbage collected. :(

[Shawn Gresham]

I was actually able to make some good progress on this today.  I have a test environment that can consistently reproduce the problem.  

A year or so ago, I had created a simple console application that built up a mapping and compared it against a baseline list of columns to attempt to detect incorrect column names.  I then created a testing framework that would run this application repeatedly and logged the results.  After a weekend of continuous runs, I got nothing.

After reading the posts here, I rebuilt the framework replacing the console application with a ASP.NET web service that configured the mappings in Application_Start and did the comparison against the baseline there.  With this setup, I got a failure within the first 40 iterations and probably had 5 or 6 by the time I hit 100 iterations.  I modified the testing framework to perform a minidump on IIS after detecting a failure and was able to determine via WinDbg that the column name error can be tracked all the way back to the HbmProperty definition. 

My next step is to boil all of this down into something I can post to the NH bug list and hope they take a look at it.

Hello Tom,

<c

...

[Oskar Berggren]

Keep up the good work Shawn! The asp.net environment is definitely more susceptible to race conditions than a simple console app.

/Oskar

For more options, visit https://groups.google.com/d/optout.

[Shawn Gresham]

I found the bug.  Now I just need to figure out the where/who/how to get it fixed.  Did anyone here create a bug on JIRA for this issue?

For anyone interested in the details, the problem code is in PropertyPath.Equals(PropertyPath) which attempts to determine equality by only using the hash code.  This works fine for smaller code bases as the default Object.GetHashCode() returns a sequential object index.  However, after garbage collection, these indices get reused as finalized objects are removed and new objects are created (see documentation on GetHashCodeEx for details) which results in more than one object getting the same hashcode.

This causes our problem because PropertyPath is used as a key for the dictionary that holds the property customizers (.Column(), .Length(), etc.) for each property in the mapping. Once garbage collection kicks in, property paths have a chance to share the same hashcode which means they will ultimately mix up their customizers for the colliding properties, thus the wrong column names.

Likely the reason why this problem was more prevalent in ASP.NET is because of the sheer object count added by the ASP.NET library and the way it handles garbage collection.

If you have your own copy of the NH source, you can fix the bug by changing NHibernate/Mapping/ByCode/PropertyPath.cs line #66 from:

return hashCode == other.GetHashCode();

To:

return hashCode == other.GetHashCode() && ToString() == other.ToString();

Shawn

[Oskar Berggren]

Nice work! I don't have access to the code at the moment, but it must be decided if PropertyPath should have reference equality or by-value equality. Based on your excerpt, the current code seems to use reference equality (but in a broken way), while your proposed change makes it neither (since it compares both the value and the reference based hashcode, it's a mix of semantics).

Please have a search in JIRA to see if you find something that looks similar.

/Oskar

[Shawn Gresham]

I can see that.  I would definitely want someone to fix it who has a firmer understanding of how changing this class would impact NH's handling of compositing and polymorphism.

I couldn't find any outstanding bugs that appear to fit the description of what we are seeing, so I created a new one:

https://nhibernate.jira.com/browse/NH-3638

--
You received this message because you are subscribed to a topic in the Google Groups "nhusers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nhusers/BZoBoyWQEvs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nhusers+u...@googlegroups.com.

[Oskar Berggren]

Thanks for your work on this. A 3.3.4 release with the fix will be available shortly. Please report back with the results.

/Oskar

[Shawn Gresham]

The fix appears to be working perfectly.  We ran 2000 iterations of our testing framework after upgrading to 3.3.4 without a single failure.  I'll start it again tonight before I head home and let it run overnight, but I don't expect to encounter anything.

Thank you for quick turn-around and the 3.3.4 release!

Shawn

...