Struggling with NHibernate memory consumption.. advise?

[Kyle]

We have just started to convert to NHibernate for our data access
layer. Our legacy database has roughly 400 tables with on average 125
columns per table. Almost all tables have a composite key. Our
application is broken up into hundreds of smaller applications that
each have their own memory space. A normal user would use roughly 3-5
applications at once time, and each application may access 15 - 20
tables in a single session.

So, here's my problem... Loading all the hbm.xml mapping files and
400 domain objects during the BuildSessionFactory() call consumes
roughly 150 MB of memory per application before any data access
occurs. What can we do to lower the memory consumption? I'm pretty
new to NHibernate, so any advice would help.

Thanks,
Kyle

[Paulo Quicoli]

Just a suggestion.... think about how much Firefox our Chrome browser consumes...  just for illustration, right now  my Firefox (with only one page opened) is consuming 141MB..

I dont think  150MB is so high for a 400 entities domain....  

Nowadays, memory is cheap and new machines are coming up in a 3, 4 Gb RAM systems, and I'm talking about Brazil.... (wich is always one step before)....

2011/5/16 Kyle <ky...@kylefisher.org>

--
You received this message because you are subscribed to the Google Groups "nhusers" group.
To post to this group, send email to nhu...@googlegroups.com.
To unsubscribe from this group, send email to nhusers+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nhusers?hl=en.

--
Paulo Quicoli
Editor Técnico: .NET Magazine, ClubeDelphi Magazine; Editor: WebMobile Magazine
www.devmedia.com.br
www.nhibernatebrasil.net
http://pauloquicoli.spaces.live.com
twitter:@pauloquicoli

[Kyle]

Well, the issue compounds itself when our clients run our application
published in Citrix. Then we quickly get to 10 - 20 users and
possibly 50 to 100 applications running from one server. Then we end
up using a ton of memory at 150 megs per app. I would like to
dynamically load domain objects at run-time, but I don't think this is
possible with NHibernate. Is it? I may have to look at EF4 or some
other solution that uses less memory. Any other suggestions?

Thanks,
Kyle

On May 16, 9:25 am, Paulo Quicoli <pauloquic...@gmail.com> wrote:
> Just a suggestion.... think about how much Firefox our Chrome browser
> consumes...  just for illustration, right now  my Firefox (with only one
> page opened) is consuming 141MB..
>
> I dont think  150MB is so high for a 400 entities domain....
>
> Nowadays, memory is cheap and new machines are coming up in a 3, 4 Gb RAM
> systems, and I'm talking about Brazil.... (wich is always one step
> before)....
>

> 2011/5/16 Kyle <k...@kylefisher.org>

[José F. Romaniello]

2011/5/16 Kyle <ky...@kylefisher.org>

 I may have to look at EF4 or some
other solution that uses less memory.

Yes, that is a good idea! 

Would you mind to re-map your 400 domain objects with EF, and when your application is all running in the same way than with nhibernate, tell us the result about memory consuption?

It will be really really nice if you can write a blog-post or something with a comparisoon.

thanks,

[Ramon Smits]

I agree with José.

:-)

But... why on earth are you spawning that much applications that all access the same database? Isn't that an awful waste of resources?

Besides that.. the whole purpose of NHibernate is to dynamically load your domain entities when needed via lazy loading. Seems to me that you are not using lazy loading and immediately load all data in memory. 

Are you keeping your sessions alive? As you should not do this. 

If you are fairly new then start with a small application and do not immediately start with such a huge model! When I hear you are having tables with more then 100 columns... spawning hundreds of process... dude.. your problem really is not anything related with NHibernate......

--

Ramon

--

You received this message because you are subscribed to the Google Groups "nhusers" group.
To post to this group, send email to nhu...@googlegroups.com.
To unsubscribe from this group, send email to nhusers+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nhusers?hl=en.

--

Ramon

[Kyle]

Unfortunately it is a legacy application with a legacy database that
we do not have the time to rework. No greenfield work here. :( I
agree it is a large waste of resources, but it is what it is. I'm
actually not doing any loading of data at this point. Everything is
setup to be lazy from what I understand, even the domain objects.

This call is where memory goes from ~18 MB to ~165 MB:
sessionFactory = configuration.BuildSessionFactory();

I do keep the session open for an application/process right now. How
can I guarantee that everything is lazy loaded?

Thanks,
Kyle

On May 16, 10:16 am, Ramon Smits <ramon.sm...@gmail.com> wrote:
> I agree with José.
>
> :-)
>
> But... why on earth are you spawning that much applications that all access
> the same database? Isn't that an awful waste of resources?
>
> Besides that.. the whole purpose of NHibernate is to dynamically load your
> domain entities when needed via lazy loading. Seems to me that you are not
> using lazy loading and immediately load all data in memory.
>
> Are you keeping your sessions alive? As you should not do this.
>
> If you are fairly new then start with a small application and do not
> immediately start with such a huge model! When I hear you are having tables
> with more then 100 columns... spawning hundreds of process... dude.. your
> problem really is not anything related with NHibernate......
>
> --
> Ramon
>
> On Mon, May 16, 2011 at 4:57 PM, José F. Romaniello

> <jfromanie...@gmail.com>wrote:
>
>
>
>
>
>
>
>
>
> > 2011/5/16 Kyle <k...@kylefisher.org>

[Walter Poch]

Can't you partition your domain, per Application?

Or at least, make a Service Layer, which will consume that 140MB, but 'll be only one running for all the users and applications.

A long shoot,

Regards,

2011/5/16 Kyle <ky...@kylefisher.org>

--
Saludos,

Walter G. Poch
Sr. .Net Developer
--------------------------------------------
Cell: +54 (9 341) 3353273
walte...@gmail.com

[Ramon Smits]

On Mon, May 16, 2011 at 5:44 PM, Kyle <ky...@kylefisher.org> wrote:

Unfortunately it is a legacy application with a legacy database that
we do not have the time to rework.  No greenfield work here. :(  I
agree it is a large waste of resources, but it is what it is.  I'm
actually not doing any loading of data at this point.  Everything is
setup to be lazy from what I understand, even the domain objects.

This call is where memory goes from ~18 MB to ~165 MB:
sessionFactory = configuration.BuildSessionFactory();

I do keep the session open for an application/process right now.  How
can I guarantee that everything is lazy loaded?

All relationship mappings in NHibernate are lazy by default but a lot of beginners think lazy is slower and make them non lazy resulting in fetching a whole aggregate or sometimes even all database data when they just wanted to select one record.

You should close the session when you can. You should have a session per unit of work. Then when you want to do something again you should create a new session. You should create a session factory once and reuse that all the time. Creating sessions is *very* cheap.

When you have created the session factory you mention that the process is occupying 140mb. Is that the memory you see in the task manager? Because .net is managed it will not release memory immediately. You could try to create the session factory and then let the garbage collector free memory by calling GC.Collect(). Only do this for testing purpose! Then please report the memory is actually still using after initialization.

By the way, just out of curiosity, why do you spawn all those processes? Do all these processes run under different credentials? What is the reason for this design. How do you communicate with all these processes? Via WCF, .net remoting, sockets? What do you use?

--

Ramon

[Kyle]

Way may have to partition the domain, or create some sort of system
where the application layer can inform the data access layer what
objects it intends to use, and then build the session factory with
only those in it. Of course, that is still less than ideal.

On May 16, 12:31 pm, Walter Poch <walter.p...@gmail.com> wrote:
> Can't you partition your domain, per Application?
>
> Or at least, make a Service Layer, which will consume that 140MB, but 'll be
> only one running for all the users and applications.
>
> A long shoot,
>
> Regards,
>

> 2011/5/16 Kyle <k...@kylefisher.org>

> walter.p...@gmail.com

[Kyle]

In the DAL, we create sessions and close sessions immediately when we:
get data, insert data, update data, or delete data. We create longer
running sessions for transactions that end with a commit or a
rollback. Otherwise, the sessions are closed. I tried GC.Collect()
immediately after BuildSessionFactory() and there was maybe a one meg
change. Nothing much. As for our product, it stems from a legacy DOS
application so most design decisions were made on that a long time
ago, including architecture. We use a wide array of methods of
sharing info between processes on the same machine and separate
machines (WM_* messages, MSMQ, etc) but generally each application
acts independently of each other. Ideally in the long term they would
slowly merge into a common business layer, and that's what we are
working towards, along with a new common DAL. Just the DAL is causing
us issues right now. :)

Thanks again.

On May 16, 1:04 pm, Ramon Smits <ramon.sm...@gmail.com> wrote:

[Walter Poch]

I don't think that sharing a big fat DAL is a good approach. Look at this posts:

• Repository is the new Singleton - http://ayende.com/blog/3955/repository-is-the-new-singleton
• The DAL should go all the way to UI - http://ayende.com/blog/3958/the-dal-should-go-all-the-way-to-ui
• More about Object Relational Mappers vs. Custom DAL - http://ayende.com/blog/1132/more-about-object-relational-mappers-vs-custom-dal
• The false myth of encapsulating data access in the DAL- http://ayende.com/blog/4567/the-false-myth-of-encapsulating-data-access-in-the-dal
• Enhanced Query Object -http://fabiomaulo.blogspot.com/2010/07/enhanced-query-object.html

Regards,

2011/5/16 Kyle <ky...@kylefisher.org>

--
You received this message because you are subscribed to the Google Groups "nhusers" group.
To post to this group, send email to nhu...@googlegroups.com.
To unsubscribe from this group, send email to nhusers+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nhusers?hl=en.

--

Saludos,

Walter G. Poch
Sr. .Net Developer
--------------------------------------------
Cell: +54 (9 341) 3353273

walte...@gmail.com

[Kyle]

Those are good if I'm going to use only NHibernate in the future.
What if I'm going to use EF4 (or EF5) someday, or go back to using
DataSets and DataTables? With a Repository pattern at least my access
to the DAL is consistent across all code and abstracted so I don't
have to update it. If specific HQL or LINQ gets into the source code,
then we have to update a lot of source to replace the DAL. Anyway,
thanks for the articles. They are very interesting reads.

I did a quick test of the same data in EF4. Memory usage was roughly
40 MB for initialization plus querying a single customer vs.
NHibernate's 150 MB for initialization plus a single customer query.
It is a significant difference and likely acceptable. I'd still like
to get NHibernate down lower if it is possible because I prefer its
flexibility compared to EF4. Any other layers I should look at?
Thanks.

On May 16, 2:14 pm, Walter Poch <walter.p...@gmail.com> wrote:
> I don't think that sharing a big fat DAL is a good approach. Look at this
> posts:
>

>    - Repository is the new Singleton -
>    http://ayende.com/blog/3955/repository-is-the-new-singleton<http://ayende.com/blog/3955/repository-is-the-new-singleton>
>    - The DAL should go all the way to UI -
>    http://ayende.com/blog/3958/the-dal-should-go-all-the-way-to-ui<http://ayende.com/blog/3958/the-dal-should-go-all-the-way-to-ui>
>    - More about Object Relational Mappers vs. Custom DAL -
>    http://ayende.com/blog/1132/more-about-object-relational-mappers-vs-c...<http://ayende.com/blog/1132/more-about-object-relational-mappers-vs-c...>
>    - *The false myth of encapsulating data access in the DAL- *
>    http://ayende.com/blog/4567/the-false-myth-of-encapsulating-data-acce...
>    - Enhanced Query Object -
>    http://fabiomaulo.blogspot.com/2010/07/enhanced-query-object.html
>
> Regards,
>
> 2011/5/16 Kyle <k...@kylefisher.org>

> walter.p...@gmail.com

[Jason Meckley]

changing data access strategies is highly unlikely. it's a completely technical problem so only devs care about it, not business persons. Not to mention dumbing down data access to the lowest common denominator will hurt the design and ability to performance tune the system; rather than improve it.

this type of encapsulation is foolish and doesn't provide any real value.

[Kyle]

Anyway, this still doesn't solve the initial problem I am having.

Is there any way to dynamically add a class mapping at runtime, post
BuildSessionFactory()? I'm looking at removing as many mappings as
possible for non-critical tables. The non-critical tables can be
retrieved using an ISqlQuery and using
Transformers.Transform.AliasToBean<T> to properly map the data to
their domain object. That works great. The issue now is updating
these rarely used tables. I'd like a way to be able to add a mapping
so can call session.Save() or session.Update(). Is there any remote
way to do this that doesn't incur a terrible performance hit? Thanks.

More testing in EF4 revealed that in the end it uses about the same
memory, just allocates it in different ways at different times.

[Raul Carlomagno]

apart from memory size, i would like to know how much the app takes
to initialize, lots of hmb deserialization, i would partition the
whole domain in sub systems like bounded contexts in ddd, and i would
build an application server offering services for its clients and this
app server would use nh internally, that could be a good way to
downsize this

[Fabio Maulo]

which is the cost of 2GB DDR3 1333Mhz in your country ?

How many hours you are spending to solve this ""problem"" ?

Which is the price of each of your hours ?

Perhaps would be better if you talk with your boss.

[Gustavo Rocha]

i dont agree that think about memory price solve the problem.

Of course it is cheap. 

But the fact is that NH i using much more memory actually than other versions. Is just BuildSessionFactory.

Take attention on my problem:

I use FluentNH, about 400 entities, use Envers for auditory and have a multi tenancy application. One session factory per host.

Each session factory consumes about 150mb.

We have comercial plans to get hundreds of clients.

Since I have hoirzontal or vertical scalabilty on hardware thats a LOT of memory.

I think everyone agree that I must have a singleton ready sessionfactory per client, right?!

I tried to put these sf´s in a standalone service who configure all and serve my webapps. But still... a LOT o memory. In the service and in the apps

I am trying everything to solve this without leaving my requirements.

[Ricardo Peres]

Have you tried profiling your application, specifically, trying to find what parts of the session factory are taking all that memory?

It really doesn't seem normal to me...

RP

[Gunnar Liljas]

This thread is 3 years old....just sayin'

Yes, one session factory per tenant is preferable, and yes, they consume too much memory. This is something that needs to be addressed.

A possible remedy could be to spin up new session factories in the background (on a different thread) and perform a bit of recycling logic.

/G

--

You received this message because you are subscribed to the Google Groups "nhusers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nhusers+u...@googlegroups.com.

To post to this group, send email to nhu...@googlegroups.com.

Visit this group at http://groups.google.com/group/nhusers.
For more options, visit https://groups.google.com/d/optout.