[Mifos-developer] Meeting schedule analysis and fix

[Chand...@sungard.com]

Hi all!

 

We had an internal competition that had its problem statements aligned with the problems that Mifos currently has. Prabaharan came up with this really good analysis and fix regarding re-architecting  the current generate meeting schedule job. Please find his documentation, fix ( a patch on the trunk) and the jprobe test results and logs attached in this mail.

                                                                                                           

Also note the improvement in the performance of the batch job ( 1322 minutes -> 36 minutes).

 

Regards

Chandan

 

From: Gopalan, Prabaharan
Sent: Monday, November 23, 2009 1:04 PM
To: Thandavarayan, Parthasarathy; Rao, Chandan
Subject: RE: Code-a-thon - Fix and documentation

 

Okay.. sorry for the delay, because I imported the mifosFirstBatch.sql script, which took 2 hrs to import and another 22 hrs for the job (without fix) to complete in that BIIIG database. Here are the stats for the full database:

 

Time taken in minutes:

Pre-fix : 1322.7 (2009-11-21/19:52:53.078/IST WARN, org.mifos.framework.components.batchjobs, GenerateMeetingsForCustomerAndSavingsHelper, execute, 139, GenerateMeetingsForCustomerAndSavings ran in 79362901)

 

Post-fix: 36.53 (2009-11-23/11:45:13.055/IST WARN, org.mifos.framework.components.batchjobs, GenerateMeetingsForCustomerAndSavingsHelper, execute, 139, GenerateMeetingsForCustomerAndSavings ran in 2192621)

 

Performance Improved by 36.722 times

 

Attached is the java file that I modified for the fix. I have two log files (pre and post fixes) amounting to 11 Mb of size. Let me know a place where I can put it so that you can take a look.

Also attached is the documentation about the issue and the details of the fix. To apply the fix, replace the existing HolidayUtils.java with the new one and rebuild the project. I already provided the information about how to debug this application in Tomcat.

 

If you need further information, please let me know.

 

Thanks

Praba.

 

P.S: I haven’t done any architectural changes per se, but just replaced the code fragment that was causing the bottleneck.

 

 

Prabaharan Gopalan • Software Engineer • SunGard • Technology Services • Divyasree Chambers Langford Road Bangalore 560025 India  

Tel +91-80-2222-0501 • Mobile +91-99165-48811  • www.sungard.com/sts

 

 

[John Woodlock]

Very nice work.

[Keith Pierce]

Very nice analysis, Chandan, and a clever fix with dramatic results.

Did you consider using Hibernate's caching mechanisms? If that mechanism resulted in similar improvement in performance, it would have the advantage of being completely transparent to the application code. Another advantage is that the cache can be configured to expire and reload periodically in an automated way.

Keith Pierce

------------------------------------------------------------------------------
Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev

[Prabaharan Gopalan]

Hi John and Keith, thanks..

@keith: there is a hibernate caching mechanism in place right now..
which is configurable. I guess, the cache gets flushed after every x
accounts got processed. x is configurable. But that was not of much
help since the cache was growing every time there is query to get the
holidays. This makes any operation on the cache much slower than the
one before till it gets flushed. Actually, caching was the first thing
that occurred to me too.

On Dec 9, 4:59 pm, Keith Pierce <krp0...@gmail.com> wrote:
> Very nice analysis, Chandan, and a clever fix with dramatic results.
>
> Did you consider using Hibernate's caching mechanisms? If that mechanism
> resulted in similar improvement in performance, it would have the advantage
> of being completely transparent to the application code. Another advantage
> is that the cache can be configured to expire and reload periodically in an
> automated way.
>
> Keith Pierce
>

> On Wed, Dec 9, 2009 at 12:48 AM, <Chandan....@sungard.com> wrote:
> >  Hi all!
>
> > We had an internal competition that had its problem statements aligned with
> > the problems that Mifos currently has. Prabaharan came up with this really
> > good analysis and fix regarding re-architecting  the current generate
> > meeting schedule job. Please find his documentation, fix ( a patch on the
> > trunk) and the jprobe test results and logs attached in this mail.
>
> > Also note the improvement in the performance of the batch job ( 1322
> > minutes -> 36 minutes).
>
> > Regards
>
> > Chandan
>

> > *From:* Gopalan, Prabaharan
> > *Sent:* Monday, November 23, 2009 1:04 PM
> > *To:* Thandavarayan, Parthasarathy; Rao, Chandan
> > *Subject:* RE: Code-a-thon - Fix and documentation

> > *Prabaharan Gopalan* • Software Engineer • SunGard • Technology Services •

[Udai Gupta]

Great, its very interesting and amazing documentation related to
meeting generation. Thanks to Prabaharan for great Job. About the fix
I am with Keith Pierce here, Hibernate should be able to give caching
that has been created in patch, its should be a matter of
configuration.

I am no Hibernate expert but I think the base of having ORM framework
like Hibernate is to provide caching so that we can avoid creating
static caching like in the patch.

AFAIR, Adam Monsen also raised this point that, it is be possible that
using hibernate (Hibernate caching configuration, Hibernate batch
queries enabled) we can speed up things in Batch job and this example
(Analysis from Prabaharan) showshow dramatic impact it could be.

Chandan, can you or Prabaharan create a Improvement issue type at
http://mifosforge.jira.com ?

Thanks,
Udai Gupta

[Prabaharan Gopalan]

something has been misunderstood here or am i the one thinking it has?

I should have been more clearer.. here is the rephrased reply:
Hibernate has a caching mechanism and that was being used previously
in the code. But that was not of much help since the cache was growing

every time there is query to get the holidays. This makes any
operation on the cache much slower than the one before till it gets

flushed. I tried to tweak the caching part of it, but the problem was
elsewhere. so i was forced to look for alternatives.. and that led me
to the holiday retrieval which was being done for every meeting that
was generated which we can avoid by storing (java object caching) it
locally..

I hope that is clear ..

[Udai Gupta]

Dear Prabaharan,

I just missed your last mail before shooting mine :)

I didn't mean to argue. I completely agree with your analysis and fix,
its just I am more concern about the other places in batch job as you
mentioned "elsewhere".

> I tried to tweak the caching part of it, but the problem was elsewhere.

It was just my opinion. I really appreciate what you have done.

Creating a Improvement issue type at http://mifosforge.jira.com will
help us to track down the issue and fix that you have raised.

Thanks,
Udai

[Udai Gupta]

+1 from me for adding Prabaharan's patch to the trunk.

Batch Jobs can have this code because Hibernate caching is a bigger
issue (probably design related) and to resolve that I think we might
need to re-implement the batch jobs.

[Keith Pierce]

Prabaharan, when you refer to the "caching mechanism in place right now", are you referring to the batch job flushing the session after every x accounts? If so, Udai and I are referring to something very different, namely Hibernate's built-in caching. It acts very much like what you've done in your patch, but it is completely transparent to application code, and configurable in an xml configuration file, for example, to re-cache every 24 hours.

Am I on the right track, or have I misunderstood you?

Keith

[Udai Gupta]

some references

http://docs.jboss.org/hibernate/core/3.3/reference/en/html/performance.html#performance-cache
http://ehcache.org/documentation/hibernate.html

I just noticed in hibernate configurations Mifos is using
HashtableCacheProvider, where in the first reference its advised that
HashtableCacheProvider should not be used in production (I haven't dig
more about that).

We have ehcache configuration file in
application/src/main/resources/org.mifos.config.resources.

Probably second level caching for Entities and queries might give us
performance enhancement at various other places in Mifos.

Udai

[Prabaharan Gopalan]

okay.. think that was not clear enough.. i'll use my regular approach
here to explain stuff:
1. The code uses (it still does use) hibernate session cache.. i was
referring to this one in my previous post. Session cache is something
that we can't do without.
2. That session cache grows in size whenever there is a query to a db
occurs.
3. This session cache gets flushed after every 1000 accounts (this
1000 is a number which comes from mifos specific xml config file)
4. Since the session cache grows big for every db call, each update
operation to an entity in the session cache takes longer than the
previous update.
5. I tried to flush the session cache sooner (for every 100/200/500
accounts), though it might seem faster for small number of customer
accounts, in the long run, clearing the session cache proves to be a
costly operation.
6. So i tried to find out what makes the session cache grow bigger and
nailed it down to the holiday utils class which makes db calls for
every meeting that gets generated.(1000s of meetings and 1000s of db
calls).
7. Since the holidays are not going to change for a year, what i did
was to retrieve the holidays the first time they are requested and put
it in a static HashMap. This avoided the 1000s of db calls that were
made ( the "elsewhere" area is actually the holiday utils class).
8. This might not be the case for every batch job, though taking a
look at the hibernate configuration would not hurt to make sure that
things are configured as best as they can be.

I am really sorry if i was confusing in my previous post ( even now i
am little concerned whether this was clear enough :-) ) And sorry
again for the long post...

On Dec 9, 9:35 pm, Udai Gupta <mailt...@gmail.com> wrote:
> some references
>
> http://docs.jboss.org/hibernate/core/3.3/reference/en/html/performanc...http://ehcache.org/documentation/hibernate.html

[Adam Monsen]

Wow, this is indeed excellent work, Prabaharan! Thank you very much
for sharing the information. Sounds like a potentially huge (36x?!)
speedup for all people running Mifos, especially those with large
databases.

I have one suggestion for the patch before we commit it to the trunk:
don't use a static member on HolidayUtils. If the cache is only for
GenerateMeetingsForCustomerAndSavingsHelper, make that batch job
initialize the cache and clear it (in a finally block) upon
completion.

In the long run, we'd much prefer if there were some way to leverage
existing cache infrastructure. I believe we already have ehcache
installed/configured and Udai mentioned that Hibernate has a second-
level cache in addition to the session cache. In general, caching
should be something that is configured once and would "just work", but
I understand and agree with Udai that batch jobs typically need to
behave differently than the normal running application (otherwise why
would we have batch jobs?).

So if we can't do it with Hibernate, have you thought of how we might
leave management of this cache up to ehcache?

http://mifosforge.jira.com/browse/MIFOS-2623

[Prabaharan Gopalan]

Thanks Adam.. using a static member was an oversight on my part.. i
completely forgot about the cleaning up aspect. Sorry about that..
i'll try to take a look at the ehcache.. that is something i have
never used but heard of several times. might not happen in the next
couple of days as my plate(s) are full for the time being.. let me see
what can be done.. Thanks again for the suggestion..

[John Woodlock]

Prabaharan can correct me if I'm wrong but the current change made will impact any piece of functionality that regenerates schedules.

If memory serves me correctly, in addition, to this "GenerateMeetingsForCustomerAndSavingsTask" batch job, it also impacts "RegenerateScheduleTask" batch job as well as loan disbursals (made after the anticipated disbursement date).

This change may also give some impetus to completely removing these two batch jobs mentioned above during Gazelle C for example (if scalability is a key focus) and catering for them within the operational system (unless there is something other non-performance reason they need to run just after midnight).


John

[Prabaharan Gopalan]

As far as my understanding of the code, it shouldn't affect any
functionality. If anything, it might improve the performance of the
other jobs that use the HolidayUtils class, since i haven't changed
any method names or anything noticeable from the perspective of an
external class/module. That class will still behave the same way as it
did before. But, I haven't tested the other jobs or pieces that use
the modified code.

On Dec 10, 11:21 am, John Woodlock <john.woodl...@gmail.com> wrote:
> Prabaharan can correct me if I'm wrong but the current change made will
> impact any piece of functionality that regenerates schedules.
>
> If memory serves me correctly, in addition, to this
> "GenerateMeetingsForCustomerAndSavingsTask" batch job, it also impacts
> "RegenerateScheduleTask" batch job as well as loan disbursals (made after
> the anticipated disbursement date).
>
> This change may also give some impetus to completely removing these two
> batch jobs mentioned above during Gazelle C for example (if scalability is a
> key focus) and catering for them within the operational system (unless there
> is something other non-performance reason they need to run just after
> midnight).
>
> John
>

> On Thu, Dec 10, 2009 at 7:15 AM, Adam Monsen <hair...@gmail.com> wrote:
>
> > If the cache is only for
> > GenerateMeetingsForCustomerAndSavingsHelper, make that batch job
> > initialize the cache and clear it (in a finally block) upon
> > completion.
>
>
>

[Keith Woodlock]

first off, I think the performance gain from Prabaharan's change is
significant and will improve the experience of users of the system so
it should go in. (post 1.4 i presume.. or can we be a bit more fluid
with our deliveries?)

john said:
>> This change may also give some impetus to completely removing these two batch jobs
>> mentioned above during Gazelle C for example (if scalability is a key focus) and catering for >> them within the operational system (unless there is something other non-performance
>> reason they need to run just after midnight).

I completely agree with this. The core problem is the existence of
these batch jobs in the first place which appears to be a 'crude
approach' to the problem of schedules and meetings. The need for
batching this operation disappears when the logic for creating
meetings/schedules is pushed back into the application domain.

Removing the batch jobs would be a great simplification of the system
and would also ease setup and operation of the application (moving
towards mifos as a platform, would be ideal not to need this batch
jobs); all good reasons for a release of its own.

On Hibernate Second Level cache talk:
I am curious as to what exactly you were going to cache here. I wasn't
aware that we were using 'second level cache' in the application, i
thought it was only the 'session cache'. Its seems then that EHCache
is our second level cache provider and most of my experience has being
using this only for 'read only' or 'Non strict read/write' caching
strategy for which the second level cache is very good (so data that
is only read and never updated e.g. seed data, or seldom updated with
non strict strategy), with regards the other caching strategies
(read/write & Transactional) I believe that these are a little
trickier to work with, you have to be careful and thoroughly test the
behaviour to verify it is doing what you think it should be doing.

KeithW

[John Woodlock]

Prabaharan, Chandan

Not sure if this fix is in 1.4 but are you going to apply and run it on the GK production application/database?

I'd be interested to see what the improvement is on that database (over the course of a week say) because I'd originally thought the number of customer_schedule rows was the biggest contributor to how long that batch job took at GK.



John

[Chand...@sungard.com]

Hi john

 

I have updated GK regarding these developments. I am trying to see if they can test this on their test server. I will compile and provide them a holiday utils class file that will have the new code in it.

 

Regards

Chandan

[Adam Feuer]

On Tue, Dec 15, 2009 at 1:10 AM, <Chand...@sungard.com> wrote:
> I have updated GK regarding these developments. I am trying to see if they
> can test this on their test server. I will compile and provide them a
> holiday utils class file that will have the new code in it.

Chandan,

Please do not do this. We strongly discourage customers from making
changes like this to production and staging systems.

I will respond off list to explain this further.

-adam
--
Adam Feuer <adamf at pobox dot com>

------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev

[Adam Monsen]

> Thanks Adam.. using a static member was an oversight on my part.. i
> completely forgot about the cleaning up aspect. Sorry about that..
> i'll try to take a look at the ehcache.. that is something i have
> never used but heard of several times. might not happen in the next
> couple of days as my plate(s) are full for the time being.. let me see
> what can be done.. Thanks again for the suggestion..

Just checking in. Has any progress been made on this?

[John Woodlock]

Just adding to Adam's question as to the progress on this one.

Is the fix okay to go as it is (i.e. been verified to work) or is more work needed on it?

Either way, I think it would be very useful to aim to have Prabaharan's fix (or something doing the same thing) in the next mifos release.  Testing a change on my own machine (and including Prabaharan's fix), I was able to update over 70k customer accounts in less than 15 mins and I believe 70k would be a huge challenge to production GK right now. 

Also, it would mean not having to focus on batch jobs during Gazelle D and hopefully they (the big ones anyway) should start to be removed as the changes Keith W is looking at get implemented.

John

[Prabaharan Gopalan]

folks, sorry for being quite for a long time.. am busy with other delivery commitments as of now and in fact somewhat lost the thread. If my memory serves me right, then we were discussing the use of static list for caching the holidays and i haven't gotten an opportunity to get into the code after that. But as long as, we cache it using some mechanism (like a caching framework) or we clear the cache once the batch job completes, i am sure that fix will work without any concerns for memory leaks and the problem of a dirty cache which might never get updated since it is static.

I am terribly sorry that I couldn't contribute more time as i would have liked.

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com

--
with luv,
GP.

[John Woodlock]

Prabaharan,

thanks for the reply.  The piece of work you did was a very big help.  I understand the points you make and will have a go at ensuring it runs okay for the batch job and other places which could make of use it (without getting dirty).

John

[Adam Monsen]

On Tue, 2010-02-09 at 16:26 +0530, Prabaharan Gopalan wrote:
> we were discussing the use of static list for caching the holidays

Prabaharan, thanks for following up!

Given the huge potential performance increase, I propose we go ahead
with your original patch ( http://tinyurl.com/y8oybpj ), but just
include a warning log message if the static cache of holidays ends up
getting to be large (say, > 100 or so). The warning could be something
like: "holiday cache is too large. Consider using a caching framework
like ehcache instead".

[Keith Woodlock]

Adam,

Just on this, John and myself have talked about this and its on our
list of quick wins:
http://www.mifos.org/developers/wiki/ScalabilityAndArchitectureDiscussion#batch-jobs

Prabaharan did a good analysis of this and correctly identified that
it was the 'Holidays' that were causing a lot of the problems with the
slowness of this batch job. While Prabaharan's patch will work, but
having looked at this problem today I would prefer to refactor this
area towards the arcitectural roadmap rather than place the static map
for retaining holidays. The problem area revolves around the
HolidayUtils class and the code that uses all its methods. see
MeetingBO.getAllDates(int o).

Regards,
KeithW.

> ------------------------------------------------------------------------------
> SOLARIS 10 is the OS for Data Centers - provides features such as DTrace,
> Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW
> http://p.sf.net/sfu/solaris-dev2dev
>

------------------------------------------------------------------------------
SOLARIS 10 is the OS for Data Centers - provides features such as DTrace,
Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW
http://p.sf.net/sfu/solaris-dev2dev

[John Woodlock]

Prabaharan's fix is now in trunk... though its implementation has been changed to fit in with the new method for generating dates for schedule installments.

I've tested this fix on my machine and here's the outcome.

73k customers (having an average of a years customer schedules behind them) had their schedules updated in 20 mins.

It's unlikely that GK would get to updating 73k customers in a night anytime soon (it represents one in every 5/6 customers needing new installments).  However, GK has more customer schedules than my database has.  It'll be interesting to see the figures during proper testing but, as far as I can see, when GK upgrade to Gazelle C release, the batch job should vary between 15mins to 30mins.

I tested with 4 holidays and with no holidays and the timing were almost exactly the same.

John

[Prabaharan Gopalan]

good to hear that.. :) .. makes me think about contributing more.. probably after a about a month or so, i should be able to shell out more time..

Thanks folks..

------------------------------------------------------------------------------
SOLARIS 10 is the OS for Data Centers - provides features such as DTrace,
Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW
http://p.sf.net/sfu/solaris-dev2dev

--
with luv,
GP.