[Mifos-developer] Issue with changing center meeting day

[Prachi....@sungard.com]

Hi,

 

We came across a case where the “view details” page of a member was showing the wrong date under “Upcoming charges”. This applied to all members and groups belonging to a particular center. The center however showed the correct date.

 

On further analysis we found that the customer_schedule table had wrong dates generated for the groups and members. The center meeting day of this particular center was changed from Tuesday to Thursday on Thursday (22-10-2009). After the batch job ran, it updated the future meeting dates in customer_schedule table with wrong dates (far in the future in 2010 in some cases, and in some cases it updated with dates in 2008). The loan schedule was updated correctly for these members.

 

We are using mifos version 1.3.0. We replicated this on a local mifos instance - See replication path below.

 

This seems to be a bug in mifos. The RegenerateScheduleTask batch job is not updating the customer schedule table correctly for members and groups.

 

Scenario Replication Path:

I replicated the scenario in my system and changed the meeting date of a center from Tuesday to Thursday on Thursday (22-10-2009).The customer schedules were correct before this meeting change was done. After the batch jobs ran the same problem came up. The customer schedules got updated starting from some random date in the future/past. The same problem came irrespective of whether I changed the meeting day from Tuesday to Thursday or Tuesday to Monday.

I also created a new member before changing the center meeting day. On creation of the member, the customer schedule table had the correct action dates. The members are applied a membership fees which they have to pay on the first meeting day. After changing the center meeting day, and after the batch jobs ran, the first meeting day got changed to some date in 2010. Hence I was unable to apply payment for the membership fees.

 

 

Is there any way to fix this scenario?

 

It is a high priority issue as members won’t be able to make the membership fee payment if the customer schedule is not correct. Also, the collection sheet page will not work properly since the meeting date is incorrect in the customer schedule table.

 

Thanks & Regards,

Prachi

 

 

 

 

 

[Jeff Brewster]

>We came across a case where the "view details" page of a member was
showing the wrong date under "Upcoming charges". This applied to all
members and >groups belonging to a particular center. The center however
showed the correct date.
>
>On further analysis we found that the customer_schedule table had wrong
dates generated for the groups and members. The center meeting day of
this >particular center was changed from Tuesday to Thursday on Thursday
(22-10-2009). After the batch job ran, it updated the future meeting
dates in >customer_schedule table with wrong dates (far in the future in
2010 in some cases, and in some cases it updated with dates in 2008).
The loan schedule >was updated correctly for these members.

Hi Prachi,
This sounds similar to a behavior noticed around savings accounts for
the same batch job - [Issue 2479] Changing Center Meeting date causes
multiple issues with savings
(https://mifos.dev.java.net/issues/show_bug.cgi?id=2479). Can you take
the latest trunk build and retest this behavior on your local instance
to see if the fix by John Woodlock for this issue resolves your issue as
well?

Here is a link to the latest stable trunk build's mifos war:
http://tinyurl.com/yftcrhj. Note this is a development build and is not
ready for production use!

Also, there are comments in this meeting requirements page
(http://www.mifos.org/developers/wiki/MeetingRequirementsForCustomers)
about how to make changes to meeting times - look under "Changing
Meeting Times". However, it appears from your description that you
followed those guidelines.

Jeff

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

[John Woodlock]

Prachi,

It's worth trying what Jeffs says... however I don't think my fix
(which was to do with a certain type of savings account) relates to
what you are saying.

On the other hand... I didn't see anything like the behaviour you are
seeing when I was testing and I did quite a lot of tests (which
involved looking at customer accounts as well) so, who knows, maybe
between your version and the current dev version someone has done
something that fixes it.

John

>Hi Prachi,
>This sounds similar to a behavior noticed around savings accounts for
>the same batch job - [Issue 2479] Changing Center Meeting date causes
>multiple issues with savings
>(https://mifos.dev.java.net/issues/show_bug.cgi?id=2479). Can you take
>the latest trunk build and retest this behavior on your local instance
>to see if the fix by John Woodlock for this issue resolves your issue as
>well?

[Prachi....@sungard.com]

Hi,

I replicated the issue with the latest trunk war - revision 16040.

Here is the link to the video of the issue replication -
http://rapidshare.com/files/298116038/mifos.flv.html

Regards,
Prachi

[Jeff Brewster]

>-----Original Message-----
>From: Prachi....@sungard.com [mailto:Prachi....@sungard.com]
>Sent: Monday, October 26, 2009 5:07 AM
>To: mifos-d...@lists.sourceforge.net
>Subject: Re: [Mifos-developer] Issue with changing center meeting day
>

>Hi,
>
>I replicated the issue with the latest trunk war - revision 16040.
>
>Here is the link to the video of the issue replication -
>http://rapidshare.com/files/298116038/mifos.flv.html
>

Prachi,
Thanks for the video. I watched and was able to see the action_date
dates in table customer_schedule show up as 2007-04-06, 2007-04-19, etc.
after executing batch jobs. Do you know if this same behavior occurs
when you don't change the center meeting to land on current day of the
week?

Also, would you be willing to post this as an issue in our issue
tracker? Here are details on our issue tracker if you haven't used it
before: http://www.mifos.org/developers/testing/issue-tracker.

Thank you!
Jeff

[Jeff Brewster]

>Prachi,
>Thanks for the video. I watched and was able to see the action_date
>dates in table customer_schedule show up as 2007-04-06, 2007-04-19,
etc.
>after executing batch jobs. Do you know if this same behavior occurs
>when you don't change the center meeting to land on current day of the
>week?
>
>Also, would you be willing to post this as an issue in our issue
>tracker? Here are details on our issue tracker if you haven't used it
>before: http://www.mifos.org/developers/testing/issue-tracker.
>
>Thank you!
>Jeff

Prachi,
I spent time trying to reproduce this today with smaller data sets with
no luck. Can you try it with a smaller data set and see if you see the
problem then as well? Also, Partha mentioned looking at the Regenerate
Schedule code - do you know if he's done that?

Regards,

[Jeff Brewster]

Prachi and Partha,
I created a new issue tracker issue today for this issue -
https://mifos.dev.java.net/issues/show_bug.cgi?id=2551

As noted below, I haven't been able to see this behavior on my test
environment with a smaller data set. Were you able to try a smaller
data set test? I'm curious if the issue you are seeing has to do with
the number of records processed.

Thanks,
Jeff

>>Also, would you be willing to post this as an issue in our issue
>>tracker? Here are details on our issue tracker if you haven't used it
>>before: http://www.mifos.org/developers/testing/issue-tracker.
>>
>>Thank you!
>>Jeff
>
>Prachi,
>I spent time trying to reproduce this today with smaller data sets with
>no luck. Can you try it with a smaller data set and see if you see the
>problem then as well? Also, Partha mentioned looking at the
Regenerate
>Schedule code - do you know if he's done that?
>

[Prachi....@sungard.com]

Hi Jeff,

I was able to replicate this issue with a smaller data set also. The
database has one branch data with 312 customers and 245 loan accounts.

Here is the link to the video -
http://rapidshare.com/files/299960064/mifos2.rar.html

And yes this behavior occurs irrespective of what day I change the
center meeting day.

Regards,
Prachi

[Udai Gupta]

Hi Prachi,

Can you also upload the compressed (zip or tar) mysql dump (sql file)
of your small dataset if that is a possible.

Thanks,
Udai

[Udai Gupta]

> Can you also upload the compressed (zip or tar) mysql dump (sql file)
> of your small dataset if that is a possible.
>

and by small dataset I meant the whole mifos database that you have
created using one branch with 312 customers and 245 loan accounts.

[Prachi....@sungard.com]

Hi,

Here is the link to the database that I used-

http://rapidshare.com/files/299990215/oneBranchData.rar.html

Regards,
Prachi

-----Original Message-----
From: Udai Gupta [mailto:mail...@gmail.com]
Sent: Friday, October 30, 2009 4:59 PM
To: Mifos software development

[Jeff Brewster]

>-----Original Message-----
>From: Prachi....@sungard.com [mailto:Prachi....@sungard.com]

>Sent: Friday, October 30, 2009 4:16 AM
>To: Jeff Brewster
>Cc: mifos-d...@lists.sourceforge.net
>Subject: RE: [Mifos-developer] Issue with changing center meeting day
>

>Hi Jeff,
>
>I was able to replicate this issue with a smaller data set also. The
>database has one branch data with 312 customers and 245 loan accounts.
>
>Here is the link to the video -
>http://rapidshare.com/files/299960064/mifos2.rar.html
>
>And yes this behavior occurs irrespective of what day I change the
>center meeting day.
>
>Regards,
>Prachi
>

Great, thank you Prachi. I've added these details to issue 2551 -
https://mifos.dev.java.net/issues/show_bug.cgi?id=2551.

[Chand...@sungard.com]

Hi

Provided a possible fix for this issue. Its working for all the datasets
I have tested with. I have updated the issue with my observations.
https://mifos.dev.java.net/issues/show_bug.cgi?id=2551.

"I have attached a patch that could be the possible fix for this. I have
verified
that it is working with Prachi's datasets in this thread

The modification was as follows
http://www.pastey.net/128929

It will be great if someone can verify this change.

The idea was that for the customers who were getting rescheduled the
start date
for the reschedule was the centre start date.. Overloaded the function
and
provided an explicit date ( next meeting day) to start the rescheduling
from

will be testing this further, because one instance was when the dates
where
rescheduled into a arbit point in the future. Need to see why that was
happening
( I haven't had the time to verify it with that dataset), but i am
pretty sure
this will fix that too. "

Are there any dependent processes that use this function (
regenerateMeetingSchedule in customerAccountBO.java?) Will need to
verify their functioning too.

regards
Chandan

-----Original Message-----
From: Jeff Brewster [mailto:jbre...@grameenfoundation.org]

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july

[Adam Monsen]

> Provided a possible fix for this issue. Its working for all the datasets
> I have tested with. I have updated the issue with my observations.

Prachi, thanks for the videos.

Chandan, Partha, thank you for updating issue 2551 with your notes, and
for attaching the patch to the issue.

We need help assuring that this code change actually fixes the issue at
hand. An automated test would be the most effective way to communicate
with us what was broken, prove this fixes it, and show expected
behavior. Can you write an automated test? This would most likely be an
integration or an acceptance test. Recall that integration tests extend
MifosIntegrationTestCase, and acceptance tests live in the
acceptanceTests module. If a unit test is possible, that would be even
better.

Also acceptable would be a full description of a manual test that could
be performed using the latest trunk code, starting with an empty
database.

A reminder that code freeze for the upcoming release (currently
codenamed Gazelle B) is scheduled for this Friday, the 20th.

[Chand...@sungard.com]

Hi Adam

I kind of worked on this today and tried to map the behavior for various
data sets.

Before I get to the details, there was something interesting I noticed
with the test testRegenerateFutureInstallments in the
customerAccountBOintegrationtest.java. The test is programmed in a way
that it sets up the meeting variables to make the getAllDates work
correctly. i.e it explicitly sets the meeting start date to the correct
value before generating the dates ( The test was working with my patch
applied, with no modification in the logic and a minor addition to the
code). My patch just does what this test does but what the regenerate
meeting schedule misses doing.

Anyway, getting back to the datasets. I used the oneBranchData.sql that
I Prachi attached in the earlier replies on this thread.

I created one member who had the following schedule.

mysql> select action_date from customer_schedule where customer_id=319;
+-------------+
| action_date |
+-------------+
| 2009-11-19 |
| 2009-11-26 |
| 2009-12-03 |
| 2009-12-10 |
| 2009-12-17 |
| 2009-12-24 |
| 2009-12-31 |
| 2010-01-07 |
| 2010-01-14 |
| 2010-01-21 |
+-------------+
10 rows in set (0.00 sec)

After the reschedule, you will notice it is currently slated to be on
Thursdays, I moved it to Mondays.

mysql> select action_date from customer_schedule where customer_id=319;
+-------------+
| action_date |
+-------------+
| 2009-11-19 |
| 2009-11-30 |
| 2009-12-07 |
| 2009-12-14 |
| 2009-12-21 |
| 2009-12-28 |
| 2010-01-04 |
| 2010-01-11 |
| 2010-01-18 |
| 2010-01-25 |
+-------------+
10 rows in set (0.00 sec)

The meetings are now being held on Mondays. It has skipped one week but
it sounds kind of logical because

The dates are 23 24 25 26 27 28 29 30, the meeting that was supposed to
be held on 26th (Thursday) got moved to 30th(next Monday). There was no
meeting held on the week of 22nd-28th. ( I guess this was done to
incorporate the case where he does the change after 23rd but before the
next meeting date(i.e 26th)?)

Anyways this is fine. After this I changed my system date to 1st of
December ran the batch job again, but after rescheduling the day back to
Thursday.. This was the output

mysql> select action_date from customer_schedule where customer_id=319;
+-------------+
| action_date |
+-------------+
| 2009-11-19 |
| 2009-11-30 |
| 2009-12-17 | { This date got changed from 7th to 17th, ie the
Thursday after next Monday}
| 2009-12-24 |
| 2009-12-31 |
| 2010-01-07 |
| 2010-01-14 |
| 2010-01-21 |
| 2010-01-28 |
| 2010-02-04 |
+-------------+
10 rows in set (0.00 sec)

This looks similar to the working of the previous scenario (The one week
skip). Is this the way mifos was programmed to work when the reschedules
happen?

Also there was another problem that Prachi encountered yesterday. This
is again tied to the above problem. Adhikar has a loan product that has
4 weekly installments. Now, when the reschedule was done all the
installments became weekly with bad start dates ( In the past)

This fix kind of fixes this problem too, but there is one issue that is
to do with already existing logic for loan rescheduling. Anyway this was
the case.. I created a loan product with similar characteristics and
created a loan account for a customer. This was the intial schedule that
was created.

mysql> select action_date from loan_schedule where account_id=565;
+-------------+
| action_date |
+-------------+
| 2009-12-21 | The loan was disbursed on 23rd, 21st of december is 28
days later . The meetings were on mondays
| 2010-01-18 |
| 2010-02-15 |
| 2010-03-15 |
| 2010-04-12 |
| 2010-05-10 |
| 2010-06-07 |
| 2010-07-05 |
| 2010-08-02 |
| 2010-08-30 |
| 2010-09-27 |
| 2010-10-25 |
| 2010-11-22 |
| 2010-12-20 |
| 2011-01-17 |
| 2011-02-14 |
| 2011-03-14 |
| 2011-04-11 |
| 2011-05-09 |
| 2011-06-06 |
| 2011-07-04 |
| 2011-08-01 |
| 2011-08-29 |
| 2011-09-26 |
| 2011-10-24 |
+-------------+
25 rows in set (0.00 sec)

I changed the meeting days to Thursday and ran the batch job again. This
was the output.

mysql> select action_date from loan_schedule where account_id=565;
+-------------+
| action_date |
+-------------+
| 2009-11-26 |
| 2009-12-24 |
| 2010-01-21 |
| 2010-02-18 |
| 2010-03-18 |
| 2010-04-15 |
| 2010-05-13 |
| 2010-06-10 |
| 2010-07-08 |
| 2010-08-05 |
| 2010-09-02 |
| 2010-09-30 |
| 2010-10-28 |
| 2010-11-25 |
| 2010-12-23 |
| 2011-01-20 |
| 2011-02-17 |
| 2011-03-17 |
| 2011-04-14 |
| 2011-05-12 |
| 2011-06-09 |
| 2011-07-07 |
| 2011-08-04 |
| 2011-09-01 |
| 2011-09-29 |
+-------------+
25 rows in set (0.00 sec)

The first repayment date is on the first Thursday after the disbursement
:( . There is some problem with the regenerate repayment schedule for
this type of loans. I think it tries to handle the first installment
with the same logic as a weekly loan and tries to find the first meeting
day in the week ( After all it is a loan that is classified as weekly)

Anyways this schedule can always be fixed with the redo loan feature. I
think if you are satisfied with the solution for the first problem you
can include this as a part of this release?

I haven't written any extra integration tests. Added one line in the
current one.

Hope the patch is all right.

Regards
Chandan

-----Original Message-----
From: Adam Monsen [mailto:amo...@grameenfoundation.org]
Sent: Thursday, November 19, 2009 6:15 AM
To: Mifos Developer Discussions
Subject: Re: [Mifos-developer] Issue with changing center meeting day

[Adam Monsen]

Thanks for the additional details. Unfortunately, Jeff and I were still
unable to reproduce the issue using Prachi's dataset.

> Anyway, getting back to the datasets. I used the oneBranchData.sql that
> I Prachi attached in the earlier replies on this thread.

Ah, great, thanks for clarifying.

> Is this the way mifos was programmed to work when the reschedules
> happen?

I don't know, but I doubt it because we cannot reproduce the issue.

> Anyways this schedule can always be fixed with the redo loan feature. I
> think if you are satisfied with the solution for the first problem you
> can include this as a part of this release?

We still need an automated test written against the trunk code: one that
fails without your fix, and succeeds with. Since issue 2551 manifests,
but no tests fail, we assume there is no current test coverage. If this
functionality is important, it should have test coverage.

We've seen spurious results when moving the system clock. Jumping the
system clock ahead past existing meetings can cause skips in the
schedule. A test could be written using, for example, DateTimeService,
but I'm not sure if one exists.

The fact that issue 2551 reproduces so easily for you and not for us
makes me think this could be some kind of timezone issue? Or maybe how
we store/manipulate dates?

[Adam Feuer]

Chandan,

If you want to get this fix into Release B, you need to provide a new
automated test that demonstrates the bug - that is, the test should
fail without your patch, demonstrating the bug. The test should pass
when your patch is installed.

Code complete is November 20 - that's tomorrow. We'll need the test
and patch by the end of the day to get it into Release B.

-adam

> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
>

--
Adam Feuer <adamf at pobox dot com>

[Chand...@sungard.com]

Hi Adam,

I have provided the tests and fixes the way it was required.

There are three files attached,

One is the complete fix, ( I took the svn diff of the revisions to get the complete patch)

"If you want to get this fix into Release B, you need to provide a new
automated test that demonstrates the bug - that is, the test should
fail without your patch, demonstrating the bug. The test should pass
when your patch is installed."

I couldn't do the split because I had already committed the code. Anyways I have provided two patches one that creates the test which will fail and then another which will fix this failing test.

[Adam Monsen]

> I have provided the tests and fixes the way it was required.

Thanks Chandan!

I don't understand your changes to the test class; I'll need to spend
some more time to get familiar with this code to finish reviewing the
patch.

I won't finish this before code freeze (today), and there isn't another
trunk maintainer available right now, so branches/sungard r16238 and
r16240 won't be merged into the trunk in time for Mifos 1.4.0 (formerly
Gazelle B).

Also, explicit repro steps would really help us verify the fix.

https://mifos.dev.java.net/issues/show_bug.cgi?id=2551

[Chand...@sungard.com]

Hi!

I have prepared a small document regarding how I arrived at the fix. It
would be great if someone can verify the arguments made in this.

Regards
Chandan

-----Original Message-----
From: Adam Monsen [mailto:amo...@grameenfoundation.org]
Sent: Saturday, November 21, 2009 5:27 AM
To: Mifos Developer Discussions
Subject: Re: [Mifos-developer] Issue with changing center meeting day

-issue 2551