[Mifos-developer] Data Migration Toolkit

[Chetan.B...@sungard.com]

Dear all,

 

SunGard would like to make available the first version of the Mifos Data Migration Toolkit. The toolkit can be used to migrate data from the Legacy system to Mifos.  The toolkit has been tested for the Mifos 2.1 version. The tool along with the instruction manual for using the tool can be accessed and downloaded from here

 

The following functions are supported by the migration toolkit in the current release.

1.       Migration of Centers, Groups, clients entity wise

a.      Client Family details

2.       Migration of Loans – both individual client and group loans

a.      Support for Flat Rate of interest type

b.      Support for Declining Balance interest rate type

3.       Migration of Individual clients i.e. clients not associated with any groups or Center

4.       Migration of groups independent of centers

5.       Migration of Additional fields – Questions groups

6.       Validations for

a.      Mandatory field in MIFOS

b.      Duplicate records check

c.       Range checks on the loan product offerings

d.      Hierarchy check to validate the loan officer -> Center -> Group->Client link

e.       Loan Product Applicability : Products applicable to groups should have loans created for the groups only and not clients

f.        Loan creation with product frequency Mismatch : Validation such that the recurrence for the loan product matches the customer/center schedule

7.       Date Validations :

a.      Loan creation date should be greater than center creation date ( or MFI joining date)

b.      Group  MFI joining date should be greater than center creation date

c.       Disbursement date in loan account cannot be less than loan creation date

d.      Should not allow future dated values for MFI joining date, loan creation date etc

8.       Migrating One Time fees

9.       Migrating Group loans

 

We would be happy help any of the Mifos users/specialist if they need assistance in using the tool. Also any feedback on the tool is welcome. We will work on the next version of this tool depending on how the Mifos product itself evolves and what role SunGard as an organization can play in the next phase of Mifos product.

 

A special thanks to Satiya and Ramya from the SunGard team for developing the key features of the toolkit.

 

Thanks

 

 

[Emily Tucker]

Congratulations Chetan and the entire SunGard team on shipping this tool.  Data migration continues to be a significant pain point for MFIs moving on to Mifos.

 

I look forward to hearing feedback from the community on this tool.

 

Cheers,

Emily

------------------------------------------------------------------------------
Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
user administration capabilities and model configuration. Take
the hassle out of deploying and managing Subversion and the
tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
_______________________________________________
Mifos-users mailing list
Mifos...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mifos-users

[Jakub Sławiński]

Is it possible to promote this tool directly on the mifos.org website?

Regards,
Jakub.

On 20.08.2011 00:12, Emily Tucker wrote:
> Congratulations Chetan and the entire SunGard team on shipping this tool.
> Data migration continues to be a significant pain point for MFIs moving on
> to Mifos.
>
> I look forward to hearing feedback from the community on this tool.
>
> Cheers,
> Emily
>
> On Fri, Aug 19, 2011 at 2:29 AM, <Chetan.B...@sungard.com> wrote:
>

>> Dear all, ****
>>
>> ** **

>>
>> SunGard would like to make available the first version of the Mifos Data
>> Migration Toolkit. The toolkit can be used to migrate data from the Legacy
>> system to Mifos. The toolkit has been tested for the Mifos 2.1 version. The
>> tool along with the instruction manual for using the tool can be accessed

>> and downloaded from here<https://docs.google.com/viewer?a=v&pid=explorer&chrome=true&srcid=0B7TIEjUszn9bZTI0ODAwMzYtNDQ3YS00MWZmLTlmZTMtYjEzNjAwNjI2NWM1&hl=en_GB>
>> ****
>>
>> ** **
>>
>> *The following functions are supported by the migration toolkit in the
>> current release.*
>>
>> ***1. ****Migration of Centers, Groups, clients entity wise*
>>
>> ***a. ****Client Family details *
>>
>> ***2. ****Migration of Loans – both individual client and group
>> loans *
>>
>> ***a. ****Support for Flat Rate of interest type*
>>
>> ***b. ****Support for Declining Balance interest rate type*
>>
>> ***3. ****Migration of Individual clients i.e. clients not
>> associated with any groups or Center *
>>
>> ***4. ****Migration of groups independent of centers*
>>
>> ***5. ****Migration of Additional fields – Questions groups *
>>
>> ***6. ****Validations for *
>>
>> ***a. ****Mandatory field in MIFOS *
>>
>> ***b. ****Duplicate records check *
>>
>> ***c. ****Range checks on the loan product offerings *
>>
>> ***d. ****Hierarchy check to validate the loan officer -> Center ->
>> Group->Client link*
>>
>> ***e. ****Loan Product Applicability : Products applicable to groups
>> should have loans created for the groups only and not clients*
>>
>> ***f. ****Loan creation with product frequency Mismatch :

>> Validation such that the recurrence for the loan product matches the

>> customer/center schedule*
>>
>> ***7. ****Date Validations : *
>>
>> ***a. ****Loan creation date should be greater than center creation
>> date ( or MFI joining date) *
>>
>> ***b. ****Group MFI joining date should be greater than center
>> creation date *
>>
>> ***c. ****Disbursement date in loan account cannot be less than loan
>> creation date *
>>
>> ***d. ****Should not allow future dated values for MFI joining date,
>> loan creation date etc *
>>
>> ***8. ****Migrating One Time fees*
>>
>> ***9. ****Migrating Group loans *
>>
>> ** **

>>
>> We would be happy help any of the Mifos users/specialist if they need
>> assistance in using the tool. Also any feedback on the tool is welcome. We
>> will work on the next version of this tool depending on how the Mifos
>> product itself evolves and what role SunGard as an organization can play in

>> the next phase of Mifos product.****
>>
>> ** **

>>
>> A special thanks to Satiya and Ramya from the SunGard team for developing

>> the key features of the toolkit.****
>>
>> ** **
>>
>> Thanks****
>>
>> ** **
>>
>> ** **

>>
>>
>> ------------------------------------------------------------------------------
>> Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
>> user administration capabilities and model configuration. Take
>> the hassle out of deploying and managing Subversion and the
>> tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>> _______________________________________________
>> Mifos-users mailing list
>> Mifos...@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mifos-users
>>
>>
>
>
>
> ------------------------------------------------------------------------------
> Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
> user administration capabilities and model configuration. Take
> the hassle out of deploying and managing Subversion and the
> tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>
>
>

> Mifos-developer mailing list
> mifos-d...@lists.sourceforge.net
> Unsubscribe or change settings at:
> https://lists.sourceforge.net/lists/listinfo/mifos-developer

------------------------------------------------------------------------------
Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
user administration capabilities and model configuration. Take
the hassle out of deploying and managing Subversion and the
tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2

Mifos-developer mailing list
mifos-d...@lists.sourceforge.net
Unsubscribe or change settings at:
https://lists.sourceforge.net/lists/listinfo/mifos-developer

[Ed Cable]

Jakub,

I agree that this is something we want to highlight and showcase on mifos.org. 

Chetan B, I can help to create a section under downloads for the toolkit. It would be nice to do a brief write-up on the toolkit in the newslog - do you want to draft up more of an overview along with some screenshots or a screencast to show it in action.

Best,

Ed

2011/8/20 Jakub Sławiński <jslaw...@soldevelo.com>

[Chetan.B...@sungard.com]

Hello,

 

A couple questions which will enable us to help you.

 

1.       Which Version of Mifos are currently using ? Actually it should not matter version you use as long as your configured Mifos schema dump is the same as the one with which Stage1 scheme is created. Anyway we will double check this and find out if any new columns got added

2.       Will it be possible to share with the us your Configured Mifos schema and also the excel sheet which you are trying to migrate ?You may attach the same against the issue in JIRA

 

 

Thanks,

Chetan

 

 

From: Hugo Technologies [mailto:in...@hugotechnologies.com]
Sent: Friday, October 21, 2011 11:50 PM
To: mifos...@googlegroups.com
Cc: Prasath, Satiya; Viswanathan, Kavita; mifos-d...@lists.sourceforge.net; Rajagopalan, Seetha; A good place to start for users or folks new to Mifos.; Toshniwal, Ramya
Subject: Re: [Mifos-users] Data Migration Toolkit

 

Hi,

 

Data Migration is key phase of the project, which plays an important role for MIF's, volunteer or specialist who wants to migrate from a legacy system to Mifos and thanks to SunGard for provide this tool.

 

We have configured it and tested with Mifos 2.2 version for one of our associates and found few issues (may be related to the version of Mifos that we have used) which are been described in jira issues Mifos Data Migration Tool MIFOSDATAMIGRATION-206 -- Issue / Gap for Stage1 Database during Data Migration using Kettle

 

With regarding to the performance and memory related problem I think this should be more of an environmental settings or configuration related. Request you to please have a look and suggest us so that we can move ahead with Data Migration and Testing on Migrated Data .

Best Regards

Team HT

[Ed Cable]

Chetan,

I wanted to try and set up a meeting this upcoming week to discuss extending the data migration tool to support Mifos 2.2.x and to extend it's functionality beyond loans to savings, etc.

We discussed this on the weekly developer call this morning, following up on the issue, that Venkat from Hugo Technologies, originally posted: http://mifosforge.jira.com/browse/MIFOSDATAMIGRATION-206

They have a strong need for this currently along with FINA who is working with a potential customer in Laos.

As proposed by Binny, rather than continue one-off data migrations with customized scripts, we would like to provide one generic tool that the community can use time and time again.  We all agreed that the toolkit that SunGard has provided is the most configurable tool and the one we would like to continue to advance.

I will email you separately to try and schedule some time for this call.

Ed

------------------------------------------------------------------------------
The demand for IT networking professionals continues to grow, and the
demand for specialized networking skills is growing even more rapidly.
Take a complimentary Learning@Cisco Self-Assessment and learn
about Cisco certifications, training, and career opportunities.
http://p.sf.net/sfu/cisco-dev2dev

[George IBERIA]

Thanks Ed, Jakub and Chetan,
It will be very interesting, I will participate and promote as much as I can this unified migration tool which will be useful for all members, users and developers.
george

[George IBERIA]

Sorry for asking again (it was discussed on the DEV meeting but need some clarifications ) :
Do you think that using BI kettle is more useful , less risky and takes less time then to implement migration using Mifos import API ? Wit with the assumption that import API may need some modification / extension / improvement, etc.  

[satiya prasath]

Hi George,

No, its wasn't a question of whats more powerful/efficient, when we
started out on the tool, the smart way to implement(re-use existing
code) the data migration tool which would be easy to maintain would
have been using the API's. But at the time of development, there were
no API's to speak of.

So the tool was built so it could work without the API's, and
eventually be replaced by the API's when its usable. Meanwhile we
decided that it was easier to use pentaho as people would be using it
for other mifos needs and would be familiar with it.

> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure
> contains a definitive record of customers, application performance,
> security threats, fraudulent activity, and more. Splunk takes this
> data and makes sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-novd2d

[Chetan.B...@sungard.com]

George,

 

When we developed the Mifos Data migration toolkit, Mifos APIs were still under development ( and I believe are in still in development phase  for certain core functions like Create client, loan )). The current migration toolkit has been built out such that if the APIs are ready to use , we should be able to replace our custom code with API calls.

 

Thanks

cChetan

[George IBERIA]

Dear Satiya and  Chetan,

I understand you , but now we have that API .
Is it possible  to consolidate our efforts and concentrait it on one technology .

(I know it is not allways possible)

[Udai Gupta]

Kettle (PDI) is very useful for building small ETL definitions
quickly, building complex data migration tool for Mifos on top of
Kettle might be just too much of effort not just in learning curve but
also getting hit by it's performance limitations.

Same applies to the Mifos REST APIs (which I assume is being referred
as Import API in thread).

Data migration which is bulk/batch process needs very high level of
optimization which should be done using Mifos code (model). I favor
this approach because having a separate module using direct code can
- Avoid dependency on another layer to get fixed for bulk processing
- Create different optimization required for bulk processing
(caching, lazy loading, memory setting)
- Create separate/specialized Queries/Daos for bulk processing
- Reuse the existing code and business rules Mifos code (DRY)
- Separate Mifos Business rules (refactoring) in Mifos core so that
those can be used easily in migration tool. (not necessarily required
but it could help in making Mifos more cleaner implementation)

The data is available in various format when you preform migration so
we should support configurable basic data parsing/migration (SQL, CSV,
XLS, XML etc) and translation, also provide easy way to add new data
implementation for unknown cases (data types).

Udai

[George IBERIA]

Dear HUGO TECHNOLOGIES - who is the progress of the data migration toolkit. When it will be available for use? If you have a docs please publish on web or give us link. How much LOE is needed for data migration ? (please give us some idea about it depending on valium and complexity of the data have to be migrated. We will be happy to contract you for this.  Thanks.  George

[info]

Hi George

We initiated and collating the required details for porting. 

Raised a query to developers community, on receiving a response we will take-up the development work.

Thanks

shiva

On Sat, Dec 10, 2011 at 2:13 AM, George IBERIA <bank...@gmail.com> wrote:

Dear HUGO TECHNOLOGIES - who is the progress of the data migration toolkit. When it will be available for use? If you have a docs please publish on web or give us link. How much LOE is needed for data migration ? (please give us some idea about it depending on valium and complexity of the data have to be migrated. We will be happy to contract you for this.  Thanks.  George

--
Regard,

HT Team

www.hugotechnologies.com

Phone : +91 40-65141823 | Skype: hugo.technologies