The data insert approach would be to
1. Create atomic stored procedures to insert specific entities(eg branch, customer)
2. Create a master procedure(mifos_collectionsubmit_data_insert procedure in GF-CreateData.sql) which will call the atomic procedures in a loop
3. Audit the data which you are inserting
4. Execute the load test
5. Delete the inserted and newly created data using the audit tables(mifos_collectionsubmit_delete procedure in GF-Rollback_Script.sql)
We have unit tested the scripts and currently we are in the process of functionally validating them. Your suggestions/comments for improving our methodology are welcome.
Thanks
Aravind
Aravind Deivendran * Project Lead * SunGard * Technology Services * 6th Floor, Embassy Icon, #3, Infantry Road, Bangalore, India
Tel +91 80 30913200 3144 * Mobile +919980962300 * www.sungard.com/sts
-----Original Message-----
From: Adam Feuer [mailto:ad...@pobox.com]
Sent: Saturday, August 29, 2009 2:19 AM
To: Mifos software development
Subject: Re: [Mifos-developer] Colection Entry Submit - Affected Tables
On Fri, Aug 28, 2009 at 7:54 AM, <Aravind.D...@sungard.com> wrote:
> I am working on getting the tables which will be affected because of
> Collection Entry Submit to create rollback scripts. Currently I am doing
> this by going through the code and identifying the tables from hibernate
> xmls. Is there any other way I could achieve this?
Aravind,
Both Krishan's way and Udai's way should work. I think Krishnan's way
is easier, as it doesn't involve programming. I think looking through
the code manually will be error prone, because you may miss certain
tables.
-adam
--
Adam Feuer <adamf at pobox dot com>
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
Hi Aravind,
Wow! This is really very nice work.
Just one thing in the email set some alarm bells ringing for me...the mention of 'stored procedures', even if for testing purposes only.
Perhaps the dev team can respond: what is the general Mifos convention with regard to the use (or avoidance) of stored procedures?
To my knowledge, Mifos to date, does not use stored procedures or any other form of database 'program units' such as triggers, functions, etc.
In general, the use of stored procedures, etc. is discouraged in applications that hope to be 'database-agnostic'. Though Mifos is far not database-agnostic today, i guess this is probably a (distant ?) goal. Certain more portable alternatives (such as Java and .NET stored procedures are now available), but their popularity is limited
Regards,
|
-----Inline Attachment Follows----- |
|
neat bit of stored procedure coding.
Keith and myself are working quite a lot on the current getting and
saving of collection sheet data.
From reading your email, it seems that you are attempting to have a
way of generating a lot of test data to drive performance
testing. Is that right? If so, although I'm not fully across
exactly what you are doing, if you have further info we may be able
to help in some way.
I looked through the main stored procedure and noticed that certain
entities like schedules aren't there yet. That might be work in
progress or it might be that you are doing something completely
different to what I'm assuming.
John W
At 08:29 PM 7/09/2009, you wrote:
>Hi,
>We are able to prepare the insert and delete scripts for tables
>required for Collection Entry Submit testing. I have attached the
>scripts here with.
>
>The data insert approach would be to
>
>1. Create atomic stored procedures to insert specific entities(eg
>branch, customer)
>2. Create a master
>procedure(mifos_collectionsubmit_data_insert procedure in
>GF-CreateData.sql) which will call the atomic procedures in a loop
>3. Audit the data which you are inserting
>4. Execute the load test
>5. Delete the inserted and newly created data using the audit
>tables(mifos_collectionsubmit_delete procedure in GF-Rollback_Script.sql)
>
>We have unit tested the scripts and currently we are in the process
>of functionally validating them. Your suggestions/comments for
>improving our methodology are welcome.
>
>Thanks
>Aravind
Since the volume of data is huge, I thought using stored procedures would be good approach. Hence I went ahead and started developing procedures which can create entities(like accounts, customers). I have also created a master proc(procedure) which calls the atomic procedures and which is flexible and customizable to meet any requirement in the future. Currently we are doing functional validation of the procs we have created, to identify any entities which we have missed to create. As you mentioned, we have also identified that customer and loan schedules are to be created. Do you see any improvements in the way we are proceeding?
Hi Krishnan,
I accept the point that using stored procedures will make the application tied to a particular database. I thought stored proc is the best way for this requirement because
1. The development and customization is very quick.
2. The data loading(incase we want to insert thousands of records) will be much faster. This will save the time we spent for our testing.
3. Always performance testing will be done on a specific hardware and software. Hence we can use software specific code to insert test data. In case we need to use other database(Eg Oracle or SQL Server) for performance testing, we need little effort to port them to the current database.
Any comments/suggestions on my view?
Thanks
Aravind
Aravind Deivendran * Project Lead * SunGard * Technology Services * 6th Floor, Embassy Icon, #3, Infantry Road, Bangalore, India
Tel +91 80 30913200 3144 * Mobile +919980962300 * www.sungard.com/sts
-----Original Message-----
From: John Woodlock [mailto:jo...@nassles.com]
Sent: Monday, September 07, 2009 8:28 PM
To: Mifos software development
I agree, we don't want database-specific stored procedures in Mifos.
However, the Mifos application and test code currently requires a MySQL
backend; I think it would be difficult to make it work with other
databases. Given that we're currently coupled with MySQL without a need
to support other databases, I guess I don't have a problem with using
stored procs in this performance test.
Maybe I'm missing something, but the thing I don't like about the stored
procedures is that they will inevitably do something differently than
Mifos, and cease to be a useful test.
Aravind, there is Java code for creating a branch, customer, group, etc.
Maybe you could create a small application that calls Mifos methods
directly, then you can rely on the persistence logic already present in
Mifos. Well, except for your step 5 earlier ("deleting the newly
inserted and created data"). But for deleting, I would instead recommend
doing something at the operating system level like filesystem snapshots
(eg LVM on Linux, Shadow Copy on Windows). You could do a bunch of
inserts onto a database stored on a snapshot, then blow away the
snapshot for the next test. This would be repeatable, extremely fast,
and allow you to avoid figuring out what's been inserted into what
tables.
Great points, Adam.
While using an application that calls a business layer or some sort of
API eventually is ideal, Aravind and I (and others) agreed the use of
stored procedure as an initial approach is worth trying given Aravind
expertise with this process and our eagerness to get quicker feedback on
these Collection Sheet Entry performance results. Aravind is validating
whether these stored procedures are doing the job from a functional
standpoint and intends to get back to us with results in a day or two (I
know he's out on Wednesday).
Jeff
________________________________
John, Aravind,
John -- your script is a great example to give an idea of how things could be done using business objects.
There are a few things to keep in mind:
TestObjectFactory gives an idea of how related business objects can be constructed but any uses of it should be carefully examined for correctness. In the past we have discovered objects created by it that were not in consistent states. As we have found these issues, we have fixed them, but as a result, I am wary of TestObjectFactory.
Unfortunately this means that we likely have tests that use these objects and pass either in spite of inconsistent state or which may mask bugs because of that state. Because of this I would suggest getting inspiration from TestObjectFactory but writing separate methods for object construction. The other thing to do is improve the correctness of TestObjectFactory—watch out when going down that path for the number of tests that fail if changes are made to TestObjectFactory and the time it will take to fix them. In the past some fixes to TestObjectFactory have led to lengthy work to repair the affected tests.
We are hoping that we can reduce the number of tests that depend on TestObjectFactory by gradually moving to an in-memory mocking approach and away from the integration tests that currently depend on TestObjectFactory.
What we really want is a set of well defined APIs for creating business objects of all kinds in Mifos. Creating APIs is on our architectural road map, but we’re not there yet.
As John mentions, the time that it takes to create objects in this way may currently be too long for practical use for data loading (depending on how much data needs to be loaded how often). We should be able to make it faster to do this with some work, but there is a question of how long it would take to optimize this versus the timeline for getting collection sheet entry in place.
Aravind -- let’s keep the discussion going and track how work is proceeding for you.
--Van
From: John Woodlock [mailto:jo...@nassles.com]
Sent: Wednesday, September 23, 2009 1:05 AM
To: Mifos software development
I just thought i would jump in on this conversation and respond to
some of your points:
> TestObjectFactory gives an idea of how related business objects can be
> constructed but any uses of it should be carefully examined for
> correctness.....
I agree that the TestObjectFactory is a bit suspect. But this is more
due to the way it persists the model objects that is builds up.
> Because of this I would suggest getting inspiration
> from TestObjectFactory but writing separate methods for object
> construction...
> We are hoping that we can reduce the number of tests that depend on
> TestObjectFactory by gradually moving to an in-memory mocking approach and
> away from the integration tests that currently depend on TestObjectFactory.
So yes, there is a cleaner way of supporting tests of all types unit,
integration and acceptance. And its true to say that currently there
are far too many integration tests. A lot of them can be converted to
unit tests through the use of Test Doubles (stubs, mocks whichever is
appropriate) but there will still be integration tests (around
dao/persistence classes) and so we will still have the problem of how
to set up data in the database in a way that supports change and helps
keep tests readable/understandable.
So talking about it from point of view for each test
Unit Tests:
In unit tests we use mocks/stub to isolate the tests but it is also
quiet common that we need to construct some of the domain model
objects that are reference in the class under test. Quiet often can be
quiet complex to setup. The best approach i have used and seen is to
used a 'builder' technique to simplfy object creation for tests and
keep tests more 'intention driven'. I first came across this technique
when reading Nat Pryce/Steve Freeman and they currently are writing a
book around TDD and OO which has a chapter devoted to this technique
with description of why and how this is done. see
http://www.mockobjects.com/book/test-data-builder.html (note: the book
itself is worth a read, most of it still online)
I have also noticed in lots of the model objects that it is typical
for them to have serveral constructors for creation of the object
which is problematic and this approach would stop test-only
constructors from leaking into model.
Integration-tests (that interact with database):
These make up our largest group of tests which hopefully will be
reduced over time. The biggest difficulty with integration tests is
getting sufficient data into the database in order to test a specific
behaviour of the system. Currently we are doing this through the
TestObjectFactory. The TestObjectFactory follows an 'Object Mother'
pattern. As stated in
(http://www.mockobjects.com/book/test-data-builder.html) there are
pros and cons to this approach and I think the use of the 'builder'
technique to costruct our POJO model objects and then use of
DAO/Persistence to save these objects to the database (keeping how we
build our model seperate from how they are persisted) is a cleaner
approach.
Another approach to getting the data into the database is to use
DBUnit like we do for the acceptance tests. The problem with this
approach is that the data to be put in is represented through XML and
this does not lend itself to making schema changes easy. (which almost
always means that those changes won't happen...) It also not that easy
to understand what data is relevant to tihs test especially when this
data is accross lots of database tables.
Every time we change a column name or remove a column/table the xml
would have to be changed. If this approach is used across thousands of
tests, it creates a huge cost. But using the builder/model/ORM
approach as used in the application code reduces the ripple effect
reducing the cost of refactoring changes on the schema/model.
So what does this mean for performance tests. The same problem exists.
Its more risky to try to duplicate all the business logic encapsulated
in the model within 'stored procedures' for the sake of setting up
data. Any changes in development would have to be placed in two places
(model and stored procedures). Whatever tests are done for performance
are something we should do all the time through development to catch
problem queries etc. Leveraging the already existent POJO model along
with DAO/Persistence mechanism is easiest way to get data into
database but will be a bit slower than stored procedures.
KeithW
I checked out this technique in the book. Very cool! I recall this
sort
of readability from some Smalltalk code I've read.
> Integration-tests (that interact with database):
Ugh, yes, our 'Object Mother' (TestObjectFactory) has definitely
turned
out to be a maintenance nightmare.
I think you included acceptance tests (black-box tests that hit the
Web
interface) in this category. The myriad DBUnit datasets do indeed
complicate modifications to the database schema.
> > TestObjectFactory gives an idea of how related business objects can
> be
> > constructed but any uses of it should be carefully examined for
> > correctness.....
>
> I agree that the TestObjectFactory is a bit suspect. But this is more
> due to the way it persists the model objects that is builds up.
Let me give an example of the kind of problem I'm thinking of here:
I did a couple of quick searches and found the test
GroupBOIntegrationTest.testUpdateCenterFailure_MeetingFrequencyMismatch(
). In this test there is a call to createCenter which defers to
TestObjectFactory.createCenter. A monthly meeting is passed in for the
creation of the center. Drilling down into the
TestObjectFactory.createCenter method that is called (line 280 for me),
a set of default fees is generated using the TestObjectFactory.getFees()
method. But getFees() generates fees that are weekly fees, so now we
have a center with monthly meetings that are mismatched to weekly fees
(where the fees should be monthly). As long as the test doesn't do
anything with fees, all is well, but if it does-- watch out!
I believe that there are other cases like this of mismatched data of one
kind or another.
It would be good to eliminate/replace methods that allow the creation of
objects with mismatched data like this (up to this point there have
always been higher priority things to do). Ultimately we should have
API methods which would cover this.
I would be interested to hear more about the issues you have seen with
persistence.
> > We are hoping that we can reduce the number of tests that depend on
> > TestObjectFactory by gradually moving to an in-memory mocking
> approach and
> > away from the integration tests that currently depend on
> TestObjectFactory.
>
> So yes, there is a cleaner way of supporting tests of all types unit,
> integration and acceptance. And its true to say that currently there
> are far too many integration tests. A lot of them can be converted to
> unit tests through the use of Test Doubles (stubs, mocks whichever is
> appropriate) but there will still be integration tests (around
> dao/persistence classes) and so we will still have the problem of how
> to set up data in the database in a way that supports change and helps
> keep tests readable/understandable.
Agreed.
> In unit tests we use mocks/stub to isolate the tests but it is also
> quiet common that we need to construct some of the domain model
> objects that are reference in the class under test.
> The best approach i have used and seen is to
> used a 'builder' technique to simplfy object creation for tests and
> keep tests more 'intention driven'. see
> http://www.mockobjects.com/book/test-data-builder.html (note: the book
> itself is worth a read, most of it still online)
Excellent article! I would like to see us try using the 'builder'
technique in our tests.
> I think the use of the 'builder'
> technique to costruct our POJO model objects and then use of
> DAO/Persistence to save these objects to the database (keeping how we
> build our model seperate from how they are persisted) is a cleaner
> approach.
Yes, this is the key thing in my mind-- we should be able to construct
objects as "plain old java objects" (aka POJOs) using something like a
builder. These could then either be used as POJOs in in-memory unit
tests or persisted for use in integration tests.
> Another approach to getting the data into the database is to use
> DBUnit like we do for the acceptance tests. The problem with this
> approach is that the data to be put in is represented through XML and
> this does not lend itself to making schema changes easy. (which almost
> always means that those changes won't happen...) It also not that easy
> to understand what data is relevant to tihs test especially when this
> data is accross lots of database tables.
Your point is well taken (and is timely since I am facing a schema
change right at this moment and the acceptance test data sets need to be
updated!) DBUnit usage was introduced since we knew how we could get
and manage consistent data sets without taking the upfront hit of
putting in place a framework for creating objects and then optimizing
it. We did go in with eyes open knowing that maintenance is where you
pay the price.
> Every time we change a column name or remove a column/table the xml
> would have to be changed. If this approach is used across thousands of
> tests, it creates a huge cost. But using the builder/model/ORM
> approach as used in the application code reduces the ripple effect
> reducing the cost of refactoring changes on the schema/model.
I agree that the builder/model approach eliminates the problems of data
set maintenance and it might well be worth trying if we have the
resources to do it. The main hurdles I see are:
* how much effort would it take to build the infrastructure we need to
create the data for testing? An incremental approach could be a good
way to answer this-- basically, try building a small example of this and
then add more to it, bit by bit and note the time and effort required
for it.
* can enough data be created fast enough. For some of the acceptance
tests, there is enough data required that it raises the question of how
long would it take to create it on the fly each time as opposed to
loading it using something like dbunit? There is much room for
optimization in Mifos, so even if some things take a while to create
now, they can probably be made faster.
> So what does this mean for performance tests. The same problem exists.
> Its more risky to try to duplicate all the business logic encapsulated
> in the model within 'stored procedures' for the sake of setting up
> data. Any changes in development would have to be placed in two places
> (model and stored procedures). Whatever tests are done for performance
> are something we should do all the time through development to catch
> problem queries etc. Leveraging the already existent POJO model along
> with DAO/Persistence mechanism is easiest way to get data into
> database but will be a bit slower than stored procedures.
Agreed. I have the same reservations about stored procedures and the
implications for maintaining performance tests over time. Over the long
term it will certainly slow development and be a source of errors and
inconsistency to have business logic duplicated in multiple places.
So I'm in complete agreement that our goal should be to have data
created using business objects and I really like the 'builder' approach
to that.
The practical question that comes into play here is what can be done in
the timeframe we have for the next release (Gazelle B) given the
resources we have.
We're targeting early November for wrapping up coding on Gazelle B. To
support the changes in collection sheet entry that have been especially
made to support Grameen Koota, we need to have performance testing done
for collection sheet entry soon.
So the questions that come to mind are:
* could we implement enough business object based database population
code fast enough for Sungard to use it for collection sheet entry
testing?
* would business object based database population code be run fast
enough to allow the testing to be done? (or could we make it run fast
enough?)
* because of their speed, are stored procedures a reasonable short term
solution even though we might not want to use them in the long run?
* is there another approach (e.g. database/filesystem snapshots) that
could be used.
It is worth noting that some of the performance testing needs to
validated on very large data sets, so most likely business object based
data creation would need to be on top of an existing real or synthetic
data set.
Enough rambling :-)
So one question is if there is an action item that should come out of
this to do a spike to create some 'builder' based data creation for
Mifos-- thoughts?
--Van
One other consideration we have with acceptance testing is that some of
our testing requires date/time sensitive data. A builder approach may
become more complex when you have to consider how to reflect the passage
of time in your test data. I think for acceptance testing we should
consider a "hybrid" of both stored XML datasets that contain the basic
test data needed for the foundation of most tests and use a builder
approach to create the objects relevant to the specific test situation.
> So the questions that come to mind are:
> * could we implement enough business object based database population
> code fast enough for Sungard to use it for collection sheet entry
> testing?
I'm curious if we have any way of doing a crude estimate of this effort.
In my opinion it'd have to be a *very* small amount of work to be timely
for Gazelle B.
> * because of their speed, are stored procedures a reasonable short
term
> solution even though we might not want to use them in the long run?
> * is there another approach (e.g. database/filesystem snapshots) that
> could be used.
Good question. I agree we should brainstorm on some additional options
pending SunGards progress with stored procedures this week. One idea
might be to take SunGard's current approach for removing data (build a
temporary table of id's to remove later) but instead execute performance
tests on customer data instead of script-generated data.
> So one question is if there is an action item that should come out of
> this to do a spike to create some 'builder' based data creation for
> Mifos-- thoughts?
Yes, I believe this would be very useful. We have other requirements
for test data that could also benefit from test data builder. For
example, demo data, data to test various reports, and stress test data.
I discussed the pros and cons of using the stored procedure approach with Adam(in person when he was here) and Jeff. Since we are almost through writing stored procs, I thought it would be good to use the procs in short term for the current load test and create the API based inserts after the test. Hope this is fine.
I went ahead and created the insert and delete scripts and did the following tests
1. Tried to insert data and do a Collection Submit Entry through the application. We were able to do. Hence the scripts are functionally correct.
2. Tried inserting one million records through the scripts. The script took around an hour and successfully inserted one million records.
3. Using the updated LoadRunner scripts, tried test using a single user. The test was successful.
I have shows the demo of stored procedures to Adam when he was here. We are going ahead with the Load Testing. The approach for the same is shared with this mail(and the latest scripts too). Your comments and suggestions welcome.
the performance test approach looks good.
Stored procs look good too (from a scan). I'll have to give it a run
a bit later. Couple of points that might not make much difference are:
a) There's no fee_schedules for customers or loans based on any
period fees but that's probably okay... maybe GK don't even use them.
b) A saving_schedule isn't being created for savings_accounts but
again GK doesn't use savings_account much either.
c) Also, AFAICS, the stored procs just create one schedule entry
where the application creates many (10 for customer and savings
accounts and X for loan_accounts depending on how the loan product is set up).
John W
So for the performance test to be truly reflective there would also
need to be data inserted into all the one-to-one related tables of
each of the core concepts (customer, loan_account, savings_account
etc)
________________________________
From: Keith Woodlock [mailto:keithw...@gmail.com]
Sent: Mon 9/14/2009 1:46 PM
To: Mifos software development
Subject: Re: [Mifos-developer] Aravind Collection Entry Submit -loaddatapatch
Muthoni
> -----------------------------------------
> This message has been scanned for viruses and dangerous content by
> MailScanner,
> and is believed to be clean.
>
> +++++++++++++++++++++++++++++++++++++++
> UNIVERSITY OF NAIROBI IS ISO CERTIFIED
>
> The University of Nairobi is committed to providing quality services to all
> its clients. The University will monitor and review its quality performance
> from time to time through an effective implementation of the Quality
> Management System based on ISO 9001:2008 standard.
>
> University of Nairobi Website: http://www.uonbi.ac.ke/
> +++++++++++++++++++++++++++++++++++++++
>
-----------------------------------------
This message has been scanned for viruses and dangerous content by MailScanner,
and is believed to be clean.
+++++++++++++++++++++++++++++++++++++++
UNIVERSITY OF NAIROBI IS ISO CERTIFIED
The University of Nairobi is committed to providing quality services to all
its clients. The University will monitor and review its quality performance
from time to time through an effective implementation of the Quality
Management System based on ISO 9001:2008 standard.
University of Nairobi Website: http://www.uonbi.ac.ke/
+++++++++++++++++++++++++++++++++++++++
------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf