[Mifos-developer] acceptance test improvements meeting

[Jeff Brewster]

Today we had a call to discuss improvements to the acceptance tests.

On the call we had: Udai, Michael V., Keith W., Jakub, Deepak P., and
myself.

We had a very good discussion! Thanks to all who took time out of their
evening to join. We focused on these problem statements:

1. Acceptance tests are slow to develop and maintain.
2. Data sets used for acceptance tests are expensive to maintain and
take up significant portion of execution time.

Observations and findings from group:

- Work towards testing pyramid with less UI based tests, more lower
level and API level tests.
- When an acceptance test fails, it's difficult to determine why tests
has failed.
- Page objects are good. The implementation of page objects could be
better.
- There are good reasons to go towards Selenium 2.0/ Webdriver -
-- an api that's easier to use,
-- has support packages,
-- gets rid of the required startup of selenium server since Webdriver
doesn't need a server
-- supports more browsers
- Michael V. has made some good progress already with Webdriver [1]. He
experimented with the Selenium Emulation for Webdriver [2], [3].
- Data sets have two usages right now - seed data and comparison of
actual/expected test results.
- Deepak said he's generating unique pre-test data via the UI instead of
creating new seed data sets.
- Many of the seed data sets are quite close in content, and have very
small differences. We need to consolidate the number of data sets used
for seed data.
- Move away from using data sets for verification of acceptance tests.
Instead use UI to validate test result.
- When reviewing Acceptance Criteria for a story, determine whether to
cover requirements as integration or at acceptance test level. Go
towards lower level tests where possible.
- developers can run tests faster using RAMDisk [4]

[1] - http://tinyurl.com/23azb8l
[2] - http://code.google.com/p/selenium/wiki/SeleniumEmulation
[3] -
http://github.com/vorburger/mifos-head/commit/5805f84b0f9169e261e3730862
346633b591a384#diff-3
[4] - http://mifosforge.jira.com/wiki/display/MIFOS/RAMDisk

Next steps:
Short term:
- Michael V. will try to run all acceptance tests with WebDriver. If it
works, we will move towards WebDriver on head master. Question:
Selenium 2.0 is still in beta. Do we wait for Selenium 2.0 to ship?
- Deepak and Vivek will look at reducing number of data sets required
for acceptance testing. Gather a list of tests that use data sets for
result validation and start changing those tests to validate via UI.
- Kojo working on testing coverage reporting for integration and
acceptance testing levels. Also working on providing more information
when test fails.

Longer term:
- Continue to add and expose APIs to allow functional testing and data
setup to be done via APIs.

------------------------------------------------------------------------------
Download new Adobe(R) Flash(R) Builder(TM) 4
The new Adobe(R) Flex(R) 4 and Flash(R) Builder(TM) 4 (formerly
Flex(R) Builder(TM)) enable the development of rich applications that run
across multiple browsers and platforms. Download your free trials today!
http://p.sf.net/sfu/adobe-dev2dev

[Vivek Singh]

This is great. I agree with all the next steps. I have some additional questions/comments.

[Context: When reviewing Acceptance Criteria for a story, determine whether to cover requirements as integration or at acceptance test level.  Go

towards lower level tests where possible.]

I agree and I would add that these tests can also be service level unit tests not going to database.

Now, when we go for lower level tests couple of things would happen.

- a non-developer would not be able to write these test on their own.

- because tester would not be directly aware of what scenarios are covered, it is likely that they would manually test the same scenarios, causing waste.

I think we should look at something like JBehave only for such tests. This provides a framework in which developer/BA/tester/customers can collaborate.

After we move to selenium-webdriver mode, we should configure our tests so that some build continues to run them with browser to verify correctness.

[Context: Deepak said he's generating unique pre-test data via the UI instead of creating new seed data sets.]

This has two issues:

a) It couples the test in question, with parts of application it is not testing.

b) Using user interface to setup data is slow.

I feel we should do this using an API which is piece we want to change, in order to run it faster and be decoupled. In our case this means we should use SQL.

If we all agree that we decide what next steps with respect to these can be.

--
Vivek Singh | +91 98452 32929 | http://sites.google.com/site/petmongrels

[Jeff Brewster]

See replies below...

>I think we should look at something like JBehave only for such tests. This provides a >framework in which developer/BA/tester/customers can collaborate.

Vivek, JBehave looks interesting. I looked at Twist over a year ago. Would you say they are similar? Can you point us to any projects that are using JBehave today to get an idea on how it is practically used?

>After we move to selenium-webdriver mode, we should configure our tests so that some build >continues to run them with browser to verify correctness.

Agreed.

>[Context: Deepak said he's generating unique pre-test data via the UI instead of creating new >seed data sets.]

I didn't make this clear in my notes, but we agreed this wasn't an optimal approach, but a tactic used to avoid creating extra data sets. I agree it increases complexity of each test and can make it slower.

>I feel we should do this using an API which is piece we want to change, in order to run it >faster and be decoupled. In our case this means we should use SQL.

I like us using an API when it exists. You mention using SQL - my concerns with SQL:
1. test writer may miss some business logic (enforced by Mifos UI) not validated at DB level
2. the SQL could break tests when ongoing db changes are made

Regards,
Jeff

[Vivek Singh]

>> I like us using an API when it exists.  You mention using SQL - my concerns with SQL:

1. test writer may miss some business logic (enforced by Mifos UI) not validated at DB level
2. the SQL could break tests when ongoing db changes are made

There are three levels using which data can be setup. App, Service-API and SQL. The higher we go we get easy correctness of data. But higher you go slower it gets. It also becomes dependent on the application. So for example:

If we setup data using Service-API: Test can fail when API didn't setup the data correctly because there is bug in the service logic. This would fail even tests which are not broken.

This can happen with SQL as well. But SQL to setup data is completely test artifact. If tests break then it would be because of wrong tests.

I feel there are few important reason because of which SQL is a better choice because Database and Test are much slower changing pieces anything in the source code. Application should not be used to setup test data to its own testing, from test purity perspective.

I understand there is steeper curve but is in general a performant and completely decoupled approach in long run.

[Jeff Brewster]

Vivek said:
>"If we setup data using Service-API: Test can fail when API didn't
setup the data correctly because there is bug in the service logic. This
would fail even tests which are not broken.
This can happen with SQL as well. But SQL to setup data is completely
test artifact. If tests break then it would be because of wrong tests.
I feel there are few important reason because of which SQL is a better
choice because Database and Test are much slower changing pieces
anything in the source code. Application should not be used to setup
test data to its own testing, from test purity perspective."

Vivek,
I understand what you are saying and agree that using the application to
set up data is not ideal, but I am very concerned about trying to write
proper SQL to create the entities needed for an acceptance test. I
think this raises the barrier to writing new tests and allows potential
for faulty or incomplete data to be inserted.

As a test writer, what I would like is a mark-up style file to define
the entities that my test requires. For example a YAML
(http://en.wikipedia.org/wiki/YAML) file that defines the data I need to
create for my test - e.g.

Loanproduct(flatweekly):
Name: loanProdAbc
ShortName: labc
Category: foo
Start Date: 2008-01-01
Applicable For: Client
MinAmount: 100
MaxAmount: 100000
DefaultAmount: 5000
RateType: Flat
MinRate: 10
MaxRate: 50
DefaultRate: 18.9
MinInstallments: 1
MaxInstallments: 30
DefaultInstallments: 12
InterestGL: 12313
PrincipalGL: 3212

So if we could have that, then we could debate about what the fixture
behind this object would do to inject the data. Thoughts?

------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev

[Vivek Singh]

>> Vivek, JBehave looks interesting.  I looked at Twist over a year ago.  Would you say they are similar?  Can you point us to any projects that are using JBehave today to get an idea on how it is practically used?

I went through our internal mailing list. There are three alternatives:

a) JBehave: Its light weight but suffers from the problem of not being a compiled language.

b) Simple DSL Java

Following summarizes it pretty well.

http://fabiopereira.me/blog/2010/05/28/bdd-with-scenario-code-dsl-sometimes-you-dont-need-a-tool/

c) Twist: It has refactoring support. Given that most of it would be driven by developers in our case, it might turn out to be beast.

I tend to agree with the blog and it might be useful to follow something like this for our requirements.

On 20 October 2010 11:44, Jeff Brewster <jbre...@grameenfoundation.org> wrote:

[Vivek Singh]

I completely understand the barrier you are talking about. This is what makes us take less than ideal approach. Mifos is not unique in this. But even when we are using SQL there are following things which would make it less ominous for test authors.

a) We can incrementally build a simple API over SQL specific to our domain. So the user would be using this API rather than plain SQL. This would reduce duplication and hide unnecessary details. This API would be a acceptance test artifact and not use any of the code elements. This API doesn't have to do any logic or validation which is present in the code. Since it is dumb it would be less brittle.

b) I feel that from testing perspective we would not be interested in setting every column of data in the table for every test. The API can provide convenience methods for these.

c) Very soon we would cover all the tables in our system, after which we can start harvesting this than constantly building it.

This sounds a lot of theory and we probably would need a working example for evaluation. I would give it a shot (after my vacation at the earliest).

 >> As a test writer, what I would like is a mark-up style file to define the entities that my test requires. For example a YAML (http://en.wikipedia.org/wiki/YAML) file that defines the data I need to create for my test.

I agree that something like YAML would be good fit for this. Sometimes I feel Java not too bad either.