improving BridgeDb testing
Martijn van Iersel
I'm looking for input from developers on how to improve testing in
BridgeDb - this is definitely an area where BridgeDb can be massively
improved.
* Uniformity. In the past I've attempted to create a single test script
to test implicit assumptions in the IDMapper interface. All IDMapper
implementations should be able to pass this. This is not unit testing,
and so I haven't found a satisfactory way to do that within the jUnit
framework. My initial attempt at this can be found here:
http://svn.bigcat.unimaas.nl/bridgedb/trunk/benchmarking/test/org/bridgedb/benchmarking/,
but it could use some attention.
* Automated builds.
We've got a homegrown online buildsystem at
http://ragamuffin.bigcat.unimaas.nl/buildsystem/benchmark/pv2.bridgedb1/. This
build system is written in python. But it's not perfect. For one thing,
we should be able to capture the output of jUnit, so we can get a single
overview of the reliability of each test.
* Offline / Online. Some tests depend on an external webservice, and
they only work when online. Rather than aborting, a failure in these
tests should produce a WARNING. I recently added a patch from Christian
that wraps a test in a try / catch and prints WARNING to stdout. That's
a start, but it's not the complete solution. To be most useful, the
warning should be captured and added to a report on the automated build
system. Also, adding such a try / catch to a lot of tests gets very
verbose - is there a better way to do this, perhaps using
aspect-oriented programming?
* Coverage. At the moment there is no formal coverage measure, how can
that be improved?
* Skipping tests - jUnit seems to be the lowest common denominator for
testing. The fact that you can't skip tests in jUnit is an issue. At the
moment, if a test is problematic it has been permanently disabled, but
that is a bad solution. Are there any alternatives to jUnit that we
should examine? Does anybody have experience in this area?
regards,
Martijn
Egon Willighagen
<mvani...@gmail.com> wrote:
> I'm looking for input from developers on how to improve testing in BridgeDb
> - this is definitely an area where BridgeDb can be massively improved.
>
> * Uniformity. In the past I've attempted to create a single test script to
> test implicit assumptions in the IDMapper interface. All IDMapper
> implementations should be able to pass this. This is not unit testing, and
> so I haven't found a satisfactory way to do that within the jUnit framework.
> My initial attempt at this can be found here:
> http://svn.bigcat.unimaas.nl/bridgedb/trunk/benchmarking/test/org/bridgedb/benchmarking/,
> but it could use some attention.
For the CDK we use JUnit testing for particular use cases, making the
minimal use case the 'unit'. But we also have really unit tests for
get/set methods.
Can you give an example of the IDMapper interfaces tests you are thinking about?
> * Automated builds.
> We've got a homegrown online buildsystem at
> http://ragamuffin.bigcat.unimaas.nl/buildsystem/benchmark/pv2.bridgedb1/.
> This build system is written in python. But it's not perfect. For one thing,
> we should be able to capture the output of jUnit, so we can get a single
> overview of the reliability of each test.
We're exploring setting up Hudson/Jenkins here at UM. That can do the
above, or just be in parallel.
> * Offline / Online. Some tests depend on an external webservice, and they
> only work when online. Rather than aborting, a failure in these tests should
> produce a WARNING. I recently added a patch from Christian that wraps a test
> in a try / catch and prints WARNING to stdout. That's a start, but it's not
> the complete solution. To be most useful, the warning should be captured and
> added to a report on the automated build system. Also, adding such a try /
> catch to a lot of tests gets very verbose - is there a better way to do
> this, perhaps using aspect-oriented programming?
>
> * Coverage. At the moment there is no formal coverage measure, how can that
> be improved?
>
> * Skipping tests - jUnit seems to be the lowest common denominator for
> testing. The fact that you can't skip tests in jUnit is an issue. At the
> moment, if a test is problematic it has been permanently disabled, but that
> is a bad solution. Are there any alternatives to jUnit that we should
> examine? Does anybody have experience in this area?
There is @Ignore, and something new is, and I haven't played with that
yet, conditional running. Also, it's possible to define dependencies
between unit tests...
Such we could use also for on/offline.
Egon
--
Dr E.L. Willighagen
Postdoctoral Researcher
Department of Bioinformatics - BiGCaT
Maastricht University (http://www.bigcat.unimaas.nl/)
Homepage: http://egonw.github.com/
LinkedIn: http://se.linkedin.com/in/egonw
Blog: http://chem-bla-ics.blogspot.com/
PubList: http://www.citeulike.org/user/egonw/tag/papers
Christian Brenninkmeijer
If you upgrade to the latest JUnit you have a few advantages
1. You no longer have to extend TestCase
2. You can use @Before and @BeforeClass to set methods to run before the
actaul tests.
3. You can use Assume.assumeTrue(....) to stop a test without generating
a failure if the assumption fails.
See example below
I would suggest writing an abstract IDMapperTest class which individual
mapper test classes can expand.
Abstract so it includes setup methods that are implementation specific.
The you can use the Assume.assumeTrue(....) to stop tests if particular
conditions are not met.
For example ofline compliation or third party WS is down.
=====
Silly example:
package uk.co.brenn.test;
import static org.junit.Assert.*;
import org.junit.Assume;
import org.junit.Before;
import org.junit.Test;
/**
* Unit test for simple App.
*/
public class AppTest {
@Before
public void beforeMethod() {
System.out.println("try");
Assume.assumeTrue(false);
System.out.println("you never get here");
}
/**
* Failing test but should never be reached.
*/
@Test
public void testApp()
{
assertTrue( false );
}
}
--
Christian Brenninkmeijer
University of Manchester
MyGrid team
Egon Willighagen
On Mon, Feb 20, 2012 at 2:16 PM, Christian Brenninkmeijer
<"christian"@mygrid.org.uk> wrote:
> If you upgrade to the latest JUnit you have a few advantages
I have various branches with code changes here:
https://github.com/egonw/BridgeDb/branches
all have matching pull requests...
One is around updating to JUnit4.
Martijn van Iersel
--
Martijn
Egon Willighagen
<mvani...@gmail.com> wrote:
> AFAIK BridgeDb is already using Junit 4, isn't it?
Yes, it was at places, but not everywhere yet. My 7-junit4 branch
moves all old Junit3 code to Juni4 formalism.
Alasdair J G Gray
* Offline / Online. Some tests depend on an external webservice, and they only work when online. Rather than aborting, a failure in these tests should produce a WARNING. I recently added a patch from Christian that wraps a test in a try / catch and prints WARNING to stdout. That's a start, but it's not the complete solution. To be most useful, the warning should be captured and added to a report on the automated build system. Also, adding such a try / catch to a lot of tests gets very verbose - is there a better way to do this, perhaps using aspect-oriented programming?
* Coverage. At the moment there is no formal coverage measure, how can that be improved?
Kilburn Building
Oxford Road
Manchester
M13 9PL, UK
Please consider the environment before printing this email.
Christian brenninkmeijer
On 13 Feb 2012, at 10:10, Martijn van Iersel wrote:
* Offline / Online. Some tests depend on an external webservice, and they only work when online. Rather than aborting, a failure in these tests should produce a WARNING. I recently added a patch from Christian that wraps a test in a try / catch and prints WARNING to stdout. That's a start, but it's not the complete solution. To be most useful, the warning should be captured and added to a report on the automated build system. Also, adding such a try / catch to a lot of tests gets very verbose - is there a better way to do this, perhaps using aspect-oriented programming?
For example to make my new WSClient build even if the server is not there I have added:
public static WSInterface createTestWSClient(){
WSInterface webService = new WSClient("http://localhost:8080/OPS-IMS");
try {
webService.isFreeSearchSupported();
} catch (Exception ex) {
System.err.println(ex);
System.out.println ("***** SKIPPING WSClientTest ******");
System.out.println ("Please make sure the server is running");
org.junit.Assume.assumeTrue(false);
}
return webService;
}
This tries a simple call and if it fails just prints a warning.
There are mechanisms within maven configure tests to run under certain conditions, look at the junit maven integration pages. Probably the same can be done with Ant.
* Coverage. At the moment there is no formal coverage measure, how can that be improved?
There is a mechanism within maven to output a coverage report which can also test that you have sufficient coverage, look at cobertura. Again the same can probably be done with Ant.
Sorry for the late response on this thread. Oh and I would support the use of a hudson server to automatically build the system whenever there is a commit.
Alasdair
Room 1.17School of Computer Science
Kilburn Building
University of Manchester
Oxford Road
Manchester
M13 9PL, UK
Please consider the environment before printing this email.
No virus found in this message.
--
Checked by AVG - www.avg.com
Version: 2012.0.1913 / Virus Database: 2114/4898 - Release Date: 03/27/12
You received this message because you are subscribed to the Google Groups "bridgedb-discuss" group.
To post to this group, send email to bridgedb...@googlegroups.com.
To unsubscribe from this group, send email to bridgedb-discu...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/bridgedb-discuss?hl=en.