Random unit tests failures

[Jeff Nelson]

Chen,

As you've seen, some unit tests will occasionally fail and you aren't able to reproduce the failure by simply re-running the test.

The reason this occurs is because a lot of the tests in concourse-sever use random data. This is done intentionally so that we get wider coverage over different kinds of input. 

A good illustration of this system can be found in StoreTest.java. An example is the testSearch method:

@Test

@Theory

public void testSearch(SearchType type) {

   String query = null;

   while (query == null) {

   query = TestData.getString();

  }

     Variables.register("query", query);

     String key = Variables.register("key", TestData.getString());

     Set<Long> records = setupSearchTest(key, query, type);

     Assert.assertEquals(records, store.search(key, query));

}

You'll notice that the "query" variable is populated by calling TestData.getString(). TestData has a lot of functions that will return random data. The test will simply call a method that corresponds to the type of data that it wants, but the actual value will be different.

You'll also notice there are calls to Variables.register. This allow us to record the random value that was used in the test. If the test fails for some reason, then all of the variables that were registered are automatically printed to the console so that we know the state of the test and can try to reproduce the error conditions.

Issue https://cinchapi.atlassian.net/browse/CON-146 has an example of the information that is printed to the console when a test fails.

Whenever we see a test failure because of random input, we take the console output and create a unit test that reproduces the exact error conditions. You'll notice that StoreTest has a lot of tests like testSearchReproA, testSearchReproB, etc. 

If a test fails because of random input but the state at the time of failure isn't printed to the console, then the appropriate thing to do is add logging to the test like I did in this commit: https://github.com/cinchapi/concourse/commit/e38b5b8d86233b718700babc336295ca08648ed6

--

Jeff Nelson

Co-Founder & CEO | Cinchapi, Inc.

je...@cinchapi.com | 267-495-7751 

Follow us on Twitter | Facebook | Github | AngelList

Start using Cinchapi for ad hoc analytics and development of real-time data.