Reasons for using proprietary testing infrastructure

[Paul Hrdina]

Dear maintainers of H2 database,

I've been using H2 as a client via Maven JAR artifacts for years - especially during my seminars at the university where I'm teaching. And I really love the simplicity, versatility and conformance to SQL standards.

What a surprise I've experienced, when I cloned the source repo for the first time and wanted to run the tests in the IDE (IntelliJ IDEA) - from the package structure in the `src/test/java` I gained false assumption that there's reasonably rich testing part, and I focused more on `src/main/java`, where I was looking for good examples of Code Smells for my other classes (on a reasonably large code base which is not under any NDA so that I can freely show it during the lecture). However, when I become curious how much test coverage there is in the project, and ran all tests in the whole module (using my standard routine), only a single JUnit test started to run and continued to run for many more minutes...

I decided to focus more attention to the tests and realized H2 is equipped with a home-baked infrastructure for organizing and running the tests. That infrastructure appeared almost 20 years ago, when JUnit 4 was already out the door. And nowadays (after 20 more years) the tests in this project still look very much the same - and completely strange for heavy users of xUnit testing infrastracture family.

Before investing more into this, I'd like to learn reasons behind so outdated mechanism for maintaining an extensive test suite (and forcing individual tests to look so weird - a single test method per class, amounts of assertions in a single method).

I'd like to ask anybody with some background information regarding tests in the H2 DB repo to give me hints why their "status quo" stays unchanged for so long (and sick from the perspective of even 20-years old xUnit patterns).

Thanks a lot,
Paul

[Thomas Mueller Graf]

Hi,

I implemented this to speed up tests. I know JUnit etc and use them for Apache Jackrabbit / Jackrabbit Oak at my job.

For H2, I think JUnit, especially at that time when I built it, was not a good match. For H2, the same tests are run with various combinations (in-memory, persisted, remote). Even with @Parameterized I'm not sure if performance would be good.

> outdated mechanism

I don't agree it's "outdated". It's just "different from what you might use for other things".

Regards,

Thomas

--
You received this message because you are subscribed to the Google Groups "H2 Database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h2-database...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/h2-database/2d04a98c-f35d-4ca2-935d-c308ce6aefe0n%40googlegroups.com.

[Andreas Reichel]

Hello Paul,

this has been subject to discussion before already. I myself also would have been in favour for a Gradle build and JUnit5 tests -- but the main developers don't share this sentiment and they have some strong points:

1) it just works (good enough)

2) there are marginal benefits, nothing really solid

3) its a lot of effort to migrate (when it just works and does not provide benefits)

So nobody wanted to die on this hill and unless a super hero takes up this task and migrates all of that on his own -- while at risk being rejected -- nothing will happen.

Cheers and best

Andreas

--

[Thomas Mueller Graf]

Hi,

(I'm not currently actively involved in the development any longer, but I want to share my thoughts.)

A) I wouldn't combine this with moving Gradle (or moving to Maven). I would really just convert the tests to JUnit. That's it. Changing the tests and changing the build are different topics in my view.

B) If someone wants to migrate the tests fully to JUnit, then I would first address the problems. How to ensure that running the same tests, when using JUnit, is not much slower? The problems I have already described: For H2, the same tests are run with various combinations (in-memory, persisted, remote). Even with @Parameterized, my fear is that tests would be much, much slower, because for each test, the server would be stopped, stopped, started,... What you would need is a way, using JUnit, to run the existing tests without slowing them down.

> some strong points:

What I wrote is _not_ one of the points you mentioned. Yes, the points are valid, but I think that's not the main issue.

But what I would add is: let's say you migrate _some_ of the tests to JUnit, but most stay with the current mechanism. In this case, you have a mix: some tests use JUnit, and some don't. Which is probably worse than the current situation.

Regards,

Thomas

To view this discussion visit https://groups.google.com/d/msgid/h2-database/5c2fb806d9f5d5f9bd88de82ab77cfe73599e91e.camel%40manticore-projects.com.

[Andreas Reichel]

On Wed, 2025-03-19 at 14:37 +0100, Thomas Mueller Graf wrote:

A) I wouldn't combine this with moving Gradle (or moving to Maven).

Greetings, I only mentioned that because it affects accessibly from any modern IDE (which is my opinion was the main advantage of touching Build and Test system).

If we continue to ignore IDEs then the current situation is just good enough.

Best and cheers

Andreas

[Paul Hrdina]

Hi Thomas and Andreas,

Thank you very much for your insights into the current state of H2 testing infrastructure (as well as the build system).

I now understand that the primary problem is not "absence of JUnit" but the orthogonality of tests, and their granularity. In the ideal world, there would be most of the unit (i.e. super-fast) tests, much less integration (maybe still relatively fast) tests and just a handful of end-to-end (possibly slow) tests. The performance tests are yet another story. If I break anything, ideally the corresponding unit tests would fail, pointing me in a very narrow way to the source of the problem.

Having just a single set (a few hundreds) of test cases and executing them in a variety of ways (environments) for varioius reasons doesn't sound like the ideal way for such a large system like H2. The resolution, therefore, won't be anything easy.

I also completely agree that "changing the tests and changing the build" are really orthogonal topics. And I can second that having "a mix: some tests use JUnit, and some don't" is definitely worse than the current situation.

Because I'm really interested in H2 for multiple reasons, it's very likely I'll give it a try and entangle the testing infrastructure. But because of the complexity and because I'm just a regular hero (not a super hero; my surname "Hrdina" literally means "Hero" in the Czech language), I don't expect miracles in a short time frame. I'll let you know once I have a protoype, showing the direction towards a better world.

Reagrds,

Paul