Arquillian support

[Sebastian Łaskawiec]

Hi Dawid,

Currently I'm struggling to get JUnitBenchmarks and Arquillian integration tests working together. 

My primary goal is to run all performance tests in nightly build and gather historical data.

I had to do several changes and perhaps it would be a good idea to create some more generic solution. Below is a list of changed I made and my thoughts about them:

• I had to add MVCC=true parameter to H2 connection string. Because H2Consumer is final, I had to copy this class and add this little String. Perhaps it would be a good idea to remove final modifier and create a method (protected?) called getConnectionUrl() or something like that.
• Now the second one is harder :) Arquillian test are performed inside the remote container. In other words, we can not assume that all JVM shutdown hooks will work (in fact the don't). That's pretty bad, because charts are generated in close method. I had to explicitly delegate close method and call it in @AfterClass. I believe there is no better option when using JUnit Rules. However in my opinion, this could be achieved when using Runner interface. This way the framework would not rely on JVM shutdown hook.
• Perhaps it would be convenient to create a configuration annotation? All consumer could be statically listed there. This would separate test configuration from environmental variables or at least would create an alternative for them. Now I'm injecting system properties in Maven Surefire plugin configuration, which isn't very intuitive. 

What do you think about these comments?

Regards

[Dawid Weiss]

Cześć Sebastian,

> My primary goal is to run all performance tests in nightly build and gather historical data.

Yeah, this seems to be on many people's radar recently. Peek at the
mailing list archives to see where the potential point of concerns are
(varying hardware, JVM upgrades, order-specific JIT optimizations,
etc.)

> I had to add MVCC=true parameter to H2 connection string. Because H2Consumer
> is final, I had to copy this class and add this little String. Perhaps it

What does this parameter do, just out of curiosity? Maybe we should
add it by default? Anyway, that final seems to be a mistake/ noise
anyway and I've just removed it.

> Now the second one is harder :) Arquillian test are performed inside the
> remote container. In other words, we can not assume that all JVM shutdown
> hooks will work (in fact the don't). That's pretty bad, because charts are
> generated in close method. I had to explicitly delegate close method and
> call it in @AfterClass. I believe there is no better option when using JUnit
> Rules. However in my opinion, this could be achieved when using Runner
> interface. This way the framework would not rely on JVM shutdown hook.

Yes, you could probably use a custom runner to do this. This would
have additional benefits:

1) people would stop complaining about ParameterizedRunner not working
properly with charts (there's no reliable way to get a test method
instance via JUnit Description API),

2) you could pass the actual Method instance to the benchmarking rule
via framework-internal mechanisms,

3) you could put the benchmarking Statement as the lowest-level entry,
before anything else gets wrapped around with rules, before-after
hooks, etc.

4) you could clean up properly without resorting to those nasty
shutdown hook hacks (or @AfterClass cleanups).

> Perhaps it would be convenient to create a configuration annotation? All
> consumer could be statically listed there. This would separate test

I honestly think a redesign from scratch would be needed here. This
project was pretty much a short prototype effort to see where we could
take this. It's grown pretty hairy and scary in many places and I
think a lot of improvements people have been talking about could be
integrated in a new design.

> What do you think about these comments?

I think they make a lot of sense but I don't see how I could handle
them by myself at the moment ;) Do you have spare development cycles
(an intern maybe? Google SoC?) that could take care of these? I'm
really swamped with work and it'll be hard to carve out some time for
this (even if it's not really a super-long effort). In any case,
thanks for bringing your voice in the discussion, these ideas make a
lot of sense.

Dawid