Microprofile integration tests

[yann Blazart]

Hello everybody, long time ago that I didn't take follow microprofile because I was really busy with my customer.

Well, I wanted to share a user point of view, something that is really important for me : integration tests :)

I can remember you that blog post of Antonio about this subject ( https://antoniogoncalves.org/2018/01/16/java-ee-vs-spring-testing/). Even if I'm not agree with all,  I admit that he touched a sensitive point.

Of my point of view, even it's a good product, Arquillian is not enough fast to do integration tests in real life projects.

To test your application against multiple implementations, yes it's a good tool.

But for simple integration tests, when you want to validate your code against specs, too too slow.

For my own projects, for my team, I use TomEE ApplicationComposer to test Java EE 7 code.I tried with Arquillian, with the amount of test ; impossible.

And now, since I'm experiencing microprofile, the best solution I found is in Meecroware, with the Meecroware Junit Runner, really fast and easy to use.

If we want that microprofile gain points in  heart of developpers, and gain some popularity against spring, at this point, I think we need more a way to do like MeecrowaveRunner than new specs, automated testing is very important. Microprofile is made to be light and fast.

This should be part of a testing spec, what do you think about that ?

Regards.

[m.reza.rahman]

I have been trying to schedule a public meeting with the Arquillian team on these very real, practical usability issues. Hopefully they will arrange that soon and finally address these gaps in an otherwise excellent tool.

Sent via the Samsung Galaxy S7, an AT&T 4G LTE smartphone

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/c0a7d32b-7989-497f-97cd-5d6a5095c542%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alasdair Nottingham]

I agree that we need a really good test story for MicroProfile and Jakarta EE. It would be good if we could do a significant amount of the testing without starting a server, a bit like Spring allows in their test framework. 

I haven’t looked at Meecroware or TomeEE ApplicationCompser, but I really think this is an area we need to do better in.

Alasdair

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/5b34feeb.1c69fb81.3c428.c964%40mx.google.com.

[James Roper]

It would be really nice if you could have a method for starting a server where you need to provide nothing more than a map of configuration overrides, and it will just start your app. This is essentially what we have with Lagom, this is what it looks like to write an integration test:

withServer(defaultSetup(), server -> {
  // the server gives you the ability to then create a REST client and make calls on the server through that
});

The withServer call not only starts your application, and bootstrap takes a few hundred milliseconds, so it's really fast. It can also work starting the server in @BeforeAll and @AfterAll annotated JUnit test methods, if you can't bear the few hundred milliseconds start up/shutdown time per test and so what to share the same server between tests.

Of course, Lagom isn't (yet) a MicroProfile server, but, I don't think any integration test API that MicroProfile offers needs to be more complex than that.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/2F10D881-A869-4D93-BC6A-D9A03900B700%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper

[yann Blazart]

I like this way to do, sympathic !

But I think it should go over than simply quickly start the server.

One big advantage I had with TomEE ApplicationComposer and I got again in MeecrowaveRunner is the ability to specify the classes used in the CDI environnement and sometime the ability to mock some of them.

This can be usefull for example when you need to call an external service (quotes or over stuffes). And this help to reduce the amount of work to do, just have a clear and fixed scope for your test.

In my big project, I have somes tests that run 3 instances of ApplicationComposer, each with their own classes, then I can test the interactions properly, closed to real, and very efficient.

Regards

[Emily Jiang]

Can I also add one of my wishes:

The new test framework should make running against different app server very simple. e.g. by just specifying the server location via a config property.

We should aim that way....

Thanks
Emily

[yann Blazart]

Of course you can ! :)

And of course it should works for all microprofile implementations.

But the 'server location' is too Arquillian oriented.

In my mind swapping implementation is more done by changing deps in maven pom.xml. Of course, changing implementation, in tests, force to change configurations of resources used in these tests.

I imagine a dependency for test api (Junit & Test ng runners...) and a dependency per microprofile impl, wich will be to run CDI container in isolated classloader.

Such test framework must work directly in maven, or eclipse or Intellij / netbeans without have to use plugins or specific configuration each time.

[Andy Guibert]

I agree that we need a better testing framework for Java EE / MP applications than what Arquillian currently offers.  However, I think that improving on Arquillian would be the easiest path forward (until someone comes up with a proposal that warrants side-stepping Arquillian entirely).  Personally, I hate needing to write up mock objects for testing, as it wastes valuable development time.  I would much rather start up a real server and add a few seconds to the test runtime than to have to write extensive mocking code and worry that the mock objects behave differently than the real application will.

Of course, when testing a microservice architecture it isn't always feasible/possible to deploy all of the microservices during the test, so mocking interactions with other services will still be necessary.

I have extensive experience with testing Java EE applications (I do a lot of work on the custom test harness that Liberty uses), and I don't see starting up a real server with the real app as a big problem.  Normally it takes <5 seconds to start a Liberty server with a basic test application, and then we can drive hundreds of live-server tests through that test application in a few seconds. No mock objects required. 

I think the main issues with Arquillian are:

1) application packaging via ShrinkWrap is too tedious 

   SOLUTION: acknowledge that standard project layouts are a thing, and leverage that pattern to be able to build an app in 1 line of code (see ShrinkHelper.defaultApp() below)

2) arquillian does a lot of server stop/starts, which eats up much of the overall time a test suite takes

   SOLUTION: reverse the antipattern of deploying lots of tiny test applications, and instead deploy the real application + some server-side testing classes (e.g. TestServletA below).  This greatly simplifies/reduces the need to interact w/ Shrinkwrap and saves on overall test time.

3) setup is more complex that simply adding a dependency to your pom.xml

   SOLUTION: App servers such as Liberty already have their own build plugins with default values for things like hostnames/ports/disk locations (which can be overridden).  If there was some sort of glue between app server plugins and Arquillian, this could all be figured out automatically and there would be no need for an arquillian.xml, getting the simplicity level down to specifying pom dependencies.

To illustrate what we use in Liberty, our test-driver class is very basic, something like:

@RunWith(FATRunner.class)

public class SimpleTest {

    public static final String APP_NAME = "app1";

    @Server("FATServer")

    @TestServlet(servlet = TestServletA.class, contextRoot = "APP_NAME")

    public static LibertyServer server;

    @BeforeClass

    public static void setUp() throws Exception {

        // assumes standard app layout for java and non-java resources. 

        // Only need to specify the server to deploy to, app name, and java packages to include

        ShrinkHelper.defaultApp(server, APP_NAME, "com.app1.web.*"); 

        server.startServer();

    }

    @AfterClass

    public static void tearDown() throws Exception {

        server.stopServer();

    }

}

Then, all of the test code itself lives in TestServletA.java, which itself is a servlet annotated with regular JUnit @Test annotations, and can get real objects injected into it:

@WebServlet("/TestServletA")

public class TestServletA extends FATServlet {

    @Inject

    MyBean bean;

    @Test

    public void basicTest() throws Exception {

        Assert.assertTrue(bean.doesSomething());

    }

}

This isn't too far off from what the Spring test suite does already.  If people are on board with this sort of approach, I'd be happy to contribute.

- Andy

IBM

[m.reza.rahman]

I think we really need to have a substantive dialog with the Arquillian team. I will follow up again on Thursday. This is almost exactly the feedback the community has already given the Arquillian team for some time now.

Sent via the Samsung Galaxy S7, an AT&T 4G LTE smartphone

-------- Original message --------

From: Andy Guibert <andy.g...@gmail.com>

Date: 7/3/18 4:03 PM (GMT-05:00)

To: Eclipse MicroProfile <microp...@googlegroups.com>

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/1925677a-4588-42e0-b8b9-f11edab70e38%40googlegroups.com.

[Bartosz Majsak]

Hi folks,

better late than never, huh? :) Holidays are slowly ending and we, Arquillian team, is back in the game. 

We would be more than happy to host a hangout with the community so please spread the word https://twitter.com/arquillian_org/status/1040165793550479361 and join the discussion on our forum. 

It's very important for us to understand the needs of the community and improve our ecosystem both from the capabilities perspective as well as documentation/knowledge sharing. Latter seems very important as confusion about the way how Shrinkwrap can be used and perceiving it as a big limitation/drawback is mainly coming from not too extensive explanation in our docs / tutorials etc. We are working on improving that.

Cheers,

Bartosz

[Arjan Tijms]

Hi,

Some things were Arquillian could be improved is to allow all options that now have to go into an arquillian.xml file to be placed directly as system properties in the pom.xml file. A number of Arquillian containers such as out Payara Micro Arquillian container and the TomEE one already support that in a proprietary way.

Another thing may be the setup of server specific resources. JBoss has its own API for that, but it would probably be good to provide a handle and a base API to support this universally.

As usability concerns, there's a separate resolver now for just taking all the output of a maven build and using that for the archive, instead of creating the archive manually with shrink-wrap. For small services that's often much easier. This is however a bit verbose to specify and probably could have its own shortcut, maybe even just an annotation.

@ArquillianResource URL injects the http address of the server and test app, which is good. However, servers may run https on another port (such as Payara does), and there's no easy way to get hold of that. Maybe an attribute or qualifier like annotation for @ArquillianResource could be used there.

As for the startup times and overal speed, many servers have embedded connectors which take the bulk of the startup time and connection overhead away, although are sometimes troublesome (class loader issues).

Another point worth looking into is parallel testing; if containers can startup at free ports (such as e.g. TomEE and Payara Micro can), it can speed up tests quite a bit. Most CPU have at least 4 to 6 cores these days. Possibly Arquillian containers can introduce explicit API for "startup at any free port", so that if it detects that mvn -TC1 or so is used, it can ask the containers universally to indeed startup at such a free port.

Kind regards,

Arjan

[Andy Guibert]

Hi all,

I've been investigating some ways to improve the testing experience for MicroProfile and Java EE apps recently. To help shape what I do and focus on, I've created this short (2 minute) survey:

https://forms.gle/sJHmLrwBZd8YPnTb9

Please take a few moments to fill out the survey and make your voice heard!

- Andy