Introducing prototype for an easy to use integration test framework for MicroProfile apps

[Andy Guibert]

Hi everyone,

Currently we seem to have a bit of a gap for testing solutions in the MicroProfile/JavaEE space, especially when it comes to integration testing. The current best option is Arquillian, but I have personally struggled with this framework a lot and I do not find it easy to use as an application developer, and it also does not always have consistent dev/prod behavior parity.

I would like to propose a simple, user-facing integration test framework based on Docker. The main idea is that the integration test framework actually starts up a container of your app, and then you write client-side tests to perform black-box testing of the application's running container. I originally outlined this concept in a blog post earlier this year. 

The setup/prereqs for this test framework is simple:

 1) Have Docker installed locally, or have access to a remote Docker host

 2) Add a single test-scoped maven dependency

 3) (optional) Have a Dockerfile in your repo, or have access to a pre-built image you would like to test

From there, we can leverage a popular existing Java library called Testcontainers to build (if needed), start, and stop the container.

Suppose we want to test a simple HelloService JAX-RS endpoint. The integration test would look something like this:

@MicroProfileTest

public class HelloServiceTest {

  @Container // This automatically starts the app container before the test runs

  public static MicroProfileApplication app = new MicroProfileApplication()

                              .withContexRoot("/myservice");

  @Inject // This gives us a REST client proxy (similar to MP Rest Client) for interacting with the remote service

  public static HelloService helloSvc;

  @Test

  public void testHelloEndpoint() {

    // Using the injected REST client proxy, we can easily invoke HTTP requests to the running container and read the response

    assertEquals("Hello world!", helloSvc.sayHello());

  }

}

I believe this has several key advantages over existing solutions:

 - Test the app _exactly_ as it would run in production (aside from env vars or mounted secrets)

 - Easy to set up and run

 - Works for any MicroProfile implementation already -- little/no vendor-specific code needs to be implemented or maintained

 - Integrates seamlessly with external resources (databases, external HTTP services, or messaging engines) because external resources were the original purpose for Testcontainers

 - Server-side logs are merged with JUnit output, making it easy to correlate server logs with their corresponding client-side test

My current prototype can be found on Github here: https://github.com/temp-xfer/mp-jee-testing

I have currently tested a JAX-RS + CDI + MongoDB + MP Rest Client app successfully with:

 - OpenLiberty / WAS Liberty

 - Wildfly

 - Payara Micro

 - TomEE

(Also works with Quarkus, but Quarkus already has its own custom testing framework)

Next steps:

Since this approach has more to do with Docker/Testcontainers than MicroProfile, I was thinking the best home for this project would be in the Testcontainers org. I have approached the Testcontainers org and they seem interested and excited about this new framework, and are willing to accept it under their org. I would like to invite other MP folks to give it a try (instructions in the README) and offer feedback. Of course, any contributions or collaboration is encouraged!

Looking forward to hearing people's comments and questions on this.

- Andy

[brian benz]

Nice!  I'm currently testing samples, I'll see how easy it is to plug this into my workflow and share feedback.  

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/5bc99978-5972-4fb9-9d14-e5a26f5392a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ken Finnigan]

From a brief look at the code it appears that this is specific to WAR projects. Is that correct?

[Gordon Hutchison]

I can clearly see the value in this and am also

happy and proud that MicroProfile support would become a

desirable feature of 'other' ecosystems/frameworks that started

independantly of MP.

Gordon

[Andy Guibert]

Hi Ken,

No, it is not just for war projects. Any Java app that is in a Docker container can make use of this. However, the dominant use case I had in mind while designing the prototype was thin-war projects. 

If the container for your app is already built, this works too. Suppose I build my quarkus app into a `my-quarkus-app:latest` container image. If I still have my application classes on my test classpath, we could drive integration tests against this container by simply specifying the container image name as a String parameter to the `new MicroProfileApplication(String imageName)` constructor like this:

@MicroProfileTest

public class HelloServiceTest {

  @Container // This automatically starts the app container before the test runs

  public static MicroProfileApplication app = new MicroProfileApplication("my-quarkus-app:latest")

                              .withContexRoot("/myservice");

  @Inject // This gives us a REST client proxy (similar to MP Rest Client) for interacting with the remote service

  public static HelloService helloSvc;

  @Test

  public void testHelloEndpoint() {

    // Using the injected REST client proxy, we can easily invoke HTTP requests to the running container and read the response

    assertEquals("Hello world!", helloSvc.sayHello());

  }

}

The underlying Testcontainer API is extremely powerful, and there are a lot of cool tricks it can do. Including virtual file system mounting for potentially re-using running containers across test runs during local dev.

On Monday, June 3, 2019 at 3:09:11 PM UTC-5, Ken Finnigan wrote:

From a brief look at the code it appears that this is specific to WAR projects. Is that correct?

To unsubscribe from this group and stop receiving emails from it, send an email to microp...@googlegroups.com.

[John Alcorn]

Hey Andy,

     This sounds like a good step.  However, most of us don't run our Docker images directly - we run our Docker images inside of Kubernetes (such as a test cluster or namespace).  Among other things, that takes care of feeding environment variables from config maps/secrets to each pod's Docker image.  Many microservices won't function properly unless they are fed the values of, say, the host/port/id/password of a database or some external service.  I've been working with one lately whose health check makes sure it got a valid MongoDB connection, and fails otherwise, causing the image to be killed.

     Does this Testcontainer framework have anything that perhaps can parse a Kube deployment yaml for a microservice and grab each field/value from the referenced config map or secret and pass that to the Docker container (via -e flags on `docker run`)?  That would be a really handy optional feature, if you've provided credentials so it can get logged in to your Kube environment for it to extract this data.  Or in a Liberty environment, maybe this approach could be used to produce a server.env file with those env vars.

     Just thinking out loud here - again, I like the proposal, just thinking of how to make it even more useful.  Thanks.

[Andy Guibert]

Hi John,

Kubernetes support has been considered at the Testcontainers layer and I've also considered it as something to do in this new layer. The intent of this test framework is to re-use as much stuff as possible from the production env to minimize dev/prod differences, so re-using YML files would be right in-line with the spirit of this thing. I just haven't gotten that far with the prototype yet =)

As far as what is possible right now, here is a pointer to an example that uses 3 containers: https://github.com/testcontainers/mp-jee-testing/blob/master/sample-apps/everything-app/src/test/java/org/aguibert/liberty/AppContainerConfig.java

 - MicroProfile app container

 - MongoDB container

 - mockserver container (to simulate external HTTP service) 

While Testcontainers doesn't have support for kube config maps yet, it does support Docker compose. 

[Gordon Hutchison]

We are certainly going to be looking to see if we can use this in the pipeline testing of our

MicroProfile Reactive Messaging Connectors in OpenLiberty...

(...and a Kafka test setup will be #1 perhaps inspired by https://docs.confluent.io/current/quickstart/ce-docker-quickstart.html :-) )

Gordon.

[Mark Swatosh]

We've done some looking at testing message based microservices. Currently I have a prototype working that uses the SmallRye Reactive Messaging implementation (in the SE container) to send and validate received messages here:
https://github.com/mswatosh/mp-jee-testing/blob/Messaging-MPRM/sample-apps/everything-app/src/test/java/org/aguibert/liberty/KafkaAndLibertyTest.java

I'm not sure that's the best direction for messaging however, since it does require loading the SE container.

-Mark

[Alexandre Touret]

Hi,

I had taken a brief look to your code. It sounds great.
Is there a way to mix Mock ( with mockito ?) and the tests run with testcontainers?

For example, If we want to run integration test on all the perimeter (MP, database,...) except one or two services ( ex. for HSM connection ) ?

Regards,
Alexandre Touret

[Ladislav Thon]

Hi,

I took a brief look and I don't think this can replace Arquillian. There's a difference (often overlooked) between integration testing and system testing -- while Arquillian has (AFAIK) been designed for the first, and allows the second, this `mp-jee-testing` project is purely system-level testing. That's not necessarily bad, but it's an important difference that I believe should be highlighted. (And I'm saying that as someone who has also struggled with Arquillian heavily and, at one point, worked on an alternative.)

I'm also not sure why @MicroProfileTest is called @MicroProfileTest and why MicroProfileApplication is called MicroProfileApplication. If I understand correctly, this testing framework can be used for all kinds of Java EE, Java SE and even non-Java applications, as long as there's a Dockerfile and an HTTP endpoint. Do you plan to make it more MicroProfile-specific?

Other than that, I like the concept and I think it has a lot of potential. For example, more things could be injected, such as URL of the tested application (MP RestClient won't fit all usecases), or it could vastly simplify setting up the tested application for remote debugger connection.

Looking forward to what grows from it :-)

LT

st 5. 6. 2019 v 8:41 odesílatel Alexandre Touret <alex...@touret.info> napsal:

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/3778a735-e79b-44d7-8faa-0dfaca024596%40googlegroups.com.

[Andy Guibert]

Hi Alexandra,

Mocking can be done, but it has to be outside of the container, because one of the key concepts is that we run tests against the exact same container that is being used in production.

Examples of mocking with this service would be:

 - Using Mockserver to mock out an external HTTP service

 - Using Jwt Token Dispenser for temp JWTs

 - Perhaps using some in-mem message queue like Qpid in place of Kafka (disclaimer: have not tested this)

Overall, only the pieces required by the app path being tested need to be mocked out. If your app has a dependency on an SSO provider, but you are driving code paths that don't require SSO login, then that part doesn't need to be mocked out.

If you think this approach will not work out for your apps, please open an issue on the GitHub repo so we can discuss in more detail: https://github.com/testcontainers/mp-jee-testing/issues

- Andy

[Andy Guibert]

Hi Ladislav,

Thanks for having a look! I agree that isn't an outright replacement for Arquillian because Arquillian does grey-box "in app" testing as well as can be done. When explaining this Testcontainers prototype, I always make sure to state "black box testing" because that's the use case I am targeting. Personally, I'm fine just testing my apps with white-box unit tests combined with black-box testcontainers tests.

> I'm also not sure why @MicroProfileTest is called @MicroProfileTest and why MicroProfileApplication is called MicroProfileApplication. If I understand correctly, this testing framework can be used for all kinds of Java EE, Java SE and even non-Java applications, as long as there's a Dockerfile and an HTTP endpoint.

Regarding naming... you are absolutely right, the applicability of this goes beyond MicroProfile. All of the names like: mp-jee-testing, @MicroProfileTest, and MicroProfileApplication are temporary names and will probably change at some point. If anyone would like to propose more general names, I'm up for it! I think it's nice to have technically-meaningful names, as opposed to made-up names (e.g. Arquillian) though.

> Do you plan to make it more MicroProfile-specific?

Great question =) In the current prototype I am catering to apps that use the core foundation of MicroProfile: JAX-RS, CDI, and JSON.

I also have more integrations planned for this framework which will position it more squarely over MicroProfile technologies, such as:

 - Defaulting container readiness to an MP Health 2.0 readiness check (i.e. GET <host>:<port>/health/ready)

 - Simplifying configuration of external MP Rest Clients

 - Provide an easy-to-use messaging client to help test MP Reactive Messaging apps

 - Simplifying creation of JWTs for apps using MP JWT

 - Provide MP GraphQL client to easily drive test requests to MP GraphQL endpoints (similar how the current prototype uses REST client to drive test requests to JAX-RS endpoints)

However, I am also planning integrations with things outside the MP space, such as:

 - Using the toxiproxy module to simulate network instability (e.g. latency, cut connections, etc), which will be great for testing MP Fault Tolerance procedures in apps

 - Some possible JDBC/JPA integration for clearing/resetting DB state between test classes/methods

The current least-common denominator that is useful here is JAX-RS. But this covers a lot of platforms: MicroProfile, JavaEE, JakartaEE, and JavaSE apps that use JAX-RS. (Note: This may expand to also be MP Reactive Messaging if those integrations go well)

I have a lot of integrations in mind here, but I'm also open to more ideas here, or requests for prioritizing certain parts ahead of others based on what people think is most useful.

- Andy

To unsubscribe from this group and stop receiving emails from it, send an email to microp...@googlegroups.com.

[Emily Jiang]

Thank Andy for the detailed info about adopting TestContainer in MicroProfile!

Personally I think testing is the most needed area to be focused area in MicroProfile. I am very excited with the proposal. Recently at conferences, TestContainer is a hot topic. Microservice is moving towards container. It is time to update test strategy accordingly. I have been looking at this recently. It seems the tests are fast and cheap to be executed, which makes it a great framework to be adopted.

I agree with the LT's comment on the annotation names. I think we should make this new test framework to be easily adopted by either MicroProfile and/or Jakarta EE.

Is there a way to mix Mock ( with mockito ?) and the tests run with testcontainers?

I think mockito is just a lib. Technically, it should be ok to be used in testcontainers if needed. I have not tried myself though. Andy, do you know?

Andy, can you do a PR under MicroProfile sanbox for further collaboration?

Thanks

Emily

[Andy Guibert]

Hi Emily,

> Andy, can you do a PR under MicroProfile sanbox for further collaboration?

I've already moved this repo into the Testcontainers org (https://github.com/testcontainers/mp-jee-testing) and was planning to continue designing and developing it there. I think given the choice between Testcontainers org and the MicroProfile org, this project makes more sense in the Testcontainers org as a MicroProfile module, because it integrates tightly with Testcontainers. I'm certainly open to suggestions on where this thing should be hosted and what it should be called though.

Perhaps we can add an agenda item to the next MP Community hangout and I can give a demo and we can discuss what makes sense for name/location on the call?

-------

>> Is there a way to mix Mock ( with mockito ?) and the tests run with testcontainers?

> I think mockito is just a lib. Technically, it should be ok to be used in testcontainers if needed. I have not tried myself though. Andy, do you know? 

You can't touch the JVM for the MP app under test, because it's running in a Docker container -- and that is by design. We can use whatever means are necessary to mock up things outside of the container for the MP app under test though. So you can use any libs on the test client, and any other containers you'd like (e.g. PostgreSQL, MongoDB, Mockserver, Kafka, Elasticsearch, etc).

- Andy

[Scott Kurz]

It's interesting to see how your latest samples have evolved from your March blog post in terms of when the Docker build is done.

It looks like so far you have all Gradle samples.   One concern I'd had with Maven is that it could be problematic to bind everything to the package phase: 'war:war', docker build, anything else, and to still get the execution order correct within the phase.

Since you used Gradle, this wasn't an issue, and your earlier samples used the composeBuild task.

In any case, your latest samples use your MicroProfileApplication  to build the image during the beginning of test execution.  If Testcontainers has this much functionality, maybe any tricky ordering could be saved for the text execution itself.  

Will be interesting to see some Maven examples too when someone gets the chance.

Scott

[Ken Finnigan]

+1 Andy.

As the testing framework is intended to be more than just testing for MP, and it's tight integration with Testcontainers, I think having it in the Testcontainers community is the best option.

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/071b896b-b0e0-42bd-8eeb-79e5f5695e99%40googlegroups.com.

[Emily Jiang]

Thank you Andy for the extra info!

As per Scott's comments, I am also interested in seeing some maven samples, considering our MicroProfile tests are maven projects.

Thanks

Emily

[Andy Guibert]

Hi Emily and Scott,

The only thing that matters from the build tool perspective (e.g. Gradle or Maven) is that we have an up-to-date app archive (usually a .war) whenever integration tests run. The alternative to this is having an up-to-date Docker image whenever ITs run.  So if we bind the "war"  goal to the correct parts of the build lifecycle such that a fresh app archive is produced BEFORE integration tests run (as Scott suggested) then it would work.

- Andy

[Emily Jiang]

It will help if there is a maven sample to demonstrate this.

Thanks

Emily

[Andy Guibert]

Hi Emily,

As requested, here is a sample app that uses this test framework prototype in a maven project: https://github.com/testcontainers/mp-jee-testing/tree/master/sample-apps/maven-app

To test it out:

1) Clone the github repo

2) go into the sample-apps/maven-app directory

3) run "mvn clean install" and you will see the tests run

- Andy

[Emily Jiang]

Thank you Andy! Tried and it worked nicely. Thanks

Emily

[Andy Guibert]

Hi everyone, I'd like to follow up on this thread by announcing the testing project as I referenced earlier is now available as MicroShed Testing under a new github organization called MicroShed.

You can find the github project here: https://github.com/MicroShed/microshed-testing

With current documentation here: https://microshed.github.io/microshed-testing/

Users: Please give it a try! I'm open to any sort of feedback on the project.

Vendors/Implementors: Although MicroShed Testing works for several runtimes out-of-the-box (e.g. Wildfly and Payara Micro), further optimizations can be done by implementing an optional "server adapter" module. It should be fairly easy to implement using the Liberty module as an example.

Thanks,

Andy Guibert

[Rudy De Busscher]

Hi Andy,

I did not check the project in detail yet, just tried to run it for Payara Micro and my main feedback is

Since it is using the /health/ready URL to verify if container is running, the implementations which can be used is restricted to MP Health 2.0 / MicroProfile 3.0 and not "Work with any JavaEE or MicroProfile runtime" as mentioned in the readme of the project. I have created issue 48 for this.

Further more I have created 2 smaller issues (and PRs for it) regarding an issue in the maven-app example and improvement of the Payara Micro DockerFile.

When time permits, I'll check the project but first impression is very good.

Thanks Andy

Rudy

[Andy Guibert]

Hi Rudy,

Thanks for trying it out! 

The readiness URL is resolved in the following way:

1) If `MicroProfileApplication.withReadinesPath(String)` is specified, it is used

2) Otherwise, if MP Health 2.0 API is available on the test classpath, then "/health/ready" is used

3) Otherwise, the app context root is used

I thought item (2) seemed convenient at the time, but I'm open to suggestions, we can discuss more on the corresponding GH issue as needed.

Also, thanks for the PRs! If you or someone else from Payara find the time it would be great to have a dedicated Payara module too (can use the liberty module as a template).

- Andy

[Rudy De Busscher]

Hi Andy,

I had already a deeper look into the code and was successful in making the Maven example running on Payara (The test class itself just defined a wrong withReadinessPath)

To make the tests even more server agnostic, I propose the following things

- Add detection for MP Health 1.x API

- Add withReadinessPath to ServerAdapter.

I'll create Issues and PRs for this. and also for a Payara ServerAdapter.

We can maybe discuss these technical issues outside of this mailing list / message Thread. I think we should create a Gitter room.

Rudy

[Andy Guibert]

Hi Rudy, good ideas on the readiness path. We can discuss more in issues and gitter.

For Rudy and anyone else who would like to collaborate on MicroShed Testing, we've created a gitter chatroom here:

https://gitter.im/MicroShed/microshed-testing

- Andy