Design Proposal: @QuarkusIntegrationTest

[Stuart Douglas]

Hi Everyone,

As I have mentioned previously I am planning on enhancing our testing capabilities, and to start with I am proposing a new approach to integration testing.

At the moment we have some limitations with regards to integration testing:

- We only provide integration testing for the actual production app for the native images

- These tests in our examples generally inherit from the corresponding unit test, this is confusing as unit tests support extra features that integration tests don't (namely injection), so this approach as the default leads to user confusion.

- We have no way of testing the actual JVM application that is generated

To solve these problems I propose we introduce @QuarkusIntegrationTest which will provide general integration test capabilities. In essence this will work almost exactly the same as our current native testing, except that it will test the generated application no matter what it is. If we are building a JVM application it will run the JVM app, if we are building a native application it works the same as our existing native tests, and so on. This will also allow us to test docker images, as we can use this to run the generated docker image and run tests against it (potentially we could even deploy to a kube pod and test in a cluster, but I am not sure how useful that will be).

From a practical point of view, this would mean the following:

- We discourage RestAssured based testing for @QuarkusTest. Instead we focus on unit style testing of components. Note that you can still test functionality without the transport layer by just injecting JAX-RS resources directly into the test. Is this something that we want to encourage as it provides a really simple way to test logic (while obviously not testing the transport layer)? Also if you really want to RESTAssured will still work fine, this is more just about what ends up in our examples/docs.

- We use @QuarkusIntegrationTest for all 'over the wire' tests, and run them in the integration tests phase. This means the actual application is being tested, but on the downside the application is not built in TEST mode so there is some inconsistency here (same as we already have for native). The actual target of this test will depend on the build output, native builds will test native, Java build will test the jar, Docker builds will run and test the docker image.

To look at a concrete example, the getting started example would have the changes in [1]. Basically the unit test now tests the endpoint without the transport layer, and the integration test runs over HTTP and would test both the JVM and native production applications.

There are some downsides/caveats to this proposal that need serious consideration:

- IDE support will not work as well for integration tests, as it requires a build to update the actual application that is being run. It will likely still be possible to have it launch from the IDE, but changes will not be automatically picked up (unless we can detect this and force a build before launch in this case).

- All integration style tests will be run in a separate process, so you would need explicit debugger support to launch the process in suspended mode, so debugging is slightly more involved.

- The tests would run in the verify phase, and need to follow the failsafe naming convention. This means you need to run 'mvn verify' rather than 'mvn test' and the tests are naming ITCase rather than TestCase. This is all 'standard' maven, however in general developers seem to be more experienced with surefire than failsafe.

In terms of implementation I have not given it a lot of thought yet, as I wanted to get feedback on the idea first. It will likely be some form of SPI that allows a test launcher to be generated in the target dir, which can then be invoked by the test runner to launch whatever the output of the build happens to be, but I don't really want to look into it in depth until we have all the requirements.

If you have any feedback I would love to hear it.

Stuart

[1] https://github.com/quarkusio/quarkus-quickstarts/commit/e1d46252ddc4b96686c3ce2fd11b73f8bb9476aa

[Rostislav Svoboda]

I think the majority of developers will use the current way. It is easy and integrated well (IDE, debugger).

I like this proposal, especially when thinking about it from Quarkus team perspective or larger project perspective.

So if the target audience is us + larger projects, then definitely +1

For single developer / easy day to day usage I give ±0 mainly because of IDE and debugger caveats.

Rostislav

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cx7zQ8y4p%3Dz6cREkDvdLD6CC6XWidtzZ7DbLvw6LjnLPw%40mail.gmail.com.

[Stephane Epardaud]

On Wed, 15 Apr 2020 at 06:37, Stuart Douglas <sdou...@redhat.com> wrote:

-... on the downside the application is not built in TEST mode 

Given the new panache-mock module, which only injects mocking interception points in TEST mode, would this mean we can't mock in ITs or that we need to inject those interception points in PROD mode too, or that we should not be mocking in ITs anyway, and mocking is for unit tests only?

--

Stéphane Épardaud

[Stuart Douglas]

Mocking is unit tests only. This is running the production app in another process, basically the same way we do for native image.

Stuart

 

--

Stéphane Épardaud

[William Burke]

On Wed, Apr 15, 2020 at 12:37 AM Stuart Douglas <sdou...@redhat.com> wrote:

Hi Everyone,

As I have mentioned previously I am planning on enhancing our testing capabilities, and to start with I am proposing a new approach to integration testing.

At the moment we have some limitations with regards to integration testing:

- We only provide integration testing for the actual production app for the native images

- These tests in our examples generally inherit from the corresponding unit test, this is confusing as unit tests support extra features that integration tests don't (namely injection), so this approach as the default leads to user confusion.

- We have no way of testing the actual JVM application that is generated

To solve these problems I propose we introduce @QuarkusIntegrationTest which will provide general integration test capabilities. In essence this will work almost exactly the same as our current native testing, except that it will test the generated application no matter what it is. If we are building a JVM application it will run the JVM app, if we are building a native application it works the same as our existing native tests, and so on. This will also allow us to test docker images, as we can use this to run the generated docker image and run tests against it (potentially we could even deploy to a kube pod and test in a cluster, but I am not sure how useful that will be).

From a practical point of view, this would mean the following:

- We discourage RestAssured based testing for @QuarkusTest. Instead we focus on unit style testing of components. Note that you can still test functionality without the transport layer by just injecting JAX-RS resources directly into the test. Is this something that we want to encourage as it provides a really simple way to test logic (while obviously not testing the transport layer)? Also if you really want to RESTAssured will still work fine, this is more just about what ends up in our examples/docs.

What you mean by "discourage RestAssured based testing for @QuarkusTest" is that, if you're writing Arc tests, you don't need to include JAX-RS and invoke over the wire?  You should just be able to inject things into the test?

 

- We use @QuarkusIntegrationTest for all 'over the wire' tests, and run them in the integration tests phase. This means the actual application is being tested, but on the downside the application is not built in TEST mode so there is some inconsistency here (same as we already have for native). The actual target of this test will depend on the build output, native builds will test native, Java build will test the jar, Docker builds will run and test the docker image.

To look at a concrete example, the getting started example would have the changes in [1]. Basically the unit test now tests the endpoint without the transport layer, and the integration test runs over HTTP and would test both the JVM and native production applications.

There are some downsides/caveats to this proposal that need serious consideration:

- IDE support will not work as well for integration tests, as it requires a build to update the actual application that is being run. It will likely still be possible to have it launch from the IDE, but changes will not be automatically picked up (unless we can detect this and force a build before launch in this case).

This makes @QuarkusIntegrationTest not worth it.  Right-click IDE support is a must for all testing.  While I agree with your idea of testing what's built, the extra test coverage is just not worth it if right-click-IDE run won't work as it will be very very rare that integration testing will pick up a bug that running in regular Quarkus test mode won't.

 

- All integration style tests will be run in a separate process, so you would need explicit debugger support to launch the process in suspended mode, so debugging is slightly more involved.

- The tests would run in the verify phase, and need to follow the failsafe naming convention. This means you need to run 'mvn verify' rather than 'mvn test' and the tests are naming ITCase rather than TestCase. This is all 'standard' maven, however in general developers seem to be more experienced with surefire than failsafe.

Is there a way to detect the difference between a maven build from cmd line and right-click-ide?  Maybe some extra system property that a quarkus plugin would set?  That way right-click-IDE run would work for an QuarkusIntegrationTest too.

Would something not work with @QuarkusIntegrationTest vs. @QuarkusTest?   My next thought is, why not just not have @QuarkusIntegrationTest and just force @QuarkusTest to work with built artifacts if run from mvn command line?  Then there's no need for an extra annotation, things are backward compatible, and you get the extra testing.

Bill

[Stuart Douglas]

On Wed, 15 Apr 2020 at 23:12, William Burke <bbu...@redhat.com> wrote:

On Wed, Apr 15, 2020 at 12:37 AM Stuart Douglas <sdou...@redhat.com> wrote:

Hi Everyone,

As I have mentioned previously I am planning on enhancing our testing capabilities, and to start with I am proposing a new approach to integration testing.

At the moment we have some limitations with regards to integration testing:

- We only provide integration testing for the actual production app for the native images

- These tests in our examples generally inherit from the corresponding unit test, this is confusing as unit tests support extra features that integration tests don't (namely injection), so this approach as the default leads to user confusion.

- We have no way of testing the actual JVM application that is generated

To solve these problems I propose we introduce @QuarkusIntegrationTest which will provide general integration test capabilities. In essence this will work almost exactly the same as our current native testing, except that it will test the generated application no matter what it is. If we are building a JVM application it will run the JVM app, if we are building a native application it works the same as our existing native tests, and so on. This will also allow us to test docker images, as we can use this to run the generated docker image and run tests against it (potentially we could even deploy to a kube pod and test in a cluster, but I am not sure how useful that will be).

From a practical point of view, this would mean the following:

- We discourage RestAssured based testing for @QuarkusTest. Instead we focus on unit style testing of components. Note that you can still test functionality without the transport layer by just injecting JAX-RS resources directly into the test. Is this something that we want to encourage as it provides a really simple way to test logic (while obviously not testing the transport layer)? Also if you really want to RESTAssured will still work fine, this is more just about what ends up in our examples/docs.

What you mean by "discourage RestAssured based testing for @QuarkusTest" is that, if you're writing Arc tests, you don't need to include JAX-RS and invoke over the wire?  You should just be able to inject things into the test?

Yes, basically you can just directly inject the JAX-RS endpoint and invoke the endpoint method to test your business logic. This works at the moment, but is not something we have documented. 

The down side is that serialization logic is not tested, so it is not as complete a test as a full integration test, but if you have complex business logic it is simpler to test this using this unit test and add an integration test to verify the over the wire behaviour.

My thinking around discouraging RESTAssured based tests here is that these tests should be reusable for testing native/docker, as they are purely over the wire. If we encourage all over the wire testing to be done in the integration test phase then this means that the tests should be reusable. At the moment they are semi reusable, where they will work sometimes, but will fail if you use some features (namely injecting into the test).

 

 

- We use @QuarkusIntegrationTest for all 'over the wire' tests, and run them in the integration tests phase. This means the actual application is being tested, but on the downside the application is not built in TEST mode so there is some inconsistency here (same as we already have for native). The actual target of this test will depend on the build output, native builds will test native, Java build will test the jar, Docker builds will run and test the docker image.

To look at a concrete example, the getting started example would have the changes in [1]. Basically the unit test now tests the endpoint without the transport layer, and the integration test runs over HTTP and would test both the JVM and native production applications.

There are some downsides/caveats to this proposal that need serious consideration:

- IDE support will not work as well for integration tests, as it requires a build to update the actual application that is being run. It will likely still be possible to have it launch from the IDE, but changes will not be automatically picked up (unless we can detect this and force a build before launch in this case).

This makes @QuarkusIntegrationTest not worth it.  Right-click IDE support is a must for all testing.  While I agree with your idea of testing what's built, the extra test coverage is just not worth it if right-click-IDE run won't work as it will be very very rare that integration testing will pick up a bug that running in regular Quarkus test mode won't.

We could actually make this work. If we don't have a freshly generated application we can auto generate one and test it, although it still would not help with debugging.

I guess you could run the whole thing in-process for the IDE launch case.

 

 

- All integration style tests will be run in a separate process, so you would need explicit debugger support to launch the process in suspended mode, so debugging is slightly more involved.

- The tests would run in the verify phase, and need to follow the failsafe naming convention. This means you need to run 'mvn verify' rather than 'mvn test' and the tests are naming ITCase rather than TestCase. This is all 'standard' maven, however in general developers seem to be more experienced with surefire than failsafe.

Is there a way to detect the difference between a maven build from cmd line and right-click-ide?  Maybe some extra system property that a quarkus plugin would set?  That way right-click-IDE run would work for an QuarkusIntegrationTest too.

This should be doable, we should be able to set a system property or create a temp file or something similar.

 

Would something not work with @QuarkusIntegrationTest vs. @QuarkusTest?   My next thought is, why not just not have @QuarkusIntegrationTest and just force @QuarkusTest to work with built artifacts if run from mvn command line?  Then there's no need for an extra annotation, things are backward compatible, and you get the extra testing.

Yes, QuarkusIntegrationTest does not support injection into tests. QuarkusIntegrationTest is also built with a launch mode of prod, so you can't really modify built time settings just for a test (e.g. running your tests with h2 and your production on postgres). 

Stuart

 

Bill

[Daniel Petisme]

Daniel PETISME
daniel....@gmail.com

I tried to sum up

QuarkusTest

Goal

- Test the application business logic

- Whitebox testing

Pre-requisities

- Quarkus augmentation

- %test profile

Features

- Manage application (re)start between tests

- Provide DI/Mocking fixtures

- Provide Transaction Management mock

DeveloperExperience

- Right-click in IDE

- Maven surefire

Questions/Comments

- The Data layer is not clear for me, if Quarkus introduces mocks for the data management, is the H2 test resource still needed? If H2 is used in unit tests, it would be tricky to explain why a dev shouldn't use RestAssured to test its API.

QuarkusIntegrationTest

Goal(s)

- Test the application network integration

- Test the native application

- BlackBox testing

Pre-requisities

- Quarkus JVM packaging

- Quarkus Native packaging

- %prod profile

Features

- Manage application (re)start between tests

- Provide TestContainers configuration

- Application Client pre-configuration (RestAssured, AMQP, Kafka)

DeveloperExperience

- Maven failsafe

Questions/Comments

- I have the feeling that we should clearly split the JVM and native mode testing but well at the end it's blackbox testing in the 2 sides...

- How do you do testing if teh %test profile is used for unit testing, here you should use %prod but the application will try to connect to the actual production third parties (DB, Kafka cluster etc) I don't think its something to promote.
- Since Junit is providing the RegisterExtension, could we consider to configura and start a quarkus application progamtically (à la ShrinkWrap) rather than running the actual application jar?

- Failsafe is not very popular compared to surefire, everybody will focus on mvn test and mvn package but not mvn verify

- Generallly speaking true integration tests are often skipped. A true production like environment is always time consuming to setup, the feedback loop is way to long and ends up delegated to a everyday CI build). IMy 2cts  it's ok for native image but not for basic HTTP testing.

Daniel PETISME
daniel....@gmail.com

 

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2czdUqGsJbH6V_YGY_ExV7mqC5F_znG4Hi9iyOGs7zqH9Q%40mail.gmail.com.

[Stephen Nimmo]

I can provide a POV regarding the testing. 

@QuarkusTest - In Spring, I can provide a profile based datasource, which I then use to spin up an embedded or container based database, i.e. https://github.com/opentable/otj-pg-embedded. I usually set this profile-based datasource to fire up on the default profile, which in my mind is "local", as my dev/test/prod environments typically have env vars specifying the appropriate profile. This datasource was also allowed to be located in the test classpath. While doing this introduces the yak shave of unit vs integration test, I tend to primarily focus on integration tests rather than unit tests as I believe they provide the greatest value, and use true "unit" tests primarily for complex logic in a single class. 

As an example, let's say I have a basic CRUD rest api. I'll have the normal Resource/Service/DAO layers. If someone would be "textbook" in their tests, there would be @QuarkusTests for the Resource, mocking the Service layer, then tests at the Service, mocking the DAO. But for initial development, if I can spin up a datasource in a relative short amount of time, I'll do so and write tests for the Resource only as long as the Resource tests provide adequate coverage for all of the services/DAO classes as well. Technically, it's an integration test because I am connecting out to the datasource but because the integration point is lightweight, I can run it locally and quickly, and it gives me the feedback I need before I commit/build/etc. I find very little value in using H2, because my experience with schema support is that H2 is fairly limited and doesn't support what I need. 

mvn test vs. mvn verify - I usually follow the 'surefire = unit, failsafe = integration' model. When I run surefire, I want to see everything that's broken so I can go fix everything at once. When I run failsafe, it typically means I think everything is good to go, and if something breaks, I wasn't expecting it (build, etc). 

Internal vs external testing - I try to write my unit tests at the class level, injecting controllers/service/classes and testing methods, and if it's "easy and fast". I try to write my integration tests to focus on external interfaces, like rest apis. But there is also a third level of tests which I classify as "regression" or "smoke", where it's a set of tests that I can run against ANY environment, test harness focused. 

But then again, this is all dependent on what services I'm connecting to. If I have an application that uses multiple datasources, kafka, JMS, a data grid, etc, then all of this can get real complicated really fast. I may make sacrifices in my unit and integration tests to mock services that are difficult or costly or slow to spin up, to get them to run fast and then concentrate on my external regression test harness to help me validate the complex stuff. 

my 2 cents. Grain of salt, etc. 

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CACHzTrbCruppD%3D0LJ4nwzJzOY2yTqiJ6G2JF9fWcA1f4T6kuAg%40mail.gmail.com.

--

Stephen Nimmo
Senior Specialist Solution Architect - Cloud Platform
Red Hat
ni...@redhat.com / M: 832.398.5867

[Stuart Douglas]

You can use both (h2 and mocks). You can also just use RestAssured, but it can cause problems if you then want to use those same tests as native/integration tests, as mocks/injection are not supported in these environments.

Maybe our inheritance based approach is the wrong way around, and the @QuarkusTest should inherit from the @NativeTest (or rather @QuarkusIntegrationTest if we go that route). It feels wrong at the moment to have the native test be the basis for all testing, but if it is just a general integration test then I think this makes a lot more sense. If you want to use injection/mocks you can just do it in the subclass, and it won't interfere with the integration test.

 

QuarkusIntegrationTest

Goal(s)

- Test the application network integration

- Test the native application

- BlackBox testing

Pre-requisities

- Quarkus JVM packaging

- Quarkus Native packaging

- %prod profile

Features

- Manage application (re)start between tests

- Provide TestContainers configuration

- Application Client pre-configuration (RestAssured, AMQP, Kafka)

DeveloperExperience

- Maven failsafe

Questions/Comments

- I have the feeling that we should clearly split the JVM and native mode testing but well at the end it's blackbox testing in the 2 sides...

From a black box PoV they should behave exactly the same, this makes it easy to use the same test for both. If you really want to limit based on environment we could add some kind of filter mechanism.

This will also let you run your tests against a docker image. At the moment if you are using mac you need a local graal install to test native image, as the docker image build creates a linux executable. If we have docker support these tests can be run against a docker image containing your native executable instead.

 

- How do you do testing if teh %test profile is used for unit testing, here you should use %prod but the application will try to connect to the actual production third parties (DB, Kafka cluster etc) I don't think its something to promote.

So it is built with prod but run with test, so runtime properties like DB urls will still work, but it must be the same type of db.

 

- Since Junit is providing the RegisterExtension, could we consider to configura and start a quarkus application progamtically (à la ShrinkWrap) rather than running the actual application jar?

@QuarkusUnitTest does this, it's a whole different type of testing. Maybe we should expose this to users as well, but you have problems with mixing the different test types. There is heaps of things we can do, but I don't think it is necessarily great to provide a lot of different ways to do things. My experience with the Shrinwrap approach is that application classes often have two many relationships, so you often have to pull in large parts of the application anyway, so it is much faster to just start the whole app once.

 

- Failsafe is not very popular compared to surefire, everybody will focus on mvn test and mvn package but not mvn verify

I agree. I think the reverse inheritance approach I mention above might fix this, as you are still running your integration tests in the unit test phase.

 

- Generallly speaking true integration tests are often skipped. A true production like environment is always time consuming to setup, the feedback loop is way to long and ends up delegated to a everyday CI build). IMy 2cts  it's ok for native image but not for basic HTTP testing.

Thanks for the feedback, I need to think about this some more.

Stuart

 

Daniel PETISME
daniel....@gmail.com

 

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2czdUqGsJbH6V_YGY_ExV7mqC5F_znG4Hi9iyOGs7zqH9Q%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CACHzTrbCruppD%3D0LJ4nwzJzOY2yTqiJ6G2JF9fWcA1f4T6kuAg%40mail.gmail.com.

[Andy Guibert]

Hi Stuart,

When we look at black-box testing an application, the runtime-specific bits that only Quarkus has become less apparent, which makes a runtime-agnostic library a bit more attractive instead of every runtime implementing their own way. This is what I've been working to achieve in a project called MicroShed Testing [1].

MicroShed has an extension for Quarkus already, among other runtimes like Liberty, Wildfly, and Payara. The ideal use case for MicroShed is block-box testing an application inside of a Docker container (as you describe). It currently has a good set of features for Quarkus integration including:

 - auto configuration of relational databases

 - auto configuration of MongoDB

 - auto configuration of Kafka and clients [2]

 - auto configuration of RestAssured or REST Client

Do you think it would be better to utilize MicroShed for black-box testing instead of creating a @QuarkusIntegrationTest anno? That way @QuarkusTest can focus on unit-testing capabilities that are tightly coupled with Quarkus, and MicroShed can focus on integrations between generic Java runtimes and their dependencies like DBs/Kafka/etc.

Side note: Currently MicroShed does not work with Quarkus 1.3 in w/ @QuarkusTest because of the classloading changes and I am working on fixing this, but if you are testing black-box in a Docker container (JVM or native) it works.

[1]: https://microshed.org/microshed-testing/

[2]: https://microshed.org/microshed-testing/features/KafkaMessaging.html

[Andrej Petras]

Hi Stuart,

I am using TestContainers for unit and integration tests. I did managed to start TestContainers for the @NativeImageTest (I disabled the NativeLaunchRunner). 

From my view this could help:

1. Rename the @NativeImageTest to @QuarkusIntegrationTest.

2. Disable or inject own implementation of the NativeLaunchRunner - In this way Quarkus project can still start native image for integration test with NativeLaunchRunner

Then it would be possible to use @QuarkusTestResource(DockerComposeTestResource.class) for unit and integration test.

With the current hack we can run the unit and integration tests with Testcontainers from IDEA and debug tests and application.

BR,

Andrej

[Stuart Douglas]

On Sat, 18 Apr 2020 at 02:11, Andy Guibert <andy.g...@gmail.com> wrote:

Hi Stuart,

When we look at black-box testing an application, the runtime-specific bits that only Quarkus has become less apparent, which makes a runtime-agnostic library a bit more attractive instead of every runtime implementing their own way. This is what I've been working to achieve in a project called MicroShed Testing [1].

MicroShed has an extension for Quarkus already, among other runtimes like Liberty, Wildfly, and Payara. The ideal use case for MicroShed is block-box testing an application inside of a Docker container (as you describe). It currently has a good set of features for Quarkus integration including:

 - auto configuration of relational databases

 - auto configuration of MongoDB

 - auto configuration of Kafka and clients [2]

 - auto configuration of RestAssured or REST Client

Do you think it would be better to utilize MicroShed for black-box testing instead of creating a @QuarkusIntegrationTest anno? That way @QuarkusTest can focus on unit-testing capabilities that are tightly coupled with Quarkus, and MicroShed can focus on integrations between generic Java runtimes and their dependencies like DBs/Kafka/etc.

So this would require docker to run the integration tests?

Stuart

 

Side note: Currently MicroShed does not work with Quarkus 1.3 in w/ @QuarkusTest because of the classloading changes and I am working on fixing this, but if you are testing black-box in a Docker container (JVM or native) it works.

[1]: https://microshed.org/microshed-testing/

[2]: https://microshed.org/microshed-testing/features/KafkaMessaging.html

On Tuesday, April 14, 2020 at 11:37:07 PM UTC-5, Stuart Douglas wrote:

Hi Everyone,

As I have mentioned previously I am planning on enhancing our testing capabilities, and to start with I am proposing a new approach to integration testing.

At the moment we have some limitations with regards to integration testing:

- We only provide integration testing for the actual production app for the native images

- These tests in our examples generally inherit from the corresponding unit test, this is confusing as unit tests support extra features that integration tests don't (namely injection), so this approach as the default leads to user confusion.

- We have no way of testing the actual JVM application that is generated

To solve these problems I propose we introduce @QuarkusIntegrationTest which will provide general integration test capabilities. In essence this will work almost exactly the same as our current native testing, except that it will test the generated application no matter what it is. If we are building a JVM application it will run the JVM app, if we are building a native application it works the same as our existing native tests, and so on. This will also allow us to test docker images, as we can use this to run the generated docker image and run tests against it (potentially we could even deploy to a kube pod and test in a cluster, but I am not sure how useful that will be).

From a practical point of view, this would mean the following:

- We discourage RestAssured based testing for @QuarkusTest. Instead we focus on unit style testing of components. Note that you can still test functionality without the transport layer by just injecting JAX-RS resources directly into the test. Is this something that we want to encourage as it provides a really simple way to test logic (while obviously not testing the transport layer)? Also if you really want to RESTAssured will still work fine, this is more just about what ends up in our examples/docs.

- We use @QuarkusIntegrationTest for all 'over the wire' tests, and run them in the integration tests phase. This means the actual application is being tested, but on the downside the application is not built in TEST mode so there is some inconsistency here (same as we already have for native). The actual target of this test will depend on the build output, native builds will test native, Java build will test the jar, Docker builds will run and test the docker image.

To look at a concrete example, the getting started example would have the changes in [1]. Basically the unit test now tests the endpoint without the transport layer, and the integration test runs over HTTP and would test both the JVM and native production applications.

There are some downsides/caveats to this proposal that need serious consideration:

- IDE support will not work as well for integration tests, as it requires a build to update the actual application that is being run. It will likely still be possible to have it launch from the IDE, but changes will not be automatically picked up (unless we can detect this and force a build before launch in this case).

- All integration style tests will be run in a separate process, so you would need explicit debugger support to launch the process in suspended mode, so debugging is slightly more involved.

- The tests would run in the verify phase, and need to follow the failsafe naming convention. This means you need to run 'mvn verify' rather than 'mvn test' and the tests are naming ITCase rather than TestCase. This is all 'standard' maven, however in general developers seem to be more experienced with surefire than failsafe.

In terms of implementation I have not given it a lot of thought yet, as I wanted to get feedback on the idea first. It will likely be some form of SPI that allows a test launcher to be generated in the target dir, which can then be invoked by the test runner to launch whatever the output of the build happens to be, but I don't really want to look into it in depth until we have all the requirements.

If you have any feedback I would love to hear it.

Stuart

[1] https://github.com/quarkusio/quarkus-quickstarts/commit/e1d46252ddc4b96686c3ce2fd11b73f8bb9476aa

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/27b56799-88e1-4381-b91d-bedbfbd73d93%40googlegroups.com.

[Stuart Douglas]

On Sat, 18 Apr 2020 at 05:26, Andrej Petras <andrej...@gmail.com> wrote:

Hi Stuart,

I am using TestContainers for unit and integration tests. I did managed to start TestContainers for the @NativeImageTest (I disabled the NativeLaunchRunner). 

From my view this could help:

1. Rename the @NativeImageTest to @QuarkusIntegrationTest.

2. Disable or inject own implementation of the NativeLaunchRunner - In this way Quarkus project can still start native image for integration test with NativeLaunchRunner

If we do this integration test change the way that the 'black box' application is launched would be pluggable, by default it would use one appropriate for whatever the package setting is.

Stuart

 

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/75be3401-adc1-424c-91bb-93416030f312%40googlegroups.com.

[Stuart Douglas]

So here is another possible option, we introduce @QuarkusIntegrationTest, and have it work in both the surefire and failsafe phases.

If you are running from the IDE, or from surefire it would work exactly the same as @QuarkusTest (so you can mix them), except that mocking and injection are banned. 

If you are running from failsafe it would test the built application, in a configurable manner. This could include:

- Running the JVM app directly

- Running the JVM app in a docker container

- Running the native app directly

- Running the native app in a docker container

- Testing the container produced by the build

- Testing the app that the build deployed to kube

- Testing an app that has been manually started

- anything else that we can think of.

This means that testing from the IDE will work as normal, and it is very clear the distinction between a 'black box' style test and a unit test.

This launch process will also need to be plugable, so it may be that we only provide some of the basic functionality, and other frameworks might add the reset (e.g. Microshed).

Unfortunately it will likely require some explicit maven config to make the same test run in both surefire and failsafe, and also some way of configuring exactly which type of test needs to be run, but these seem like solvable problems (we could also keep using the inheritance approach, with an empty integration test inheriting from the surefire test). 

Stuart

[Georgios Andrianakis]

I was thinking about this again today and I really think it's something we should add (and I volunteer to work on it).

What do y'all think?

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cx5sV8xoaj7z8mi1cCGgMrpC4zMBa_wOFA1V6EcnF7LLw%40mail.gmail.com.

[William Burke]

On Wed, Apr 15, 2020 at 12:37 AM Stuart Douglas <sdou...@redhat.com> wrote:

Hi Everyone,

As I have mentioned previously I am planning on enhancing our testing capabilities, and to start with I am proposing a new approach to integration testing.

At the moment we have some limitations with regards to integration testing:

- We only provide integration testing for the actual production app for the native images

- These tests in our examples generally inherit from the corresponding unit test, this is confusing as unit tests support extra features that integration tests don't (namely injection), so this approach as the default leads to user confusion.

- We have no way of testing the actual JVM application that is generated

To solve these problems I propose we introduce @QuarkusIntegrationTest which will provide general integration test capabilities. In essence this will work almost exactly the same as our current native testing, except that it will test the generated application no matter what it is. If we are building a JVM application it will run the JVM app, if we are building a native application it works the same as our existing native tests, and so on. This will also allow us to test docker images, as we can use this to run the generated docker image and run tests against it (potentially we could even deploy to a kube pod and test in a cluster, but I am not sure how useful that will be).

From a practical point of view, this would mean the following:

- We discourage RestAssured based testing for @QuarkusTest. Instead we focus on unit style testing of components. Note that you can still test functionality without the transport layer by just injecting JAX-RS resources directly into the test. Is this something that we want to encourage as it provides a really simple way to test logic (while obviously not testing the transport layer)? Also if you really want to RESTAssured will still work fine, this is more just about what ends up in our examples/docs.

- We use @QuarkusIntegrationTest for all 'over the wire' tests, and run them in the integration tests phase. This means the actual application is being tested, but on the downside the application is not built in TEST mode so there is some inconsistency here (same as we already have for native). The actual target of this test will depend on the build output, native builds will test native, Java build will test the jar, Docker builds will run and test the docker image.

To look at a concrete example, the getting started example would have the changes in [1]. Basically the unit test now tests the endpoint without the transport layer, and the integration test runs over HTTP and would test both the JVM and native production applications.

There are some downsides/caveats to this proposal that need serious consideration:

- IDE support will not work as well for integration tests, as it requires a build to update the actual application that is being run. It will likely still be possible to have it launch from the IDE, but changes will not be automatically picked up (unless we can detect this and force a build before launch in this case).

Lack of IDE support is a pretty big deal.  This full app end testing should be used sparingly as to do otherwise is a big detriment to productivity.  For JVM, I honestly don't see the benefits of it.  Native is another story because of how sensitive Graal is.  If you bring in @IntegrationTEst, there should be a big huge warning that the developer is no longer gonna be able to right-click-debug these tests.

[Stuart Douglas]

The big benefit for JVM would be that it lets us explicitly test the different output formats (uber-jar, fast-jar etc). If @NativeTest was @IntegrationTest then in our non-native CI runs we could test the various formats and make sure they all work as expected.

This also open up the possibility of explicitly testing docker images, so you could run the resulting image and execute the same set of tests against it (which would be really useful on mac, if you do a container-build on MacOS you end up with a linux executable that you can't test). You could even have kube based tests where you test the functionality while deployed in a cluster.

We also could have support for IDE support (just not debug in the IDE). It would be simple enough (if slightly hacky) to check if there are any .class files that are newer than the existing artifact, and if so rebuild before running (which will suck for native). It won't allow debugging, but we could add an easy way to start in suspended mode so you can manually attach the debugger.

Stuart

 

[Georgios Andrianakis]

It's been a long time coming, but I finally was able to carve out a little time to implement this.

You can find the pull request here: https://github.com/quarkusio/quarkus/pull/15009

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cwzuX_4tbKnwmUZrxKXdJ%2BDwtt2HQe2mPP1jGEgKP8v9w%40mail.gmail.com.