QuarkusMainTest & Mutable Jars: Questions and challenges

[Dominik Guhr]

Hey all,

I have some questions around the QuarkusMainTest and Quarkus(Main)IntegrationTest concepts, trying to create such tests for our Keycloak.X Distribution. 

Context:

In our quarkus based Keycloak distribution, we're using the mutable-jar approach right now. We have a picocli wrapped with a /kc.[sh|bat] script, which could be used like this:

./kc.sh build -> runs reaugmentation with provided commands / applies new config (needs server restart when config is buildtime property).

./kc.sh start -> starts the actual Keycloak.X quarkus application in prod mode

./kc.sh start-dev -> starts the same in devmode.

./kc.sh show-config  -> Print out the current configuration.

./kc.sh import -> Import data from a directory or a file.

...

Goals:

I want to provide the ability to create integrationtests for keycloak.x, which:

- have good IDE / debugging experience (Run everything in one jvm, no remote debugging needed if avoidable)

- rely on the great features of the Quarkus testing framework (e.g. using profiles, CDI, Mocking etc.)

What we tried so far:

- Using @QuarkusMainTest together with @Launch, to test command mode commands. But also, to test commands such as show-config or --help itself. So, in general: commands which are using plain picocli  without starting the actual quarkus application.

- Using an extension to run ./kc.sh start / start-dev non-blocking in the same jvm, and have a LaunchResult. 

Challenges:

1. Start of short-lived CLI-Commands:

For the @Launch command, it seems the KeycloakMainTestExtension.java class expects a quarkus instance to start/stop. For short-runnning commands, we just execute the picocli command and then do a system.exit(0). This atm leads to the test running, the IDE showing the logs, the tests being debuggable, but the actual test never getting executed, due to no startup/stop event.

2. Application startup using command mode:

We thought about writing an integration test which starts the application in prodmode (no http allowed, no ssl/tls setup given via config) and checks if the logs contain the respective error message as a starting point. 

like so: 

```

@Test

@Launch({ "start" })

public void testFailStartNoTls(LaunchResult result) {

Assertions.assertTrue(result.getOutput()

.contains("Key material not provided to setup HTTPS"));

}

```

This leads to Quarkus starting up, but "running forever" (until the timeout reached) and the actual test also never gets executed, because we have no launchresult with exit status. I think this is due to QuarkusMainTestExtension using BlockingStart when I understand correctly. So essentially, the challenge here would be to have a non-blocking start, and then some way to trigger the shutdown. 

In general, I think about having integration tests like this: 

1. run e.g. "./kc.sh build --http-enabled=true" via @Launch. (does the reaug under the hood). Check if new config is applied.

2. Run e.g. "./kc.sh show-config" or simply a "./kc.sh <command> --help" command (pure cli)

3. run "kc.sh start" via @Launch and try to make requests via restassured in the actual test. 

So, to ask actual questions after wrapping up the challenges I face atm: 

1. Are we doing something conceptually wrong here? It feels a bit like working against the framework. If so, please suggest a better way to do this. 

2. Any Hint, codewise or else, is greatly appreciated to come over this challenges. I'd also be happy to contribute, but am lacking the understanding of all the quarkus test extensions atm.

I know this is a bit vague, sorry for that. Will try to come back with more precise questions later when I had more time to investigate. I still wanted to post it here in hope of any feedback.

Regards,

Dominik

[Erin Schnabel]

The QuarkusMainTest approach is for the “run and done” case. Continuous testing will work, it will discover your main, but it assumes that @Launch will run a command to completion and exit. It is quite possible that this may not apply to your use case, as you do expect something to start and keep running. 

You might consider a custom Junit 5 Test extension.. they aren’t hard to write, and would give you better control over what is run (and how). 

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/0c95a6cf-f22f-4ce0-8d1d-1d4c798283ecn%40googlegroups.com.

--

Thanks,

Erin

 

------
Erin Schnabel <ebul...@redhat.com>

@ebullientworks

[Max Rydahl Andersen]

On 26 Oct 2021, 13:35 +0200, Erin Schnabel <ebul...@redhat.com>, wrote:

The QuarkusMainTest approach is for the “run and done” case. Continuous testing will work, it will discover your main, but it assumes that @Launch will run a command to completion and exit. It is quite possible that this may not apply to your use case, as you do expect something to start and keep running. 

You might consider a custom Junit 5 Test extension.. they aren’t hard to write, and would give you better control over what is run (and how). 

Well, shouldn’t normal integration test work here ? 

Also, seems like they have both use cases - short and long lived. 

And as you say Erin - the QuarkusMainTest is for short lived; not long lived.

Comments below:

On Tue, Oct 26, 2021 at 2:09 AM Dominik Guhr <dg...@redhat.com> wrote:

Hey all,

I have some questions around the QuarkusMainTest and Quarkus(Main)IntegrationTest concepts, trying to create such tests for our Keycloak.X Distribution. 

Context:
In our quarkus based Keycloak distribution, we're using the mutable-jar approach right now. We have a picocli wrapped with a /kc.[sh|bat] script, which could be used like this:

./kc.sh build -> runs reaugmentation with provided commands / applies new config (needs server restart when config is buildtime property).
./kc.sh start -> starts the actual Keycloak.X quarkus application in prod mode
./kc.sh start-dev -> starts the same in devmode.
./kc.sh show-config  -> Print out the current configuration.
./kc.sh import -> Import data from a directory or a file.
...

Goals:
I want to provide the ability to create integrationtests for keycloak.x, which:
- have good IDE / debugging experience (Run everything in one jvm, no remote debugging needed if avoidable)
- rely on the great features of the Quarkus testing framework (e.g. using profiles, CDI, Mocking etc.)

What we tried so far:
- Using @QuarkusMainTest together with @Launch, to test command mode commands. But also, to test commands such as show-config or --help itself. So, in general: commands which are using plain picocli  without starting the actual quarkus application.
- Using an extension to run ./kc.sh start / start-dev non-blocking in the same jvm, and have a LaunchResult. 

Challenges:

1. Start of short-lived CLI-Commands:
For the @Launch command, it seems the KeycloakMainTestExtension.java class expects a quarkus instance to start/stop. For short-runnning commands, we just execute the picocli command and then do a system.exit(0). This atm leads to the test running, the IDE showing the logs, the tests being debuggable, but the actual test never getting executed, due to no startup/stop event.

You should not call system.exit directly - ever :) 

Let Picocli/quarkus handle that. 

That enables much easier testing but also a good practice anyways 

2. Application startup using command mode:
We thought about writing an integration test which starts the application in prodmode (no http allowed, no ssl/tls setup given via config) and checks if the logs contain the respective error message as a starting point. 

like so: 
```
@Test
@Launch({ "start" })
public void testFailStartNoTls(LaunchResult result) {
Assertions.assertTrue(result.getOutput()
.contains("Key material not provided to setup HTTPS"));
}
```
This leads to Quarkus starting up, but "running forever" (until the timeout reached) and the actual test also never gets executed, because we have no launchresult with exit status. I think this is due to QuarkusMainTestExtension using BlockingStart when I understand correctly. So essentially, the challenge here would be to have a non-blocking start, and then some way to trigger the shutdown. 

Yeah - this feels like being more like a “traditional” quarkus integration test.

not actually sure if we have a way to pass in arguments to those integration tests… Erin/Stuart - do you know ? 

In general, I think about having integration tests like this: 
1. run e.g. "./kc.sh build --http-enabled=true" via @Launch. (does the reaug under the hood). Check if new config is applied.
2. Run e.g. "./kc.sh show-config" or simply a "./kc.sh <command> --help" command (pure cli)
3. run "kc.sh start" via @Launch and try to make requests via restassured in the actual test. 

Btw. You keep mentioning kc.sh here - those won’t be called/used in these scenarios.

If you truly want these called I would suggest looking into something like karate - but then its all external/black box testing. Which can be perfectly fine.

So, to ask actual questions after wrapping up the challenges I face atm: 
1. Are we doing something conceptually wrong here? It feels a bit like working against the framework. If so, please suggest a better way to do this. 

One is to just not call system.exit directly..ever ;) 

2. Any Hint, codewise or else, is greatly appreciated to come over this challenges. I'd also be happy to contribute, but am lacking the understanding of all the quarkus test extensions atm.

I know this is a bit vague, sorry for that. Will try to come back with more precise questions later when I had more time to investigate. I still wanted to post it here in hope of any feedback.

Regards,
Dominik
 --
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/0c95a6cf-f22f-4ce0-8d1d-1d4c798283ecn%40googlegroups.com.

--

Thanks,Erin ------Erin Schnabel <ebul...@redhat.com>@ebullientworks --

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJwjk9wb-XVonvFmHSrYDZxGHch%3DEBkds57fw7aDCNdEWxSbpA%40mail.gmail.com.

[Stuart Douglas]

On Tue, 26 Oct 2021 at 17:09, Dominik Guhr <dg...@redhat.com> wrote:

Hey all,

I have some questions around the QuarkusMainTest and Quarkus(Main)IntegrationTest concepts, trying to create such tests for our Keycloak.X Distribution. 

Context:

In our quarkus based Keycloak distribution, we're using the mutable-jar approach right now. We have a picocli wrapped with a /kc.[sh|bat] script, which could be used like this:

./kc.sh build -> runs reaugmentation with provided commands / applies new config (needs server restart when config is buildtime property).

./kc.sh start -> starts the actual Keycloak.X quarkus application in prod mode

./kc.sh start-dev -> starts the same in devmode.

./kc.sh show-config  -> Print out the current configuration.

./kc.sh import -> Import data from a directory or a file.

...

Goals:

I want to provide the ability to create integrationtests for keycloak.x, which:

- have good IDE / debugging experience (Run everything in one jvm, no remote debugging needed if avoidable)

- rely on the great features of the Quarkus testing framework (e.g. using profiles, CDI, Mocking etc.)

What we tried so far:

- Using @QuarkusMainTest together with @Launch, to test command mode commands. But also, to test commands such as show-config or --help itself. So, in general: commands which are using plain picocli  without starting the actual quarkus application.

- Using an extension to run ./kc.sh start / start-dev non-blocking in the same jvm, and have a LaunchResult. 

Challenges:

1. Start of short-lived CLI-Commands:

For the @Launch command, it seems the KeycloakMainTestExtension.java class expects a quarkus instance to start/stop. For short-runnning commands, we just execute the picocli command and then do a system.exit(0). This atm leads to the test running, the IDE showing the logs, the tests being debuggable, but the actual test never getting executed, due to no startup/stop event.

System.exit() will break everything, use Quarkus.asyncExit() to shutdown the application.

 

2. Application startup using command mode:

We thought about writing an integration test which starts the application in prodmode (no http allowed, no ssl/tls setup given via config) and checks if the logs contain the respective error message as a starting point. 

like so: 

```

@Test

@Launch({ "start" })

public void testFailStartNoTls(LaunchResult result) {

Assertions.assertTrue(result.getOutput()

.contains("Key material not provided to setup HTTPS"));

}

```

This leads to Quarkus starting up, but "running forever" (until the timeout reached) and the actual test also never gets executed, because we have no launchresult with exit status. I think this is due to QuarkusMainTestExtension using BlockingStart when I understand correctly. So essentially, the challenge here would be to have a non-blocking start, and then some way to trigger the shutdown. 

This sounds like you just want a normal @QuarkusTest, but with the ability to pass parameters, basically something like:  https://github.com/quarkusio/quarkus/compare/main...stuartwdouglas:startup-params?expand=1

 

If this is what you need I can add a test and get this in.

In general, I think about having integration tests like this: 

 

Firstly we will never touch kc.sh, we have no knowledge of it.

 

1. run e.g. "./kc.sh build --http-enabled=true" via @Launch. (does the reaug under the hood). Check if new config is applied.

This is a very specific type of test, you are likely going to need to use something custom to test this. You won't get the nice IDE integration but that is not really possible for this type of test.

 

2. Run e.g. "./kc.sh show-config" or simply a "./kc.sh <command> --help" command (pure cli)

Pure CLI should be taken care of by @Launch and QuarkusMainTest

 

3. run "kc.sh start" via @Launch and try to make requests via restassured in the actual test. 

The commit I linked above should cover this I think?

 

Stuart

So, to ask actual questions after wrapping up the challenges I face atm: 

1. Are we doing something conceptually wrong here? It feels a bit like working against the framework. If so, please suggest a better way to do this. 

2. Any Hint, codewise or else, is greatly appreciated to come over this challenges. I'd also be happy to contribute, but am lacking the understanding of all the quarkus test extensions atm.

I know this is a bit vague, sorry for that. Will try to come back with more precise questions later when I had more time to investigate. I still wanted to post it here in hope of any feedback.

Regards,

Dominik

--

[Max Rydahl Andersen]

On 27 Oct 2021, 00:08 +0200, sdou...@redhat.com, wrote:

This sounds like you just want a normal @QuarkusTest, but with the ability to pass parameters, basically something like:  github.com/quarkusio/quarkus/compare/main...stuartwdouglas:startup-params?expand=1

If this is what you need I can add a test and get this in.

This makes sense to enable IMO. It’s a missing feature to complete the cli testing story.

[Dominik Guhr]

Hey,

first of all thank you all for your suggestions and the support. That's great! Sadly, I can't manage to answer today, but please expect a better answer after I did some tests myself, e.g. with removing system.exit and so on :) 

Thanks again, and best regards 

Dominik

[Dominik Guhr]

Hi,
thanks again for the support first of all. Found some time to carefully read your responses.

1. For long-lived Tests the https://github.com/quarkusio/quarkus/compare/main...stuartwdouglas:startup-params?expand=1 would be a great to have in quarkus imo, not only for our use case. So yes Stuart, would be great to have it inside. :) 

2. For CLI-Only tests (and in general) I'll change the exit behaviour, and I think it'll work then. thanks for the hint! :) 

3. for kc.sh: don't mind that, it's our universe and if we really want to test this we'll create an extension I think. so it's more concise java - jar ... Quarkus-run.jar <params> then instead of the shellscript which runs the tests if I understand correctly.

Best regards,

Dominik

[Stuart Douglas]

It turned out to be slightly more complex than I expected, but this should do it:

https://github.com/quarkusio/quarkus/pull/21052

Stuart

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/567a614d-1406-4b06-b4dc-2531cd9b8ba8n%40googlegroups.com.

[Dominik Guhr]

@Stuart: Thanks for that, that'll definitely come in handy! :)

One other thing i'd like to ask about the short lived tests: 

- I removed the System.exit calls now and exchanged for Quarkus.asyncExit(); instead. Now, instead of immediately terminating, the test "runs forever", so it's essentially stuck it seems. Debugging it, it looks like the CompleteableFuture in StartupActionImpl::runMainClassBlocking() never gets completed. 

Before declaring this a bug or something, let's validate a basic assumption of mine:

- I know that we're only starting quarkus via Quarkus.run(...) when specific commands are used ("start", "start-dev", another thing is "build", which triggers the reaugmentation of the mutable jar under the hood). Running others (e.g. "--help", "show-config") won't execute Quarkus.run(), but instead provide other cli functionality such as showing the currently applied config. 

So, my question: Is it really intended that I call Quarkus.asyncExit() even when no Quarkus.run() is called beforehand? If so, I think there is a bug in the temrination logic of QuarkusMainTest. Interesting enough, it works for @QuarkusMainIntegrationTest, but this is not my preferred option, because it's not running in the same jvm then.

To help understand my intentions and our usage of quarkus, it may help to see some code, so here's my current WIP branch: [1], the test I am struggling with [2], our Main class: [3] and the package containing the other commands: [4].

Would be great if you could clarify :)

[1] https://github.com/DGuhr/keycloak/tree/quarkus-tests/quarkus

[2] https://github.com/DGuhr/keycloak/blob/quarkus-tests/quarkus/integration-tests/src/test/java/org/keycloak/it/cli/StartupTest.java

[3] https://github.com/DGuhr/keycloak/blob/quarkus-tests/quarkus/runtime/src/main/java/org/keycloak/quarkus/runtime/KeycloakMain.java

[4] https://github.com/DGuhr/keycloak/blob/quarkus-tests/quarkus/runtime/src/main/java/org/keycloak/quarkus/runtime/cli/command/ (especially abstractStartCommand which calls KeycloakMain.start() may be of interest)

Best regards,

Dominik

[Stuart Douglas]

So the issue here is that in the 'has parameters' case you don't actually start Quarkus, just run the CLI code directly, so the test blocks forever waiting for Quarkus to start. I will need to update the test runner to handle this case.

Stuart

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/c9678769-3e34-4be9-b27b-c899cf12c18fn%40googlegroups.com.

[Stuart Douglas]

https://github.com/quarkusio/quarkus/pull/21078 should help.

Stuart

[Pedro Igor]

Thanks, Stuart.

I'm looking at your changes. While I'm building the project :) let me ask you if it makes sense to change the `io.quarkus.deployment.steps.MainClassBuildStep#generateMainForQuarkusApplication` to read a configuration option (e.g.: quarkus.package.run-main-class=true or similar) and generate code to run the main-class directly instead of generating code to run the main class through Quarkus.run.

In our case, the main method of the main-class is executed prior to actually running the app and we run the quarkus app internally depending on the command you choose.

[Pedro Igor]

I did some changes here [1] and they now work as expected so that our main class is firstly executed and then the application is bootstrapped.

Your changes also help but we are not able to assert errors properly because our exception handling logic is not executed (they are part of our main class). And we are still bootstrapping the application prior to running our main class with all validations we need (similar to how users are going to use the server). Ideally, we should bootstrap extensions (such as web, hibernate, etc) only when executing specific commands.

Please, let me know if these changes make sense Or if we are going too far as a Quarkus CLI Application :)

[1] https://github.com/pedroigor/quarkus/commit/70c0f0857f59c8f028133bdec3e7b85e6f56a5d3

[Pedro Igor]

As an update, looks like v2.4.1 is working as expected. My main-class setup was wrong.

The main class is indeed called prior to bootstrapping the application.

[Erin Schnabel]

Glad to hear it!

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/70291280-e061-47db-a71c-130cc6aa74a2n%40googlegroups.com.