Hot reload for testing?

[Benjamin Thomas]

Hello,

I'm intrigued by Quarkus and have a noobish question I hope you'll forgive me ;)

A little bit of background I've been primarily a ruby developer for quite a few years now, dabbled in golang and some other languages and dipping my toes into java land for about a year.

I'm at a point where I realise I very much enjoy static typing. I like the design of the java language, and all the tooling around it. But I have always been frustrated by the slowness of startup times. If my dev env isn't snappy it really bothers me.

Anyways here comes along Quarkus and indeed the hot reload feature looks exactly what I was looking for.

I'm wondering though, what's the story for testing? I went trough the getting started guide and didn't find anything addressing the issue of slow test feedback loop (which bothers me too ^^)

Are you planning to address this? Is this an non issue for java devs?

For clarity, I'm also a rails delevoper. Now I wouldn't say that I like the framework but it is productive. Anything dev or test related transparently goes through an application loader named "spring". This loader caches the loaded state of the application, thusly vastly improving the response times of any test or dev related operations.

At the end of the getting started guide here: https://quarkus.io/guides/getting-started, you implement a GreetingResourceTest class and run the tests through mvn test ; that's pretty slow.

I'm a little confused as from a developer's productivity perspective, I'd be tempted to go straight for an integration test via httpie for example and go through the dev hot reload mecanism instead to have a fast feed back loop and a better dev experience.

Am I misunderstanding something?

[Stuart Douglas]

You should be able to just run them directly from the IDE.

We have talked about continuous testing where we run the tests after every hot reload which we can probably make pretty fast, however it has never really gone anywhere as it is fairly complex I am not really sure if this is something people actually want.

Stuart

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/30f193ee-9e62-46e8-a138-5e9f336ba550n%40googlegroups.com.

[Jason Greene]

Hi Banjamin, 

You  can see what Stuart describes in action, in a past Quarkus Insights Episode:

https://youtu.be/OCPFdpvL1Q0?t=900

(15 mins in in case the link doesn’t work)

Basically right click and run test in Intellij IDEA (Although use any IDEA you like)

Lots of other good stuff in that episode!

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cxjPsB9Qy-VaN2_181cYs%3D%3DiBgPAbkUN061%2BNwt3u9hGw%40mail.gmail.com.

[Eduardo Ramirez Ronco]

I believe Benjamin might be referring to something like Infinitest (https://infinitest.github.io/).

Infinistest was a plugin for Eclipse and IntelliJ, unfortunately unmaintained, that ran tests continuously and when the application code changed it tracked which classes were changed to identify which tests needed to be rerun.

Having the Dev Mode do that and give feedback through the IDE plugin would be a killer feature for TDD practitioners.

Eduardo Ramírez ronco

Services Content Architect

Red Hat Global Learning Service

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/2DD2C96B-9722-426C-872B-DA6AAF05B3B2%40redhat.com.

[Stuart Douglas]

On Thu, 18 Feb 2021 at 22:27, Eduardo Ramirez Ronco <eram...@redhat.com> wrote:

I believe Benjamin might be referring to something like Infinitest (https://infinitest.github.io/).

Infinistest was a plugin for Eclipse and IntelliJ, unfortunately unmaintained, that ran tests continuously and when the application code changed it tracked which classes were changed to identify which tests needed to be rerun.

I don't know if you can really track which tests need to be re-run based on class changes. We could write a custom JUnit Engine, fire it up in dev mode, and run the test suite on change. This would all be in an isolated CL, on a different port, and should work fine.

If lots of people are genuinely interested in this I can do it, I'm just not sure if it is actually worth doing, and there would be a lot of complexity around things like multi module projects.

Stuart

 

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAFif861U%2B3mQuFPvC%2BQg16U54x4AMEPctw1Va1R-WGo-chwgHg%40mail.gmail.com.

[Arek R.]

I'm not an experienced quarkus dev but IMHO GreetingResourceTest is an example of integration test or even contract/e2e test thus normally you I don't want to execute it like every time. They're slower by design and I'm running them during the final phase of the development when business logic is mostly done or while building the package, also running another database instance via TestContainers.

For every day work I'm using pure unit tests or SoapUI  as a rest client then there's no need to restart the server at all, debugger is working etc

[Loïc MATHIEU]

> If lots of people are genuinely interested in this I can do it, I'm just not sure if it is actually worth doing, and there would be a lot of complexity around things like multi module projects.

@Stuart Douglas we already discussed this functionality some time ago and I do think it could be a very good addition to Quarkus. At the time, others agree but we also agree this would not be easy to implement.

More and more languages offer this kind functionality, we can start with something very limited in terms of functionality (no module supported) and see if it gains attraction.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/f62e7f25-eb12-407e-928f-0c227e90357en%40googlegroups.com.

[Georgios Andrianakis]

On Thu, Feb 18, 2021 at 6:53 PM Loïc MATHIEU <loik...@gmail.com> wrote:

> If lots of people are genuinely interested in this I can do it, I'm just not sure if it is actually worth doing, and there would be a lot of complexity around things like multi module projects.

@Stuart Douglas we already discussed this functionality some time ago and I do think it could be a very good addition to Quarkus. At the time, others agree but we also agree this would not be easy to implement.

More and more languages offer this kind functionality, we can start with something very limited in terms of functionality (no module supported) and see if it gains attraction.

So basically you just want to be able to run the entire testsuite right, not have a way to detect which tests are associated with the source code changes (I don't see how that would even be possible), right?

Le jeu. 18 févr. 2021 à 13:56, Arek R. <arkadiusz...@gmail.com> a écrit :

I'm not an experienced quarkus dev but IMHO GreetingResourceTest is an example of integration test or even contract/e2e test thus normally you I don't want to execute it like every time. They're slower by design and I'm running them during the final phase of the development when business logic is mostly done or while building the package, also running another database instance via TestContainers.

For every day work I'm using pure unit tests or SoapUI  as a rest client then there's no need to restart the server at all, debugger is working etc

środa, 17 lutego 2021 o 19:25:51 UTC+1 benjamin....@gmail.com napisał(a):

Hello,

I'm intrigued by Quarkus and have a noobish question I hope you'll forgive me ;)

A little bit of background I've been primarily a ruby developer for quite a few years now, dabbled in golang and some other languages and dipping my toes into java land for about a year.

I'm at a point where I realise I very much enjoy static typing. I like the design of the java language, and all the tooling around it. But I have always been frustrated by the slowness of startup times. If my dev env isn't snappy it really bothers me.

Anyways here comes along Quarkus and indeed the hot reload feature looks exactly what I was looking for.

I'm wondering though, what's the story for testing? I went trough the getting started guide and didn't find anything addressing the issue of slow test feedback loop (which bothers me too ^^)

Are you planning to address this? Is this an non issue for java devs?

For clarity, I'm also a rails delevoper. Now I wouldn't say that I like the framework but it is productive. Anything dev or test related transparently goes through an application loader named "spring". This loader caches the loaded state of the application, thusly vastly improving the response times of any test or dev related operations.

At the end of the getting started guide here: https://quarkus.io/guides/getting-started, you implement a GreetingResourceTest class and run the tests through mvn test ; that's pretty slow.

I'm a little confused as from a developer's productivity perspective, I'd be tempted to go straight for an integration test via httpie for example and go through the dev hot reload mecanism instead to have a fast feed back loop and a better dev experience.

Am I misunderstanding something?

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/f62e7f25-eb12-407e-928f-0c227e90357en%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJLxjVE6tyAtx4tvZ6memaFXPkhTjUO5Boxqr2G0UJ50GPXEsA%40mail.gmail.com.

[Jason Greene]

On Feb 18, 2021, at 10:55 AM, Georgios Andrianakis <gand...@redhat.com> wrote:

MATHIEU <loik...@gmail.com> wrote:

> If lots of people are genuinely interested in this I can do it, I'm just not sure if it is actually worth doing, and there would be a lot of complexity around things like multi module projects.

@Stuart Douglas we already discussed this functionality some time ago and I do think it could be a very good addition to Quarkus. At the time, others agree but we also agree this would not be easy to implement.

More and more languages offer this kind functionality, we can start with something very limited in terms of functionality (no module supported) and see if it gains attraction.

So basically you just want to be able to run the entire testsuite right, not have a way to detect which tests are associated with the source code changes (I don't see how that would even be possible)

Wouldn’t automatic running tests as you change things be super annoying?  Like you are trying to type or read a log and your focus changes and some random test get stuck. Or you are debugging and the tests start running on you interfering.

[Loïc MATHIEU]

> So basically you just want to be able to run the entire testsuite right, not have a way to detect which tests are associated with the source code changes (I don't see how that would even be possible), right?

Right.

> Wouldn’t automatic running tests as you change things be super annoying?  Like you are trying to type or read a log and your focus changes and some random test get stuck. Or you are debugging and the tests start running on you interfering.

The idea is to be able to launch 'mvn quarkus:test' on a terminal, forget it, code, get back to it and see if it's still green.
This should not be done on running apps, or when running 'quarkus:dev'.
As I said, a lot of languages / framework do this (on JS side mainly).

[Jason Greene]

I guess you need some sort of metadata to describe the association between classes and tests and then maybe some hotkey to disable it globally when you like. Although it would be very reliant on naming conventions, and easy to get something to fall through the gaps (e.g. you change a dependent class and it doesn’t run)

[Georgios Andrianakis]

Exactly, I don't see myself ever using such a thing...

[Arek R.]

The root problem was that development loop was slow. It was slow because actually the author run integration tests using default test profile where we've kind of contract tests against black box, database is created on each execution, hot replacement doesn't work etc etc. This is not the way development should be done. Test profile is for tests and dev is for dev. So one should use a rest client or there's should a way to run integration test against running dev server

[Jason Greene]

On Feb 18, 2021, at 11:11 AM, Loïc MATHIEU <loik...@gmail.com> wrote:



> So basically you just want to be able to run the entire testsuite right, not have a way to detect which tests are associated with the source code changes (I don't see how that would even be possible), right?

Right.

> Wouldn’t automatic running tests as you change things be super annoying?  Like you are trying to type or read a log and your focus changes and some random test get stuck. Or you are debugging and the tests start running on you interfering.

The idea is to be able to launch 'mvn quarkus:test' on a terminal, forget it, code, get back to it and see if it's still green.
This should not be done on running apps, or when running 'quarkus:dev'.
As I said, a lot of languages / framework do this (on JS side mainly).

Is there a good example I can check out of the ideal automated testing workflow in the JS world? 

[Manyanda Chitimbo]

@jason....@redhat.com 

In the JS/TS world, I've used both Jest and Mocha in watch mode.

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJMo4%2Be6UAHrnZWdZ6yHTYjKqqEfY3qC-0bB__PXpR8Y%3DER5VA%40mail.gmail.com.

--

Manyanda Chitimbo.

[Georgios Andrianakis]

What is the user experience with those? Does it work well?

[Benjamin Thomas]

(sending again I forgot to reply all sorry)

Hi Douglas,

Thanks for the feedback. I did run the test through the IDE (intellij) and found things sluggish, maybe something's wrong with my setup or I'm misunderstanding something but basically I've got 8s of building and 2s of actual test execution.

This has always been my experience with java though, and granted my computer isn't that new. But then again how much power would you need for dumping a hello world endpoint?

This is in contrast to the dev server, where I incur the same startup cost once (roughly 10s), but then I don't have to wait around anymore and I get relatively quick feedback.

You received this message because you are subscribed to a topic in the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/quarkus-dev/qB3iW0ciLqI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cxjPsB9Qy-VaN2_181cYs%3D%3DiBgPAbkUN061%2BNwt3u9hGw%40mail.gmail.com.

--

Benjamin Thomas

[Benjamin Thomas]

Thanks Jason, I'll look into it further. Channel looks interesting :)

--

Benjamin Thomas

[Benjamin Thomas]

I wasn't really talking about this, but here is an example of what is being mentioned (it's ruby though) : https://github.com/guard/guard-minitest

Basically, a file watcher detects when test files change, then the said test file's tests are then executed. Pretty nice to have but I wouldn't say it's a must have.

In javascript land Cypress works like that too (in a much more natural way, no setup required)

--
You received this message because you are subscribed to a topic in the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/quarkus-dev/qB3iW0ciLqI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJMo4%2Be6UAHrnZWdZ6yHTYjKqqEfY3qC-0bB__PXpR8Y%3DER5VA%40mail.gmail.com.

--

Benjamin Thomas

[Manyanda Chitimbo]

I'd not complain about the experience, you get the mode by simply setting the flag and you are good to go. 

Files changes are picked by in a matter of milliseconds and worked even for larger projects. The runner will only execute the tests that are affected by corresponding file changes, if a test is not concerned by a change it won't be rerun: Jest provides the `watchAll` option to rerun all tests when it detects file changes. 

It is a good mode especially for teams practicing TDD (which has been my case), leaving the re-running the tests to the test runner and the developer entirely focusing on what matters most.  

--

Manyanda Chitimbo.

[Michał Szynkiewicz]

Back in the day, I played with scala and sbt revolver allowed doing sth like this. And for a small project the experience was really good.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANwCmXJEuN47EJ2s4r_%2BQSZfK3GXbNCGgVUk31AVHVCm1Onm8A%40mail.gmail.com.

[Benjamin Thomas]

Yes that's the kind of slowness I'm talking about. As soon as you go over a second, it starts to feel painful IMO.

I don't understand the details so I'll probably make a fool of myself but I'll give you a little more input.

It'd feel natural to have a "quarkus:test" command, analogous to "quarkus:dev" that would load the app and keep it in a running state, dedicated for testing. And that's where my testing stuff would call to. That way this behavior would be opt-in and you wouldn't change a thing in regards to CI pipeline workflows, etc.

So something like:

• Step 1: run ./mvnw quarkus:test in a separate terminal
• Step 2 : run a dedicated wrapper script that would call into the loaded test env ./bin/test-runner ./src/test/java/invalid/bt/GreetingResourceTest.java

If you have that, then implementing a global test runner is trivial since the file watcher would execute step 2 upon file change basically.

Conceptually, I'd say those are the same problem :

1. I changed code : I want to see result in my browser ASAP
2. I changed code : I want to see result in my test output ASAP

And I understand you've solved problem #1.

Maybe food for thought, I could point you out to another project, still rails related but written in golang: https://github.com/burke/zeus

"Zeus preloads your Rails app so that your normal development tasks such as console, server, generate, and specs/tests take less than one second.

...More generally, Zeus is a language-agnostic application checkpointer for non-multithreaded applications. Currently only ruby is targeted, but explicit support for other languages is possible."

Basically, this tool loads your app once (in the background), then uses forking upon each invocation to it. That way, there is no need to wait after the initial boot.

It's very similar in concept to the "spring" process mentioned at the start of the thread, but I'm bringing it here since I remembered reading the "language-agnostic" concept a while back so maybe it could provide a starting point or inspiration.

I understand I'm comparing apples to oranges, but maybe some kind of forking could be useful. Forking is a bad experience for Windows users though, but with WSL being a thing now, maybe that would be ok.

Le jeu. 18 févr. 2021 à 23:05, Stuart Douglas <sdou...@redhat.com> a écrit :

On Fri, 19 Feb 2021 at 06:31, Benjamin Thomas <benjamin....@gmail.com> wrote:

Hi Douglas,

Thanks for the feedback. I did run the test through the IDE (intellij) and found things sluggish, maybe something's wrong with my setup or I'm misunderstanding something but basically I've got 8s of building and 2s of actual test execution.

On my machine the Getting Started test is about 4s total, running the hibernate one with my new DevServices based database is around 9s, including starting postgres as a docker container.

We could probably get a fast feedback loop here, but it needs some thought. If you have a larger test suite and are only interested in one test a 'continuous testing' approach that runs all the tests will still be slower than

just running the one you are interested in inside your IDE.

One fairly radical idea is to have @QuarkusTest check if a continuous testing process for the current app is already running locally, and if so send it a message to delegate the test execution to the existing JVM. This would likely

have a lot of pitfalls though (e.g. debug won't work as you expect). I doubt it is worth the complexity.

Overall I am not super convinced that this is worth it, but it is probably worth doing up a proof of concept just to get a feel for if it would actually be useful.

Stuart

 

This has always been my experience with java though, and granted my computer isn't that new. But then again how much power would you need for dumping a hello world endpoint?

This is in contrast to the dev server, where I incur the same startup cost once (roughly 10s), but then I don't have to wait around anymore and I get relatively quick feedback.

Le mer. 17 févr. 2021 à 22:16, Stuart Douglas <sdou...@redhat.com> a écrit :

You received this message because you are subscribed to a topic in the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/quarkus-dev/qB3iW0ciLqI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cxjPsB9Qy-VaN2_181cYs%3D%3DiBgPAbkUN061%2BNwt3u9hGw%40mail.gmail.com.

--

Benjamin Thomas

--

Benjamin Thomas

[Jason Greene]

Thanks for this feedback! We will see what we can come up with.

On Feb 19, 2021, at 1:02 AM, Benjamin Thomas <benjamin....@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CABSP4jnmTRLA%3DmHRVPskOvdZO2zC6SUjgy2SyG0cAfPx0FfoZg%40mail.gmail.com.

[Max Rydahl Andersen]

just to add to this - I got similar feedback from times where I presented Quarkus (and jbang)
to people working in javascript land.

They had moved from java to javascript and they really missed the feature of having continuous testing running;
especially one that was not noisy. i.e. don't show tons of logs - just a indicator of which test is running and then
mark it green or red.

They referenced things like jestjs an mochas as an example of it.

Mind you, I don't think you need a massive dependency graph detection algorithm on which tests affects what.

Just a dumb bytecode scan of class refs for the first layer of references classes or even just a source code scan for references to the class.

Could also just to start use junit category/pattern matching to state which tests you care about having triggered.

I'm pretty sure if we just enables the mechanism of auto-running tests based on changes detected and make the
reporting clean and simple then we will see many be happy.

You can see example on how the "UI looks" like here: https://github.com/facebook/jest/pull/2362

Much cleaner and informative than a "classic" maven/gradle test run imo.

/max

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJMo4%2BfiTj4ON1DzJLLtZ5KjORntjqJv24g_QKsxR57bGRfdMg%40mail.gmail.com.

/max
https://xam.dk/about

[Stuart Douglas]

On Fri, 19 Feb 2021, 11:08 pm Max Rydahl Andersen, <mand...@redhat.com> wrote:

just to add to this - I got similar feedback from times where I presented Quarkus (and jbang)
to people working in javascript land.

They had moved from java to javascript and they really missed the feature of having continuous testing running;
especially one that was not noisy. i.e. don't show tons of logs - just a indicator of which test is running and then
mark it green or red.

They referenced things like jestjs an mochas as an example of it.

Mind you, I don't think you need a massive dependency graph detection algorithm on which tests affects what.

Just a dumb bytecode scan of class refs for the first layer of references classes or even just a source code scan for references to the class.

This won't work, most of our tests use http, so no bytecode dependencies.

I have an idea that should give close to 100% accuracy, if it works out. I should have a proof of concept ready soonish.

Stuart

[Max Andersen]

can't we based on URL paths figure out their end point and reverse from there ? :)

/max

On 20 Feb 2021, 07:51 +0100, Stuart Douglas <sdou...@redhat.com>, wrote:

On Fri, 19 Feb 2021, 11:08 pm Max Rydahl Andersen, <mand...@redhat.com> wrote:

just to add to this - I got similar feedback from times where I presented Quarkus (and jbang)
to people working in javascript land.

They had moved from java to javascript and they really missed the feature of having continuous testing running;
especially one that was not noisy. i.e. don't show tons of logs - just a indicator of which test is running and then
mark it green or red.

They referenced things like jestjs an mochas as an example of it.

Mind you, I don't think you need a massive dependency graph detection algorithm on which tests affects what.

Just a dumb bytecode scan of class refs for the first layer of references classes or even just a source code scan for references to the class.

This won't work, most of our tests use http, so no bytecode dependencies.

I have an idea that should give close to 100% accuracy, if it works out. I should have a proof of concept ready soonish.

Stuart

Could also just to start use junit category/pattern matching to state which tests you care about having triggered.

I'm pretty sure if we just enables the mechanism of auto-running tests based on changes detected and make the
reporting clean and simple then we will see many be happy.

You can see example on how the "UI looks" like here: https://github.com/facebook/jest/pull/2362
<>

Much cleaner and informative than a "classic" maven/gradle test run imo.

/max

[Benjamin Thomas]

I've done a bit of digging and managed to come up with this: https://github.com/benjamin-thomas/fast-java-feedback-loop-demo

What do you think? Not sure how it would scale but I'm pretty happy with this so far, and I can't see huge drawbacks to doing this.

I'd love to get feedback from you experienced Java developers.

--

Benjamin Thomas

[Max Andersen]

Hi Benjamin,

I looked at your repo but I'm missing what this does that quarkus:Dev doesn't ?


Can you elaborate?

/max

[Jason Greene]

I am pretty sure Stuart’s approach will be foolproof and robust, but I don’t want to spoil his eminent victory with details :)

I think if for some reason we can’t go with that (I think it’s like Stuart’s solution will work great), a derivative model like you mention + hinting metadata (@TestUses(FooEntity)) would be a decent fallback option.

I am pretty confident we can get a nice watch console solution going soon, call it phase 1. 

The next tricky part though that I think is important for devs that come from Java, is getting the IDE integration components working.  Eduardo (thanks Eduardo!) forwarded one project’s IDE and it looks they went the custom route. Ideally we would get the existing infrastructure functioning via some special remote test execution engine and perhaps some hinting to the IDE. I think Stuart mentioned in this thread there is some issues with debugging not going to the right process, so that’s what I mean where we would need to work with the IDE venders or do some special hacks (perhaps autogen run/debug/launch configs). In the meantime we could be sure the console prints out source links with line numbers so that IDE hot linking works.

On Feb 20, 2021, at 3:09 AM, Max Andersen <mand...@redhat.com> wrote:



can't we based on URL paths figure out their end point and reverse from there ? :)

/max

On 20 Feb 2021, 07:51 +0100, Stuart Douglas <sdou...@redhat.com>, wrote:

On Fri, 19 Feb 2021, 11:08 pm Max Rydahl Andersen, <mand...@redhat.com> wrote:

just to add to this - I got similar feedback from times where I presented Quarkus (and jbang)
to people working in javascript land.

They had moved from java to javascript and they really missed the feature of having continuous testing running;
especially one that was not noisy. i.e. don't show tons of logs - just a indicator of which test is running and then
mark it green or red.

They referenced things like jestjs an mochas as an example of it.

Mind you, I don't think you need a massive dependency graph detection algorithm on which tests affects what.

Just a dumb bytecode scan of class refs for the first layer of references classes or even just a source code scan for references to the class.

This won't work, most of our tests use http, so no bytecode dependencies.

I have an idea that should give close to 100% accuracy, if it works out. I should have a proof of concept ready soonish.

Stuart

Could also just to start use junit category/pattern matching to state which tests you care about having triggered.

I'm pretty sure if we just enables the mechanism of auto-running tests based on changes detected and make the
reporting clean and simple then we will see many be happy.

You can see example on how the "UI looks" like here: https://github.com/facebook/jest/pull/2362
<>

Much cleaner and informative than a "classic" maven/gradle test run imo.

/max

[Benjamin Thomas]

Hi Max,

quarkus:dev looks great the way it is. Then again I’ve only had a brief look at the framework but it’s the first time I’ve seen such a feature from a Java framework.

I was then surprised to conclude that the testing experience was still comparatively slow thus my post.

IMO a framework is that much better if it provides the tooling around a great dev experience.

My demo displays a client/server architecture that enables a very fast feedback loop for testing purposes (that’s what’s interesting in our case).

It doesn’t address the slowness of the integration test mentioned at the start of the thread since I’m not sure of the inner workings of Quarkus but it could maybe be useful for more isolated test areas.

I’m just throwing it out there, if anything I found a cool way to run and test isolated Java code very quickly while developing.

If such a client/server dev tool were supported by a framework, I would gladly use it considering the productivity benefits(all those seconds add up).

In short my aim was to point towards an interesting concept in a practical way.

I’m not sure how applicable it is to Quarkus’ design though

--

Benjamin Thomas

[Jason Greene]

Thanks for sharing Benjamin. I think we can get the Quarkus experience very close to what you are looking for. I am glad you have a local solution though in the meantime.

We would love your feedback trying it out when it lands. 

On Feb 20, 2021, at 11:48 AM, Benjamin Thomas <benjamin....@gmail.com> wrote:



[Benjamin Thomas]

For sure, I’ll be on the lookout ;)

Thanks Jason

--

Benjamin Thomas

[Stuart Douglas]

So I have started looking at this, and the big question is how it integrates with dev mode. I think the expectation would be that a developer would want both dev mode and continuous testing to be active at once, so we need to be careful they don't interfere with each other.

My thinking at this point is that it probably makes the most amount of sense to include all this logic directly in the dev mode process. This will make things like integrating it with the DevUI much easier, but it does have some challenges. In particular how should we handle console output? Should we just output the test results directly to the console

after each reload, or do we want to do something else (e.g. only expose them in DevUI).

We could have a mvn quarkus:test command, so you can have all the test results in a dedicated console (and potentially allow interactive input), but if the core logic is in the dev mode process then this would have to attach to a running dev mode process somehow, and won't work without it (not sure if this is a big deal or not, we can add reconnect logic to make sure it can survive restarts, and if no dev mode process is present we can just wait for one to start). This does mean that you can't just do mvn quarkus:test though, you also need mvn quarkus:dev.

What are peoples thoughts around this?

Stuart

[George Gastaldi]

The idea looks appetizing. I like your second proposal, but I'd call it `mvn quarkus:dev-test` that does quarkus:dev and run the tests when the Hot-reload is triggered (or through a property that enables this feature when using quarkus:dev).

Obviously this is easier said than done but I think it's a good feature.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cwCQTkmzqPgdJMH3JJzL3U0ydpYx%2B1-5mkxDb_5j7P2wA%40mail.gmail.com.

[Georgios Andrianakis]

On Mon, Feb 22, 2021 at 4:28 AM George Gastaldi <ggas...@redhat.com> wrote:

The idea looks appetizing. I like your second proposal, but I'd call it `mvn quarkus:dev-test` that does quarkus:dev and run the tests when the Hot-reload is triggered (or through a property that enables this feature when using quarkus:dev).

Obviously this is easier said than done but I think it's a good feature.

Em dom., 21 de fev. de 2021 22:43, Stuart Douglas <sdou...@redhat.com> escreveu:

So I have started looking at this, and the big question is how it integrates with dev mode. I think the expectation would be that a developer would want both dev mode and continuous testing to be active at once, so we need to be careful they don't interfere with each other.

My thinking at this point is that it probably makes the most amount of sense to include all this logic directly in the dev mode process. This will make things like integrating it with the DevUI much easier, but it does have some challenges. In particular how should we handle console output? Should we just output the test results directly to the console

after each reload, or do we want to do something else (e.g. only expose them in DevUI).

We could have a mvn quarkus:test command, so you can have all the test results in a dedicated console (and potentially allow interactive input), but if the core logic is in the dev mode process then this would have to attach to a running dev mode process somehow, and won't work without it (not sure if this is a big deal or not, we can add reconnect logic to make sure it can survive restarts, and if no dev mode process is present we can just wait for one to start). This does mean that you can't just do mvn quarkus:test though, you also need mvn quarkus:dev.

What are peoples thoughts around this?

Sounds to me like the second option would be cleaner but perhaps a little more cumbersome.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJgecQrFe1B5c1dLNTtcKo3ujZ8jWxY5EuWsTZ7FDHjd1g%40mail.gmail.com.

[Alexey Loubyansky]

On Mon, Feb 22, 2021 at 2:43 AM Stuart Douglas <sdou...@redhat.com> wrote:

So I have started looking at this, and the big question is how it integrates with dev mode. I think the expectation would be that a developer would want both dev mode and continuous testing to be active at once, so we need to be careful they don't interfere with each other.

My thinking at this point is that it probably makes the most amount of sense to include all this logic directly in the dev mode process.

It feels like in perspective this is what we would want. As I imagine it, even if it is to be done as a kind of separate mode there will be a lot of common base parts.

 

This will make things like integrating it with the DevUI much easier, but it does have some challenges. In particular how should we handle console output? Should we just output the test results directly to the console

after each reload, or do we want to do something else (e.g. only expose them in DevUI).

IIUC, we could start with exposing them. Then the test logs would still be on the disk. We could simply provide the links to them for a start.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cwCQTkmzqPgdJMH3JJzL3U0ydpYx%2B1-5mkxDb_5j7P2wA%40mail.gmail.com.

[Ioannis Canellos]

Stuart Douglas writes:

> So I have started looking at this, and the big question is how it
> integrates with dev mode. I think the expectation would be that a developer
> would want both dev mode and continuous testing to be active at once, so we
> need to be careful they don't interfere with each other.
>
> My thinking at this point is that it probably makes the most amount of
> sense to include all this logic directly in the dev mode process. This will
> make things like integrating it with the DevUI much easier, but it does
> have some challenges. In particular how should we handle console output?
> Should we just output the test results directly to the console
> after each reload, or do we want to do something else (e.g. only expose
> them in DevUI).

I think that out of the box, the behavior should remain as is, unless we use a flag that enables that.

>
> We could have a mvn quarkus:test command, so you can have all the test
> results in a dedicated console (and potentially allow interactive input),
> but if the core logic is in the dev mode process then this would have to
> attach to a running dev mode process somehow, and won't work without it
> (not sure if this is a big deal or not, we can add reconnect logic to make
> sure it can survive restarts, and if no dev mode process is present we can
> just wait for one to start). This does mean that you can't just do mvn
> quarkus:test though, you also need mvn quarkus:dev.

The idea of using `quarkus:test` is nice, though the name might be bit misleading.

Could we provide an option to start both `quarkus:dev` and `quarkus:test` in a singe command?

>>>>> Much cleaner and informative than a "classic" maven/gradle test run imo.
>>>>>
>>>>> /max
>>>>>
>>>>> Thanks for this feedback! We will see what we can come up with.
>>>>>
>>>>> On Feb 19, 2021, at 1:02 AM, Benjamin Thomas <
>>>>> benjamin....@gmail.com> wrote:
>>>>>
>>>>> 
>>>>> Yes that's the kind of slowness I'm talking about. As soon as you go
>>>>> over a second, it starts to feel painful IMO.
>>>>>
>>>>> I don't understand the details so I'll probably make a fool of myself
>>>>> but I'll give you a little more input.
>>>>>
>>>>> It'd feel natural to have a "quarkus:test" command, analogous to
>>>>> "quarkus:dev" that would load the app and keep it in a running state,
>>>>> dedicated for testing. And that's where my testing stuff would call to.
>>>>> That way this behavior would be opt-in and you wouldn't change a thing
>>>>> in regards to CI pipeline workflows, etc.
>>>>>
>>>>> So something like:
>>>>>

>>>>> - Step 1: run ./mvnw quarkus:test in a separate terminal
>>>>> - Step 2 : run a dedicated wrapper script that would call into the

>>>>> loaded test env ./bin/test-runner
>>>>> ./src/test/java/invalid/bt/GreetingResourceTest.java
>>>>>
>>>>> If you have that, then implementing a global test runner is trivial
>>>>> since the file watcher would execute step 2 upon file change basically.
>>>>>
>>>>> Conceptually, I'd say those are the same problem :
>>>>>

>>>>> 1. I changed code : I want to see result in my browser ASAP
>>>>> 2. I changed code : I want to see result in my test output ASAP

>>>>>>>>> <https://groups.google.com/d/msgid/quarkus-dev/30f193ee-9e62-46e8-a138-5e9f336ba550n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>> .

>>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>> the Google Groups "Quarkus Development mailing list" group.
>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>> https://groups.google.com/d/topic/quarkus-dev/qB3iW0ciLqI/unsubscribe
>>>>>>>> .
>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>> quarkus-dev...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cxjPsB9Qy-VaN2_181cYs%3D%3DiBgPAbkUN061%2BNwt3u9hGw%40mail.gmail.com

>>>>>>>> <https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cxjPsB9Qy-VaN2_181cYs%3D%3DiBgPAbkUN061%2BNwt3u9hGw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Benjamin Thomas*
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> *Benjamin Thomas*

>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Quarkus Development mailing list" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to quarkus-dev...@googlegroups.com.
>>>>> To view this discussion on the web visit

>>>>> https://groups.google.com/d/msgid/quarkus-dev/CABSP4jnmTRLA%3DmHRVPskOvdZO2zC6SUjgy2SyG0cAfPx0FfoZg%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/quarkus-dev/CABSP4jnmTRLA%3DmHRVPskOvdZO2zC6SUjgy2SyG0cAfPx0FfoZg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .

>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Quarkus Development mailing list" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to quarkus-dev...@googlegroups.com.
>>>>> To view this discussion on the web visit

>>>>> https://groups.google.com/d/msgid/quarkus-dev/CAJMo4%2BfiTj4ON1DzJLLtZ5KjORntjqJv24g_QKsxR57bGRfdMg%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/quarkus-dev/CAJMo4%2BfiTj4ON1DzJLLtZ5KjORntjqJv24g_QKsxR57bGRfdMg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>> /max
>>>>> https://xam.dk/about
>>>>>
>>>>
>>>
>>> --
>>> *Benjamin Thomas*
>>>
>>> --
>> *Benjamin Thomas*
>>
>>

[Benjamin Thomas]

I'm not sure I follow but I would keep 2 distinct processes (at least for the POC). Some tools already exist to merge STDOUTs such as:

• https://github.com/ddollar/forego
• https://github.com/strongloop/node-foreman
  

If quarkus:test has a dependency on quarkus:dev, then quarkus:test could fail early or loop until a connection to quarkus:test is acquired.

I personally use tmux to start all my dev processes so launching several foreground processes isn't a problem for me. It works nicely in conjunction with a docker-compose based setup so I could help out if you'd like some suggestions and simplify the dev experience to run one command only. If providing a purely docker based dev env is desirable of course.

I haven't tried the DevUI tool, but if the logic is centralized behind it, it'd be nice if I could construct my own tools by providing an HTTP API. It would give maximum flexibility for text editor integrations, and custom workflows.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/87tuq4tt84.fsf%40gmail.com.

--

Benjamin Thomas

[Loïc MATHIEU]

Hi,

I think that dev mode and continuous test mode (or watch test mode) should be separate, it's different ways to work: dev mode is for manual testing and test mode is for automated testing. I don't think we need to think of a mix approach now.

But any approach you propose would already be a very good addition so both are OK for me ;)

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CABSP4j%3DwM3KOXfsavnw%2BF48RYgR24c-_bNQTQ6ECZQtcktdSVA%40mail.gmail.com.