DevServices 2.0

[Stuart Douglas]

Hi Everyone,

I have been thinking a bit about DevServices 2.0, and I think there are a few additional features we could add:

- Persistent Containers

Basically give the containers we start an ID that ties them to the current project, and allow them to survive shutdown and be re-used. This would be really useful for things like running a test from the IDE, as we would not need to wait for the container to start each time.

- Kube support

If a developer does not have docker installed, but does have access to a cluster, it would be great if we could use that cluster to run their services. Either start the services on the cluster and port forward, or seamlessly setup remote-dev and run everything on the cluster. This won't be zero-config like DevServices is locally, but could still be a hugely useful tool.

- Shared / Configurable Containers

This is based on something Clement brought up today, Kafka is not very useful if you only have a single application connected to it. I think it would be useful to specify other applications as dev services targets, and have them automatically wire themselves up via service bindings.

Basically if I have a producer and a consumer Kafka app, and I am working on the consumer, I could configure Quarkus to automatically start the producer app, and have it wire up the same Kafka based on service binding information. Potentially this could be transitive for other Quarkus applications, so in an ideal world you could ask to start your dependencies, and they would also start their dependencies, and everything would just wire itself up automatically (using service bindings to resolve shared services).

I am not going into specifics here, because there is a lot of complexity in implementing this well, but I am interested in people's thoughts.

Stuart

[clement escoffier]

Hello,

Le mar. 1 juin 2021 à 13:09, Stuart Douglas <sdou...@redhat.com> a écrit :

Hi Everyone,

I have been thinking a bit about DevServices 2.0, and I think there are a few additional features we could add:

- Persistent Containers

Basically give the containers we start an ID that ties them to the current project, and allow them to survive shutdown and be re-used. This would be really useful for things like running a test from the IDE, as we would not need to wait for the container to start each time.

When do you stop them? It is up to the user to kill the running containers once done, or do we need a session concept? Test Containers is using Ryuk as a watchdog, the same kind of mechanism could be used.

 

- Kube support

If a developer does not have docker installed, but does have access to a cluster, it would be great if we could use that cluster to run their services. Either start the services on the cluster and port forward, or seamlessly setup remote-dev and run everything on the cluster. This won't be zero-config like DevServices is locally, but could still be a hugely useful tool.

- Shared / Configurable Containers

This is based on something Clement brought up today, Kafka is not very useful if you only have a single application connected to it. I think it would be useful to specify other applications as dev services targets, and have them automatically wire themselves up via service bindings.

Basically if I have a producer and a consumer Kafka app, and I am working on the consumer, I could configure Quarkus to automatically start the producer app, and have it wire up the same Kafka based on service binding information. Potentially this could be transitive for other Quarkus applications, so in an ideal world you could ask to start your dependencies, and they would also start their dependencies, and everything would just wire itself up automatically (using service bindings to resolve shared services).

Another solution might be to implement some very light discovery/inventory mechanism that would allow quarkus runs (dev and test) to check the dev services that are running. A rough idea would be to have a specific port and start a simple HTTP server if the port is available (otherwise it means the discovery service is already running). When you start a dev service, you register the service (the same key/value as we have today, maybe a bit more). So, when we need to start a dev service, we can connect to that discovery service. If it's there, and the service is available, we just use that, if not we start it and register it.  The discovery service would kill itself if nothing is registered for some time.

Such a mechanism could be used for the persistent containers if we can track the start/stop of the quarkus app. When the last one stops, we terminate all the persistent containers. 

Note that the inventory can be also a container and we can use the docker host API to discover it.

Finally, as we are using test containers, we could query the label of running containers (and add labels when we start them). It would not enable persistent containers, but avoid running an inventory service. 

Clement

 

I am not going into specifics here, because there is a lot of complexity in implementing this well, but I am interested in people's thoughts.

 

Stuart

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cyRiKfqHezfQNHawzRu-H_%3DX2UnWRbOo83o9RiMVz7%2BQw%40mail.gmail.com.

[Stephane Epardaud]

On Tue, 1 Jun 2021 at 13:09, Stuart Douglas <sdou...@redhat.com> wrote:

- Persistent Containers

Basically give the containers we start an ID that ties them to the current project, and allow them to survive shutdown and be re-used. This would be really useful for things like running a test from the IDE, as we would not need to wait for the container to start each time.

Very interesting. I've mostly used a local PG instance for my dev work, and I've always appreciated that whatever test data I put in there while developing my app survived all restarts, until I changed my schema and hibernate recreated the DB and dropped records (in hindsight I would have prefered it tried to evolve my data, but hey).

So, +1 for surviving the data, but I'm not even entirely sure how (in the case of Hibernate) we'd keep the data around on the next start.

As for auto-shutting down the container after a period of unuse, I've no idea how to do that, but it'd be extra nice if we could dump the data to file on quarkus exit, just in case the container is missing at the next restart.

BTW, I'm pretty sure we could reuse existing containers for all applications, just by having them use a separate database. And I'd also love it if we could use existing databases rather than use containers. On Linux it's really trivial to have a PG running without having to docker anything.

 

- Kube support

If a developer does not have docker installed, but does have access to a cluster, it would be great if we could use that cluster to run their services. Either start the services on the cluster and port forward, or seamlessly setup remote-dev and run everything on the cluster. This won't be zero-config like DevServices is locally, but could still be a hugely useful tool.

Yeah, this makes sense to me, but I'm not sure how common that is for people to have k8s but no docker. Sounds a bit fringe, no?

 

- Shared / Configurable Containers

This is based on something Clement brought up today, Kafka is not very useful if you only have a single application connected to it. I think it would be useful to specify other applications as dev services targets, and have them automatically wire themselves up via service bindings.

Basically if I have a producer and a consumer Kafka app, and I am working on the consumer, I could configure Quarkus to automatically start the producer app, and have it wire up the same Kafka based on service binding information. Potentially this could be transitive for other Quarkus applications, so in an ideal world you could ask to start your dependencies, and they would also start their dependencies, and everything would just wire itself up automatically (using service bindings to resolve shared services).

This sounds _very_ interesting, but what use are auto-started dependency apps if they don't spew out relevant mock information? I'm thinking of micro-services and the other discussion we had around testing and mocking remote services. It's one thing to spin up the remote service rather than mock it, but it still needs to reply intelligible stuff to any calls we make to it, and I'm not sure at all how to do that. 

[Georgios Andrianakis]

On Tue, Jun 1, 2021 at 4:27 PM clement escoffier <clement....@gmail.com> wrote:

Hello,

Le mar. 1 juin 2021 à 13:09, Stuart Douglas <sdou...@redhat.com> a écrit :

Hi Everyone,

I have been thinking a bit about DevServices 2.0, and I think there are a few additional features we could add:

- Persistent Containers

Basically give the containers we start an ID that ties them to the current project, and allow them to survive shutdown and be re-used. This would be really useful for things like running a test from the IDE, as we would not need to wait for the container to start each time.

When do you stop them? It is up to the user to kill the running containers once done, or do we need a session concept? Test Containers is using Ryuk as a watchdog, the same kind of mechanism could be used.

Can't we stop the container when the application starts but then attempt to start it (as it is named) if the application starts again? You won't get much of performance benefit, but you will get persistent data

 

- Kube support

If a developer does not have docker installed, but does have access to a cluster, it would be great if we could use that cluster to run their services. Either start the services on the cluster and port forward, or seamlessly setup remote-dev and run everything on the cluster. This won't be zero-config like DevServices is locally, but could still be a hugely useful tool.

This would also make for a cool demo :)

- Shared / Configurable Containers

This is based on something Clement brought up today, Kafka is not very useful if you only have a single application connected to it. I think it would be useful to specify other applications as dev services targets, and have them automatically wire themselves up via service bindings.

Basically if I have a producer and a consumer Kafka app, and I am working on the consumer, I could configure Quarkus to automatically start the producer app, and have it wire up the same Kafka based on service binding information. Potentially this could be transitive for other Quarkus applications, so in an ideal world you could ask to start your dependencies, and they would also start their dependencies, and everything would just wire itself up automatically (using service bindings to resolve shared services).

Another solution might be to implement some very light discovery/inventory mechanism that would allow quarkus runs (dev and test) to check the dev services that are running. A rough idea would be to have a specific port and start a simple HTTP server if the port is available (otherwise it means the discovery service is already running). When you start a dev service, you register the service (the same key/value as we have today, maybe a bit more). So, when we need to start a dev service, we can connect to that discovery service. If it's there, and the service is available, we just use that, if not we start it and register it.  The discovery service would kill itself if nothing is registered for some time.

Such a mechanism could be used for the persistent containers if we can track the start/stop of the quarkus app. When the last one stops, we terminate all the persistent containers. 

Note that the inventory can be also a container and we can use the docker host API to discover it.

Finally, as we are using test containers, we could query the label of running containers (and add labels when we start them). It would not enable persistent containers, but avoid running an inventory service. 

Clement

 

I am not going into specifics here, because there is a lot of complexity in implementing this well, but I am interested in people's thoughts.

 

Stuart

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cyRiKfqHezfQNHawzRu-H_%3DX2UnWRbOo83o9RiMVz7%2BQw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAKW6fifMz6GWYq8e%2BrHCEpX5%2BjuP-1xAp2Maab9rn8Kq3_SyeQ%40mail.gmail.com.

[Erin Schnabel]

On Tue, Jun 1, 2021 at 11:02 AM Stephane Epardaud <stephane...@gmail.com> wrote:

On Tue, 1 Jun 2021 at 13:09, Stuart Douglas <sdou...@redhat.com> wrote:

- Persistent Containers

Basically give the containers we start an ID that ties them to the current project, and allow them to survive shutdown and be re-used. This would be really useful for things like running a test from the IDE, as we would not need to wait for the container to start each time.

Very interesting. I've mostly used a local PG instance for my dev work, and I've always appreciated that whatever test data I put in there while developing my app survived all restarts, until I changed my schema and hibernate recreated the DB and dropped records (in hindsight I would have prefered it tried to evolve my data, but hey).

So, +1 for surviving the data, but I'm not even entirely sure how (in the case of Hibernate) we'd keep the data around on the next start.

As for auto-shutting down the container after a period of unuse, I've no idea how to do that, but it'd be extra nice if we could dump the data to file on quarkus exit, just in case the container is missing at the next restart.

BTW, I'm pretty sure we could reuse existing containers for all applications, just by having them use a separate database. And I'd also love it if we could use existing databases rather than use containers. On Linux it's really trivial to have a PG running without having to docker anything.

I use Linux, and I still almost always use docker to start services for app development rather than using anything natively.. docker-compose is just too easy (and means my setup is the same since I tend to migrate between my mac and my linux box). (and yes, I use docker, not podman).

I "usually" am not using transactional data. I never learned it (in Java, I did in Ruby?!), and didn't want to, so I hopped to NoSQL and skipped everything else. For those apps, I have something around to prime the DB if it starts up and is empty. Evolution of data is a lot more fluid with NoSQL (though there are other hazards.. ).

When to start/stop resources is definitely a question. I _personally_ would favor (convenient) explicit action, rather than too much magic. If I start it, then I know that I need to stop it. I also know (or can indicate) that I want it to keep running, or that I want it to drop all tables and start over, or that I want data to be remembered at stop, or...Explicit actions can be magic'ed later (once we have a better feel for how it should work).

 

- Kube support

If a developer does not have docker installed, but does have access to a cluster, it would be great if we could use that cluster to run their services. Either start the services on the cluster and port forward, or seamlessly setup remote-dev and run everything on the cluster. This won't be zero-config like DevServices is locally, but could still be a hugely useful tool.

Yeah, this makes sense to me, but I'm not sure how common that is for people to have k8s but no docker. Sounds a bit fringe, no?

Well, there are those podman/buildah crazy people. :-P So you could totally have buildah making things, pushing off to repo somewhere else.. blahblah. I think it's definitely rarer. Not what I would address with the first pass.

 

 

- Shared / Configurable Containers

This is based on something Clement brought up today, Kafka is not very useful if you only have a single application connected to it. I think it would be useful to specify other applications as dev services targets, and have them automatically wire themselves up via service bindings.

Basically if I have a producer and a consumer Kafka app, and I am working on the consumer, I could configure Quarkus to automatically start the producer app, and have it wire up the same Kafka based on service binding information. Potentially this could be transitive for other Quarkus applications, so in an ideal world you could ask to start your dependencies, and they would also start their dependencies, and everything would just wire itself up automatically (using service bindings to resolve shared services).

This sounds _very_ interesting, but what use are auto-started dependency apps if they don't spew out relevant mock information? I'm thinking of micro-services and the other discussion we had around testing and mocking remote services. It's one thing to spin up the remote service rather than mock it, but it still needs to reply intelligible stuff to any calls we make to it, and I'm not sure at all how to do that. 

If I could wire up arbitrary running/additional services, this could be useful. I have an example of that (gameontext.org) .. where we can use docker-compose or kube to set up 5 or so services.. so even if I'm working on one of them, there are others that can be wired up / injected. (They don't need to operate on mock data, they just need to be there and operate normally)...

 

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAKU9E9sfcARgaDHEuWvWaNyE4A0-ZgsSLejn%3DY24JMuhWfHY4Q%40mail.gmail.com.

[Stephane Epardaud]

On Tue, 1 Jun 2021 at 17:41, Erin Schnabel <ebul...@redhat.com> wrote:

 (They don't need to operate on mock data, they just need to be there and operate normally)... 

Well, yes but to me "operate normally" means have enough data to work. Like, either have an existing prod data set, or is receiving events from external sources to pass on to my app… None of which will happen here.

[Stuart Douglas]

On Tue, 1 Jun 2021 at 23:27, clement escoffier <clement....@gmail.com> wrote:

Hello,

Le mar. 1 juin 2021 à 13:09, Stuart Douglas <sdou...@redhat.com> a écrit :

Hi Everyone,

I have been thinking a bit about DevServices 2.0, and I think there are a few additional features we could add:

- Persistent Containers

Basically give the containers we start an ID that ties them to the current project, and allow them to survive shutdown and be re-used. This would be really useful for things like running a test from the IDE, as we would not need to wait for the container to start each time.

When do you stop them? It is up to the user to kill the running containers once done, or do we need a session concept? Test Containers is using Ryuk as a watchdog, the same kind of mechanism could be used.

I was thinking a timed session, so the containers hang around 10m or so after shutdown, just so you don't have to pay for the startup cost each time.

 

 

- Kube support

If a developer does not have docker installed, but does have access to a cluster, it would be great if we could use that cluster to run their services. Either start the services on the cluster and port forward, or seamlessly setup remote-dev and run everything on the cluster. This won't be zero-config like DevServices is locally, but could still be a hugely useful tool.

- Shared / Configurable Containers

This is based on something Clement brought up today, Kafka is not very useful if you only have a single application connected to it. I think it would be useful to specify other applications as dev services targets, and have them automatically wire themselves up via service bindings.

Basically if I have a producer and a consumer Kafka app, and I am working on the consumer, I could configure Quarkus to automatically start the producer app, and have it wire up the same Kafka based on service binding information. Potentially this could be transitive for other Quarkus applications, so in an ideal world you could ask to start your dependencies, and they would also start their dependencies, and everything would just wire itself up automatically (using service bindings to resolve shared services).

Another solution might be to implement some very light discovery/inventory mechanism that would allow quarkus runs (dev and test) to check the dev services that are running. A rough idea would be to have a specific port and start a simple HTTP server if the port is available (otherwise it means the discovery service is already running). When you start a dev service, you register the service (the same key/value as we have today, maybe a bit more). So, when we need to start a dev service, we can connect to that discovery service. If it's there, and the service is available, we just use that, if not we start it and register it.  The discovery service would kill itself if nothing is registered for some time.

This could be useful.

 

Such a mechanism could be used for the persistent containers if we can track the start/stop of the quarkus app. When the last one stops, we terminate all the persistent containers. 

That won't really help with the 'I just want to run one test from the IDE scenario though', although maybe with continuous testing that is less of an issue.

 

Note that the inventory can be also a container and we can use the docker host API to discover it.

Finally, as we are using test containers, we could query the label of running containers (and add labels when we start them). It would not enable persistent containers, but avoid running an inventory service. 

We could also potentially share some containers, e.g. if we have a DevServices postgres we could just connect to the existing one and create a new database (so isolate via database rather than container).

Stuart

[Stuart Douglas]

On Wed, 2 Jun 2021 at 01:02, Stephane Epardaud <stephane...@gmail.com> wrote:

On Tue, 1 Jun 2021 at 13:09, Stuart Douglas <sdou...@redhat.com> wrote:

- Persistent Containers

Basically give the containers we start an ID that ties them to the current project, and allow them to survive shutdown and be re-used. This would be really useful for things like running a test from the IDE, as we would not need to wait for the container to start each time.

Very interesting. I've mostly used a local PG instance for my dev work, and I've always appreciated that whatever test data I put in there while developing my app survived all restarts, until I changed my schema and hibernate recreated the DB and dropped records (in hindsight I would have prefered it tried to evolve my data, but hey).

So, +1 for surviving the data, but I'm not even entirely sure how (in the case of Hibernate) we'd keep the data around on the next start.

As for auto-shutting down the container after a period of unuse, I've no idea how to do that, but it'd be extra nice if we could dump the data to file on quarkus exit, just in case the container is missing at the next restart.

BTW, I'm pretty sure we could reuse existing containers for all applications, just by having them use a separate database. And I'd also love it if we could use existing databases rather than use containers. On Linux it's really trivial to have a PG running without having to docker anything.

I was not really thinking about data persistence, more about startup time, but that is definitly something we could look at.

That won't be 'zero config' though, as you would need to configure your local postgres, could still be really convenient though as you could potentially just configure one set of credentials in an env var in your .bashrc and then all your quarkus apps will just work without docker.

 

 

- Kube support

If a developer does not have docker installed, but does have access to a cluster, it would be great if we could use that cluster to run their services. Either start the services on the cluster and port forward, or seamlessly setup remote-dev and run everything on the cluster. This won't be zero-config like DevServices is locally, but could still be a hugely useful tool.

Yeah, this makes sense to me, but I'm not sure how common that is for people to have k8s but no docker. Sounds a bit fringe, no?

Apparently a lot of corporate environments disable docker on developers laptops for security reasons, but will have access to kube/openshift.

 

 

- Shared / Configurable Containers

This is based on something Clement brought up today, Kafka is not very useful if you only have a single application connected to it. I think it would be useful to specify other applications as dev services targets, and have them automatically wire themselves up via service bindings.

Basically if I have a producer and a consumer Kafka app, and I am working on the consumer, I could configure Quarkus to automatically start the producer app, and have it wire up the same Kafka based on service binding information. Potentially this could be transitive for other Quarkus applications, so in an ideal world you could ask to start your dependencies, and they would also start their dependencies, and everything would just wire itself up automatically (using service bindings to resolve shared services).

This sounds _very_ interesting, but what use are auto-started dependency apps if they don't spew out relevant mock information? I'm thinking of micro-services and the other discussion we had around testing and mocking remote services. It's one thing to spin up the remote service rather than mock it, but it still needs to reply intelligible stuff to any calls we make to it, and I'm not sure at all how to do that. 

The remote service might actually be a mock (just a mock in a container). Also it will depend a lot on what the remote service does, potentially you should be able to quickly spin up a fully functioning app environment locally, and just use it like you would the normal app.

Stuart