Design Proposal: Automatic Database Containers

[Stuart Douglas]

Hi Everyone,

Given that everyone seemed happy with the basic idea I proposed in the other thread I thought I would start a design discussion thread to discuss exactly how to do this. This will be a fairly visible change if we do it, so we need to get it right.

I propose the following:

- For each JDBC extension that has a container image available publicly we add the ability to automatically start the datasource in dev and test mode. If there is no public image we could still add support for it, however it would require the user to configure the image and will not be as seamless. Testcontainers has a hard dependency on JUnit 4, however as this is only used in the -deployment modules it won't pollute the ClassPath. Apparently it is possible to remove this dep if you provide a few empty JUnit classes, so this may be a solution.

- We make quarkus.database.kind optional in many situations:

  * If only DB extension is installed then this is assumed to be the DB in use, and the kind defaults to this DB

  * If the launch mode is test and a JDBC URL is provided then we infer the DB kind from the URL. We can do this in test mode because we know it is not going to change at runtime.

  * If the launch mode is test and there is only one jdbc extension installed with a scope of 'test' then this test scoped extension is assumed to be the default (we do kinda have this information available in CurateOutcomeBuildItem, although we may need a bit more metadata in the extensions).

  * Note that both reactive and JDBC extensions need to be considered when making this determination, although if you have both the reactive and JDBC version of the same DB then it will still be possible to infer the type

This means that the majority of applications won't need to configure this, which IMHO fits in with our 'convention over configuration' approach (why should the user need to configure the DB type if there is only a single possible value it could be?).

- We add the ability to automatically start a database container in dev and test mode, based on the following conditions:

  * quarkus.database.type can be automatically inferred (i.e. only one DB extension, or only one test scoped DB extension)

  * An explicit configuration option is set to start the container (in this we ignore the configured URL, and just start on localhost, but will use the configured username if present)

  * A global config option will be added to turn this off

  * Datasources will have a config option to set the container image that should be used

I am not 100% sure what the correct behaviour is if you have both a reactive and JDBC driver for the same database is (i.e. do you start one postgres container and they share it, or do they both get their own container). My gut feeling is that in this case they should share it.

Hopefully what this means is that for the majority of applications you don't need any datasource config to use Quarkus. If you are in dev or test mode Quarkus will just give you a database on startup, and if you are in production these settings are generally provided through env vars. This is not only something that demos really well and helps people get started, but it would also IMHO be a huge help for developers who work on a lot of different apps. You don't need to worry about setting up local databases or making sure you have the correct docker container and DB settings, everything just works out of the box.

Stuart

[Alex Soto Bueno]

Hi Stuart, I'd say that it is a really nice addition for testing purposes, so yes good and how about:

• Configuration parameters to configure the container (username, password, image, tag, database name)
• Make this automatic process configurable by profile (I think it is using profiles) so for example only integration tests use this approach.
• Another case which will be really nice to support is Kafka. 

Alex.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2czofZHOZqx3OO2zbTqtUYC7%2B1tbp_fe%2BcZ-MEEn-e2tbA%40mail.gmail.com.

[Stuart Douglas]

On Tue, 2 Feb 2021, 6:43 pm Alex Soto Bueno, <asot...@redhat.com> wrote:

Hi Stuart, I'd say that it is a really nice addition for testing purposes, so yes good and how about:

• Configuration parameters to configure the container (username, password, image, tag, database name)

Do you actually need this though (with the exception of image: tag)? We will automatically connect with the correct settings, so it shouldn't really matter.

• Make this automatic process configurable by profile (I think it is using profiles) so for example only integration tests use this approach.

It will use our normal configuration system so profiles will work as normal.

• Another case which will be really nice to support is Kafka. 

One thing at a time :-)

Stuart

[noel.o...@gmail.com]

Testcontainers currently only runs with Docker but there's work going on to make it work with Podman. It'd be good to be able to select one or the other.

[George Gastaldi]

I remember seeing TestContainers work with JUnit 5, it's just not implemented as a @Rule. 

Do we really need to depend on Junit 4 in the deployment modules?

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/69afc343-ed60-4e15-b964-2c383d2eccd1n%40googlegroups.com.

[Georgios Andrianakis]

On Tue, Feb 2, 2021 at 6:03 AM Stuart Douglas <sdou...@redhat.com> wrote:

Hi Everyone,

Given that everyone seemed happy with the basic idea I proposed in the other thread I thought I would start a design discussion thread to discuss exactly how to do this. This will be a fairly visible change if we do it, so we need to get it right.

I propose the following:

- For each JDBC extension that has a container image available publicly we add the ability to automatically start the datasource in dev and test mode. If there is no public image we could still add support for it, however it would require the user to configure the image and will not be as seamless. Testcontainers has a hard dependency on JUnit 4, however as this is only used in the -deployment modules it won't pollute the ClassPath. Apparently it is possible to remove this dep if you provide a few empty JUnit classes, so this may be a solution.

I really like the idea

- We make quarkus.database.kind optional in many situations:

  * If only DB extension is installed then this is assumed to be the DB in use, and the kind defaults to this DB

  * If the launch mode is test and a JDBC URL is provided then we infer the DB kind from the URL. We can do this in test mode because we know it is not going to change at runtime.

  * If the launch mode is test and there is only one jdbc extension installed with a scope of 'test' then this test scoped extension is assumed to be the default (we do kinda have this information available in CurateOutcomeBuildItem, although we may need a bit more metadata in the extensions).

  * Note that both reactive and JDBC extensions need to be considered when making this determination, although if you have both the reactive and JDBC version of the same DB then it will still be possible to infer the type

This means that the majority of applications won't need to configure this, which IMHO fits in with our 'convention over configuration' approach (why should the user need to configure the DB type if there is only a single possible value it could be?).

- We add the ability to automatically start a database container in dev and test mode, based on the following conditions:

  * quarkus.database.type can be automatically inferred (i.e. only one DB extension, or only one test scoped DB extension)

  * An explicit configuration option is set to start the container (in this we ignore the configured URL, and just start on localhost, but will use the configured username if present)

  * A global config option will be added to turn this off

  * Datasources will have a config option to set the container image that should be used

These rules sound appropriate to me.

I am not 100% sure what the correct behaviour is if you have both a reactive and JDBC driver for the same database is (i.e. do you start one postgres container and they share it, or do they both get their own container). My gut feeling is that in this case they should share it.

I'd like to hear from Guillaume and Sanne about this.

Hopefully what this means is that for the majority of applications you don't need any datasource config to use Quarkus. If you are in dev or test mode Quarkus will just give you a database on startup, and if you are in production these settings are generally provided through env vars. This is not only something that demos really well and helps people get started, but it would also IMHO be a huge help for developers who work on a lot of different apps. You don't need to worry about setting up local databases or making sure you have the correct docker container and DB settings, everything just works out of the box.

Stuart

--

[Sanne Grinovero]

On Tue, 2 Feb 2021, 10:41 Georgios Andrianakis, <gand...@redhat.com> wrote:

On Tue, Feb 2, 2021 at 6:03 AM Stuart Douglas <sdou...@redhat.com> wrote:

Hi Everyone,

Given that everyone seemed happy with the basic idea I proposed in the other thread I thought I would start a design discussion thread to discuss exactly how to do this. This will be a fairly visible change if we do it, so we need to get it right.

I propose the following:

- For each JDBC extension that has a container image available publicly we add the ability to automatically start the datasource in dev and test mode. If there is no public image we could still add support for it, however it would require the user to configure the image and will not be as seamless. Testcontainers has a hard dependency on JUnit 4, however as this is only used in the -deployment modules it won't pollute the ClassPath. Apparently it is possible to remove this dep if you provide a few empty JUnit classes, so this may be a solution.

I really like the idea

- We make quarkus.database.kind optional in many situations:

  * If only DB extension is installed then this is assumed to be the DB in use, and the kind defaults to this DB

  * If the launch mode is test and a JDBC URL is provided then we infer the DB kind from the URL. We can do this in test mode because we know it is not going to change at runtime.

  * If the launch mode is test and there is only one jdbc extension installed with a scope of 'test' then this test scoped extension is assumed to be the default (we do kinda have this information available in CurateOutcomeBuildItem, although we may need a bit more metadata in the extensions).

  * Note that both reactive and JDBC extensions need to be considered when making this determination, although if you have both the reactive and JDBC version of the same DB then it will still be possible to infer the type

This means that the majority of applications won't need to configure this, which IMHO fits in with our 'convention over configuration' approach (why should the user need to configure the DB type if there is only a single possible value it could be?).

- We add the ability to automatically start a database container in dev and test mode, based on the following conditions:

  * quarkus.database.type can be automatically inferred (i.e. only one DB extension, or only one test scoped DB extension)

  * An explicit configuration option is set to start the container (in this we ignore the configured URL, and just start on localhost, but will use the configured username if present)

  * A global config option will be added to turn this off

  * Datasources will have a config option to set the container image that should be used

These rules sound appropriate to me.

I am not 100% sure what the correct behaviour is if you have both a reactive and JDBC driver for the same database is (i.e. do you start one postgres container and they share it, or do they both get their own container). My gut feeling is that in this case they should share it.

I'd like to hear from Guillaume and Sanne about this.

It's a good question, and I'm not sure, but it seems a reasonable default to start with.

I would expect that for dev purposes one would definitely want to reuse the container, perhaps if separation is required one would go for a different schema/dbname while reusing the container instance. But yes it seems at least likely one would want to share the same database so let's go with such a default, additional options can be added later as use cases emerge.

Hopefully what this means is that for the majority of applications you don't need any datasource config to use Quarkus. If you are in dev or test mode Quarkus will just give you a database on startup, and if you are in production these settings are generally provided through env vars. This is not only something that demos really well and helps people get started, but it would also IMHO be a huge help for developers who work on a lot of different apps. You don't need to worry about setting up local databases or making sure you have the correct docker container and DB settings, everything just works out of the box.

Stuart

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2czofZHOZqx3OO2zbTqtUYC7%2B1tbp_fe%2BcZ-MEEn-e2tbA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-kr0aLx%2By4E4d4kH8QvA1okf2Hm%3Dp2C32SRk8Gr3vSokA%40mail.gmail.com.

[Stuart Douglas]

It's a hard dependency, the container classes extend TestRule.

Stuart

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJiFCXsMXficmxLUywEFwU8_9Ri3EgwttbmqpmH5dS%2BMpA%40mail.gmail.com.

[Loïc Hermann]

Hi All,

I really like this idea. Just some additional thought: in a multi module project it might be required that the db is only started once (but not necessarily). Also it would be usefull to be able to inject data with dbunit.

Regards,

Loïc

[Alex Soto Bueno]

And we can even make the process one step further and I know I am hijacking the thread but how about generating the Kubernetes YAML file with the deployment of the DB in it (or make it configurable to generate it), so can you imagine for a developer and even anyone demoing Quarkus doing a ./mvnw package -Dquarkus.kubernetes.deploy=true and all the app up and running even with the dependencies? And then with Kafka we could add Strimzi, Vault, ...

Ok I am getting crazy hahaha

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/d1757f4b-c9b5-4b35-9073-fbd66ee63de7n%40googlegroups.com.

[George Gastaldi]

FYI I created https://github.com/testcontainers/testcontainers-java/issues/3756 asking to remove the hard dependency on JUnit 4.

[Stuart Douglas]

On Thu, 4 Feb 2021 at 22:15, Loïc Hermann <loic.p...@gmail.com> wrote:

Hi All,

I really like this idea. Just some additional thought: in a multi module project it might be required that the db is only started once (but not necessarily).

We can't really handle that, at least not without a huge amount of work.

 

Also it would be usefull to be able to inject data with dbunit.

We could investigate this, but from the tests PoV it should be no different to any other database, so it should be fine.

Stuart

 

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/d1757f4b-c9b5-4b35-9073-fbd66ee63de7n%40googlegroups.com.

[Stuart Douglas]

On Thu, 4 Feb 2021 at 23:30, Alex Soto Bueno <asot...@redhat.com> wrote:

And we can even make the process one step further and I know I am hijacking the thread but how about generating the Kubernetes YAML file with the deployment of the DB in it (or make it configurable to generate it), so can you imagine for a developer and even anyone demoing Quarkus doing a ./mvnw package -Dquarkus.kubernetes.deploy=true and all the app up and running even with the dependencies? And then with Kafka we could add Strimzi, Vault, ...

Ok I am getting crazy hahaha

I have actually thought a bit about this, mostly in the context of the dev console.

Basically can we define all the services that an app needs in a relatively generic way, so that we can automatically start them no matter the environment?

Not just databases and Kafka, but also potentially images containing other Quarkus apps that expose HTTP services. If we had this then you could easily enable

end to end testing in any environment, and make it really easy to setup local dev. You could also do things like actually run integration tests on a kube

cluster to test in the environment you are going to run in.

This is still all very hand wavey at this point though, I think this is a good start and something we can build on.

Stuart

 

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAOaD6k%2BNPhLizCEcqABDJMjtuVJMsvSNe%2B4Q%3D2qgvD5fuc2KHA%40mail.gmail.com.

[Stuart Douglas]

There have been no real objections to this, so I will start implementing it next week.

Stuart

[Stuart Douglas]

I have a 70%(ish?) implementation here: https://github.com/stuartwdouglas/quarkus/tree/testcontainers-demo

What I have:

- Support for automatically starting a postgresql or h2 database when running in dev or test mode if one is not configured (note that H2 does not run in a container, just in process)

- Support for both JDBC and Reactive (if both are present the datasource is shared)

- Support for native tests (containers are started by the NativeTestExtension and config is passed into the executable)

- Automatic detection of the DB type if only one driver is present

- Automatic detection of the DB type during tests if only one test scoped DB driver is present (e.g. if you have a postgres app, but you include the h2 extension with scope test for testing purposes it will default to postgres in dev mode and h2 when running tests)

What is still to come:

- Lots of tests

- Docs

- Support for other open source databases

- Support for customising the container images (and other config)

- The enhanced readiness support that Sanne mentioned

Hopefully I will have a PR ready to go fairly soon.

Stuart

[Stuart Douglas]

Draft PR is here: https://github.com/quarkusio/quarkus/pull/14960

There is still a bit of cleanup I need to do (in particular deal with shutdown better when running tests), but it is mostly all there.

Stuart

[Alex Soto Bueno]

Great!!! When you mention "creating them in process" means running docker run right? Could it also be configured to start it using podman?. And ok even more complicated scenario that could be implemented later doing a kubectl create + kubectl port-forward.

Just some ideas, I want to see it and trust me it will simplify my work a lot. 

Thanks.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2czv11kH_sYjh2GnfP81EN7XwESCsiS3D-XvK6bJSa76aw%40mail.gmail.com.

[Stuart Douglas]

On Wed, 10 Feb 2021 at 20:48, Alex Soto Bueno <asot...@redhat.com> wrote:

Great!!! When you mention "creating them in process" means running docker run right? Could it also be configured to start it using podman?. And ok even more complicated scenario that could be implemented later doing a kubectl create + kubectl port-forward.

The in-process ones are Derby and H2, which I can just launch directly. Postgresql, MySQL and MariaDB are launched using testcontainers.

I could make them run using docker/podman directly, although I am pretty sure I would end up just re-implementing testcontainers.

In terms of running them in a cluster I have thought about that, but figured I would try and get the basics working first (I actually mentioned this on twitter the other day as a solution for devs who can't run docker on their laptop).

Stuart

[Alex Soto Bueno]

On Wed, Feb 10, 2021 at 10:54 AM Stuart Douglas <sdou...@redhat.com> wrote:

On Wed, 10 Feb 2021 at 20:48, Alex Soto Bueno <asot...@redhat.com> wrote:

Great!!! When you mention "creating them in process" means running docker run right? Could it also be configured to start it using podman?. And ok even more complicated scenario that could be implemented later doing a kubectl create + kubectl port-forward.

The in-process ones are Derby and H2, which I can just launch directly. Postgresql, MySQL and MariaDB are launched using testcontainers.

I could make them run using docker/podman directly, although I am pretty sure I would end up just re-implementing testcontainers.

Yes that's true.

[Sanne Grinovero]

On Wed, 10 Feb 2021 at 10:00, Alex Soto Bueno <asot...@redhat.com> wrote:
>
>
>
> On Wed, Feb 10, 2021 at 10:54 AM Stuart Douglas <sdou...@redhat.com> wrote:
>>
>>
>>
>> On Wed, 10 Feb 2021 at 20:48, Alex Soto Bueno <asot...@redhat.com> wrote:
>>>
>>> Great!!! When you mention "creating them in process" means running docker run right? Could it also be configured to start it using podman?. And ok even more complicated scenario that could be implemented later doing a kubectl create + kubectl port-forward.
>>
>>
>> The in-process ones are Derby and H2, which I can just launch directly. Postgresql, MySQL and MariaDB are launched using testcontainers.
>>
>> I could make them run using docker/podman directly, although I am pretty sure I would end up just re-implementing testcontainers.

That won't be necessary, Testcontainers can use podman now.

The exception is some containers attempt to do "stuff" [1] which
podman blocks; one of these is DB2, which has been a big annoyance for
me as it means I can't run the full Quarkus integration suite without
several extra hoops. MySQL also has a couple issues - but barring
these the others should just work.

If you want to try it, you can run the tests as usual, but need to
start the podman daemon before:

DOCKER_HOST="unix:/run/user/$(id -u)/podman/podman.sock" podman
system service -t 0

I hope such issues will get ironed out at a different level, and would
suggest to stick with supporting just Testcontainers.

Thanks!

1 - that's the best I could figure.. no idea.

> To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAOaD6k%2Bj%3DzdMUMZe7ifcRuO7YVqdU_8uyqgazstaogG5P7SpPg%40mail.gmail.com.

[Stuart Douglas]

That won't be necessary, Testcontainers can use podman now.

The exception is some containers attempt to do "stuff" [1] which
podman blocks; one of these is DB2, which has been a big annoyance for
me as it means I can't run the full Quarkus integration suite without
several extra hoops. MySQL also has a couple issues - but barring
these the others should just work.

If you want to try it, you can run the tests as usual, but need to
start the podman daemon before:

DOCKER_HOST="unix:/run/user/$(id -u)/podman/podman.sock"  podman
system service -t 0

I hope such issues will get ironed out at a different level, and would
suggest to stick with supporting just Testcontainers.

We may end up wanting more than what Testcontainers can provide, if we want to support actually starting this on other (potentially remote) container runtimes such as Kube/Openshift.

The PR has been moved out of draft now, if CI is green in theory it should be ready to go.

Something Max brought up on the PR is 'Does this mean developers need docker now', the answer to that is only if they want to use container based DevDB databases (Postgres, MySQL, MariaDB). If you have an

existing application with a DB configured normally then DevDB will not kick in, and you will notice no difference. The only time this takes effect is if you have a database extension installed but don't have it configured.

Stuart

[Alex Soto Bueno]

Hi, also I was thinking maybe not for the first version that since TestContainers support Docker Compose, this feature could check if there is a docker-compose.yml file at src/main/docker or src/test/docker and if there is one, start it.

Just an idea.

[Stuart Douglas]

This PR is now passing CI, and is ready for review.

Stuart