Should applications with an unconfigured datasource even start?

197 views
Skip to first unread message

Yoann Rodiere

unread,
Dec 8, 2023, 10:29:26 AM12/8/23
to Quarkus Development mailing list
Hi,

Some time ago we received a report [1] that applications that "forget" to set a JDBC URL for a datasource might start and be considered ready/healthy, even though the datasource is not, actually, ready.

Applications that use these datasources through a configured Hibernate ORM/Reactive persistence unit actually won't start, because Hibernate checks the datasources are configured for each persistence unit on startup [2].

But applications that don't use Hibernate, or that have no configured persistence unit, will indeed start and appear to work well (health checks will report "UP")... except they'll fail to serve any request that involves the datasource.

This is quite confusing, and arguably downright dangerous.
Worse, the use cases are scarce. After some thinking, I could only come up with these:
  1. Some applications may require a particular datasource for some of their endpoints only, and thus developers might want to deploy the application without configuring the datasource.
  2. We always define a default datasource, so applications that rely exclusively on named datasources would need to start even if the default datasource is not configured.
To get out of the current situation, I would like to suggest that we change this behavior:
  1. I'm working on [3] to introduce a way to *explicitly* mark a datasource as "inactive" at runtime. This should address the first use case above (and more, but that's beside the point).
  2. We will no longer define a default datasource at build time if a named datasource is configured, and there is no configuration for the default datasource. This should make sure we don't break the second use case above.
    It's also similar to what we do for persistence units in Hibernate ORM/Reactive [4]
  3. If a datasource defined at build time (default or named) is unconfigured at runtime, we'll fail on startup and recommend configuring that datasource or deactivating it.
Item 2 might require some changes in Smallrye Config as it seems a Map using `@WithUnnamedKey` [5] [6] will currently always contain the unnamed key... at least in some cases. Hopefully that's something that can be addressed, but this might delay the change past the 3.7 feature freeze. Which I guess could be a good thing anyway.

Back to the proposal: any thoughts, opinions on the general approach?

Thanks,


Yoann Rodière
Hibernate Team

Georgios Andrianakis

unread,
Dec 12, 2023, 3:21:36 AM12/12/23
to yo...@hibernate.org, Quarkus Development mailing list
On Fri, Dec 8, 2023 at 5:29 PM Yoann Rodiere <yo...@hibernate.org> wrote:
Hi,

Some time ago we received a report [1] that applications that "forget" to set a JDBC URL for a datasource might start and be considered ready/healthy, even though the datasource is not, actually, ready.

Applications that use these datasources through a configured Hibernate ORM/Reactive persistence unit actually won't start, because Hibernate checks the datasources are configured for each persistence unit on startup [2].

But applications that don't use Hibernate, or that have no configured persistence unit, will indeed start and appear to work well (health checks will report "UP")... except they'll fail to serve any request that involves the datasource.

This is quite confusing, and arguably downright dangerous.

I completely agree.
 
Worse, the use cases are scarce. After some thinking, I could only come up with these:
  1. Some applications may require a particular datasource for some of their endpoints only, and thus developers might want to deploy the application without configuring the datasource.
  2. We always define a default datasource, so applications that rely exclusively on named datasources would need to start even if the default datasource is not configured.
To get out of the current situation, I would like to suggest that we change this behavior:
  1. I'm working on [3] to introduce a way to *explicitly* mark a datasource as "inactive" at runtime. This should address the first use case above (and more, but that's beside the point).
  2. We will no longer define a default datasource at build time if a named datasource is configured, and there is no configuration for the default datasource. This should make sure we don't break the second use case above.
    It's also similar to what we do for persistence units in Hibernate ORM/Reactive [4]
  3. If a datasource defined at build time (default or named) is unconfigured at runtime, we'll fail on startup and recommend configuring that datasource or deactivating it.
Item 2 might require some changes in Smallrye Config as it seems a Map using `@WithUnnamedKey` [5] [6] will currently always contain the unnamed key... at least in some cases. Hopefully that's something that can be addressed, but this might delay the change past the 3.7 feature freeze. Which I guess could be a good thing anyway.

Back to the proposal: any thoughts, opinions on the general approach?

Makes sense to me

Thanks,


Yoann Rodière
Hibernate Team

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAEqagprg%2BnWLhp%2BtCDqAg5jnFiDYczSN-uZENOP-v3Puf2rf7w%40mail.gmail.com.

Georgios Andrianakis

unread,
Dec 12, 2023, 3:47:04 AM12/12/23
to Guillaume Smet, yo...@hibernate.org, Quarkus Development mailing list
FWIW, when I encountered this in the past, I was surprised by the behavior but never made the connection that it only happened when Hibernate was not used.

On Tue, Dec 12, 2023 at 10:45 AM Guillaume Smet <guillau...@gmail.com> wrote:
Just to give some context: we worked hard to make sure that adding an extension wouldn't "break" an application. And that was a conscious decision.

Now we could change that but that's not a decision that is so obvious.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-mdT-rTZy9cGNkJWDAqpqR-v5i%2B%3DVyNP-VvNB6cBYvBew%40mail.gmail.com.

Guillaume Smet

unread,
Dec 12, 2023, 4:17:26 AM12/12/23
to Quarkus Development mailing list
(Retrying as my email got rejected...)

Just to give some context: we worked hard to make sure that adding an extension wouldn't "break" an application. And that was a conscious decision.

Now we could change that but that's not a decision that is so obvious.

Yoann Rodiere

unread,
Dec 12, 2023, 4:26:29 AM12/12/23
to guillau...@gmail.com, Quarkus Development mailing list
> Now we could change that but that's not a decision that is so obvious.

Believe me, I realize that, having banged my head against the wall for the past few days trying to find a different solution. The current behavior is simply incompatible (by design) with providing early failures when configuration is missing.

In short: you can't create a default datasource implicitly (without any config) at build time and fail on missing runtime config, while also expecting applications that don't provide any config to start in production without any failure. Because such applications would have an implicit default datasource that lacks runtime config.

To clarify the consequences, I think my proposal would require changes to applications in the following cases:
  1. [This is the point of the change]
    Applications that have build-time configuration for a datasource (default or named) but no runtime configuration for that datasource, and do not use that datasource.
    For those, Quarkus would fail at start because the datasource is defined at build time but not configured at runtime.
    The solution would be to explicitly deactivate the datasource at runtime. Instruction could be included in the error message on startup.
  2. [This is a necessary consequence of having an implicit default datasource (no db-kind)]
    Applications that have no build-time configuration at all for any datasource, and no runtime configuration for the default datasource.
    For those, Quarkus would fail at start because the default datasource is defined at build time but not configured at runtime.
    The solution would be to explicitly deactivate the datasource at runtime (quarkus.datasource.active = false). Or we could provide a way to disable the extension at runtime (quarkus.datasource.enabled = false). Instruction could be included in the error message on startup.
  3. [This is necessary if we want to avoid breaking applications that only use named datasources, which hopefully are more frequent than applications that use both the default and named datasources]
    Applications that have build-time configuration for a named datasource, but no build-time configuration for the default datasource, and use the default datasource at runtime.
    For those, Quarkus would not create the default datasource during the build/static init, which would result in build/static init failures, or runtime failures in worst cases (people using `Arc.container()` to retrieve the Datasource bean).
    The solution would be to provide build-time settings for the default datasource, like we mandate for named datasources. Essentially we'd go from "you need build-time settings (db-kind) for named datasources" to "you need build-time settings for all datasources as soon as you use named datasources" (the behavior of the Hibernate ORM extension for persistence units).

Alternatives to my proposal:
  • Status quo: applications fail late when there's missing configuration. And by "late", I mean, possibly a few minutes after a deployment in production was deemed successful.
  • Require explicit configuration at build time for all datasources, including the default one: we no longer create a default datasource implicitly, ever.

Yoann Rodière
Hibernate Team

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/8b8d2e78-72c4-413f-91b8-d2d4cfe51687n%40googlegroups.com.

Max Rydahl Andersen

unread,
Dec 12, 2023, 4:35:45 AM12/12/23
to yo...@hibernate.org, guillau...@gmail.com, Quarkus Development mailing list
Interesting conundrum.

How if anything will this affect dev mode ? can I still run quarkus dev and be able to fix it without restart?

/max

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAEqagprCcW7etS1%3D%2BDYgkdWKZDdhp4qjYFcocw8EJCqOF%3D-oPw%40mail.gmail.com.


--

Stephane Epardaud

unread,
Dec 12, 2023, 4:44:21 AM12/12/23
to mand...@redhat.com, yo...@hibernate.org, guillau...@gmail.com, Quarkus Development mailing list
How does an application look when it's using a datasource without Hibernate?

Sergey Beryozkin

unread,
Dec 12, 2023, 4:48:40 AM12/12/23
to stephane...@gmail.com, mand...@redhat.com, yo...@hibernate.org, guillau...@gmail.com, Quarkus Development mailing list
Sorry if it is not too related, in OIDC, if the connection has not been established at startup, an attempt will be done to re-establish it at the first request, as we had many issues when a test container was an extra sec too slow at startup so the early connection failed, etc

Thanks Sergey

On Tue, Dec 12, 2023 at 9:44 AM Stephane Epardaud <stephane...@gmail.com> wrote:
How does an application look when it's using a datasource without Hibernate?

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

Yoann Rodiere

unread,
Dec 12, 2023, 5:02:56 AM12/12/23
to Stephane Epardaud, mand...@redhat.com, guillau...@gmail.com, Quarkus Development mailing list
> How if anything will this affect dev mode ?

In dev mode, assuming you have dev services for you database, a datasource always has runtime config as soon as it has build-time config.
So, the problem I'm trying to solve does not really affect dev mode.

The proposal will affect dev mode for applications in case 3 in my previous message:

> [This is necessary if we want to avoid breaking applications that only use named datasources, which hopefully are more frequent than applications that use both the default and named datasources]
> Applications that have build-time configuration for a named datasource, but no build-time configuration for the default datasource, and use the default datasource at runtime.
> For those, Quarkus would not create the default datasource during the build/static init, which would result in build/static init failures, or runtime failures in worst cases (people using `Arc.container()` to retrieve the Datasource bean).

> can I still run quarkus dev and be able to fix it without restart?

As I said above if you use dev services, you probably don't need to fix anything, as the runtime config is there.

And In cases impacted by this change, or if you don't use dev services, startup will fail for a datasource without runtime config, so yes you'll need to restart.
I think that's an area where we'd rather have behavior similar to prod, no?

Note I didn't see any specific code related to hot reload in the datasource/agroal extensions,  so as far as I understand Quarkus will restart if you change any build-time or runtime datasource config anyway.

> How does an application look when it's using a datasource without Hibernate?

Bad, of course :D

More seriously:
  1. I know some applications do use JDBC directly, and they probably look very verbose and generally awful, but to each their own.
  2. For reactive datasources that's probably more common, more legitimate, and easier on the eye as the reactive SQL clients offer a more concise API.
  3. There are extensions in Quarkus Core that use JDBC directly without Hibernate, for things like security or storing the transaction log.
  4. There are probably Quarkiverse extensions that provide a different framework on top of JDBC, though I don't personally know any.
> Sorry if it is not too related, in OIDC, if the connection has not been established at startup, an attempt will be done to re-establish it at the first request, as we had many issues when a test container was an extra sec too slow at startup so the early connection failed, etc

I don't think that's related; we're talking about internal consistency here: we will fail if a JDBC URL isn't set for the datasource, regardless of whether the database is reachable or not.

Yoann Rodière
Hibernate Team

Sergey Beryozkin

unread,
Dec 12, 2023, 5:08:46 AM12/12/23
to yo...@hibernate.org, Stephane Epardaud, mand...@redhat.com, guillau...@gmail.com, Quarkus Development mailing list
Thanks for the clarification

Sergey

 

Yoann Rodière
Hibernate Team


On Tue, Dec 12, 2023 at 10:44 AM Stephane Epardaud <stephane...@gmail.com> wrote:
How does an application look when it's using a datasource without Hibernate?

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

Stephane Epardaud

unread,
Dec 12, 2023, 5:18:45 AM12/12/23
to Yoann Rodiere, mand...@redhat.com, guillau...@gmail.com, Quarkus Development mailing list


On Tue, 12 Dec 2023 at 11:02, Yoann Rodiere <yo...@hibernate.org> wrote:
Bad, of course :D

Haha :)
 
More seriously:
  1. I know some applications do use JDBC directly, and they probably look very verbose and generally awful, but to each their own.
  2. For reactive datasources that's probably more common, more legitimate, and easier on the eye as the reactive SQL clients offer a more concise API.
  3. There are extensions in Quarkus Core that use JDBC directly without Hibernate, for things like security or storing the transaction log.
  4. There are probably Quarkiverse extensions that provide a different framework on top of JDBC, though I don't personally know any.

So, it seems to me that we could broaden the Hibernate contract, which boils down to being able to say that an extension depends on a datasource being configured, and thus refuse to start. At the moment, only Hibernate does that. But we could extend it to our Reactive SQL client extension, or the security extention or transaction log storage extension _if_ they are configured to require a datasource, and similarly for other extensions.

This would generalise the Hibernate behaviour to any extension, and would allow users to add the jdbc/datasource extension without breaking anything, if they are not used, and only as long as we can prove (via build items) that they are not used.

Or am I off topic?
--
Stéphane Épardaud

Yoann Rodiere

unread,
Dec 12, 2023, 6:28:17 AM12/12/23
to Stephane Epardaud, mand...@redhat.com, guillau...@gmail.com, Quarkus Development mailing list
> This would generalise the Hibernate behaviour to any extension, and would allow users to add the jdbc/datasource extension without breaking anything, if they are not used, and only as long as we can prove (via build items) that they are not used.
>
> Or am I off topic?

Not quite :)

But the thing is, Reactive SQL client extensions *are* the datasource extensions. They don't *use* the datasources, they define them and provide them for use directly.
Similarly, people using JDBC directly also only rely on the Agroal extension.
There's no other extension at play there, so just having those extensions present doesn't tell us anything.

So we would also have to detect whether the CDI bean for a Datasource or Reactive SQL client is injected anywhere at build time.
We'd effectively make the datasource beans removable, where they used to be unremovable. With the consequences we know on people who rely exclusively on `Arc.container().instance(DataSource.class)`.
Same for applications that don't inject the datasource at all, but only rely on some side-effects of their existence, like health checks; though I suppose that's a weaker argument.

Maybe that's a gentler way of breaking applications, indeed?

Yoann Rodière
Hibernate Team

Stephane Epardaud

unread,
Dec 12, 2023, 9:24:14 AM12/12/23
to Yoann Rodiere, mand...@redhat.com, guillau...@gmail.com, Quarkus Development mailing list
On Tue, 12 Dec 2023 at 12:28, Yoann Rodiere <yo...@hibernate.org> wrote:
But the thing is, Reactive SQL client extensions *are* the datasource extensions. They don't *use* the datasources, they define them and provide them for use directly.
Similarly, people using JDBC directly also only rely on the Agroal extension.
There's no other extension at play there, so just having those extensions present doesn't tell us anything.

OK, too bad.
 
So we would also have to detect whether the CDI bean for a Datasource or Reactive SQL client is injected anywhere at build time.
We'd effectively make the datasource beans removable, where they used to be unremovable. With the consequences we know on people who rely exclusively on `Arc.container().instance(DataSource.class)`.
Same for applications that don't inject the datasource at all, but only rely on some side-effects of their existence, like health checks; though I suppose that's a weaker argument.

Can we detect that those beans are injected at build time, then? Of course, not via the API, but via @Inject. If we can, then we can fail startup with the same meaningful message as Hibernate, no? Isn't that what we're looking for, rather than make them removable? 

Yoann Rodiere

unread,
Dec 12, 2023, 9:33:58 AM12/12/23
to Stephane Epardaud, Martin Kouba, mand...@redhat.com, guillau...@gmail.com, Quarkus Development mailing list
> Can we detect that those beans are injected at build time, then?

@Martin Kouba will confirm, but at the very least the code to detect whether a bean is injected is already present in Quarkus or Arc; I don't know how it would remove beans otherwise.
There may even be a build item for other extensions to inspect injection points already; I remember dealing with something like that.

> If we can, then we can fail startup with the same meaningful message as Hibernate, no? Isn't that what we're looking for, rather than make them removable?

Right. If a datasource is active and unconfigured, we'd fail on startup if we know it's injected somewhere, or deactivate it automatically and fail on first use if it's only "injected" through CDI's dynamic API.

That would probably work... Not a perfect solution, but probably safer for 99% of use cases.

Yoann Rodière
Hibernate Team

Max Rydahl Andersen

unread,
Dec 12, 2023, 9:37:04 AM12/12/23
to Yoann Rodiere, Stephane Epardaud, guillau...@gmail.com, Quarkus Development mailing list


/max
https://xam.dk/about

On 12 Dec 2023, at 11:02, Yoann Rodiere wrote:

> As I said above if you use dev services, you probably don't need to fix anything, as the runtime config is there.
>
> And In cases impacted by this change, or if you don't use dev services, startup will fail for a datasource without runtime config, so yes you'll need to restart.
>
> I think that's an area where we'd rather have behavior similar to prod, no?

yes and no...its okey it fails but it should not exit...at least try to avoid it.

so be clear what we aim for is that you can run quarkus:dev on an app and even if it is broken or does break that the user
can change the config, add/remove deps, fix the code and then devmode will pick up those and recover because of the dev mode restart.

> Note I didn't see any specific code related to hot reload in the datasource/agroal extensions,  so as far as I understand Quarkus will restart if you change any build-time or runtime datasource config anyway.

yeah, its just that the failure should not be hard enough failure to break devmode :)

/max

Yoann Rodiere

unread,
Dec 12, 2023, 9:42:36 AM12/12/23
to Max Rydahl Andersen, Stephane Epardaud, guillau...@gmail.com, Quarkus Development mailing list
> > Note I didn't see any specific code related to hot reload in the datasource/agroal extensions,  so as far as I understand Quarkus will restart if you change any build-time or runtime datasource config anyway.
>
> yeah, its just that the failure should not be hard enough failure to break devmode :)

Then I'll reassure you: I don't intend to call `System.exit` :]

The "failure" here would just be a ConfigurationException thrown during a CDI bean initialization or by a runtime bytecode recorder, like we already have many in the codebase. Nothing new or exotic.

Yoann Rodière
Hibernate Team

Martin Kouba

unread,
Dec 12, 2023, 10:14:03 AM12/12/23
to yo...@hibernate.org, Stephane Epardaud, mand...@redhat.com, guillau...@gmail.com, Quarkus Development mailing list

On 12. 12. 23 15:33, Yoann Rodiere wrote:
> > Can we detect that those beans are injected at build time, then?
>
> @Martin Kouba <mailto:mko...@redhat.com> will confirm, but at the very
> least the code to detect whether a bean is injected is already present
> in Quarkus or Arc; I don't know how it would remove beans otherwise.
> There may even be a build item for other extensions to inspect injection
> points already; I remember dealing with something like that.

BeanDiscoveryFinishedBuildItem#getInjectionPoints() if you're interested
in "class-based" injection points or
SynthesisFinishedBuildItem#getInjectionPoints() if you also need
synthetic injection points as well.

And for both build items there's also the
#getBeanResolver().resolveBeans() method that can help with type-safe
resolution rules.

>
> > If we can, then we can fail startup with the same meaningful message
> as Hibernate, no? Isn't that what we're looking for, rather than make
> them removable?
>
> Right. If a datasource is active and unconfigured, we'd fail on startup
> if we know it's injected somewhere, or deactivate it automatically and
> fail on first use if it's only "injected" through CDI's dynamic API.
>
> That would probably work... Not a perfect solution, but probably safer
> for 99% of use cases.
>
> Yoann Rodière
> Hibernate Team
> yo...@hibernate.org <mailto:yo...@hibernate.org>
> --
> You received this message because you are subscribed to the Google
> Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to quarkus-dev...@googlegroups.com
> <mailto:quarkus-dev...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/quarkus-dev/CAEqagpp8dHJzudhKK7pfK0kts5WkS7JxtQVo4Hdu_L0KT_jwPg%40mail.gmail.com <https://groups.google.com/d/msgid/quarkus-dev/CAEqagpp8dHJzudhKK7pfK0kts5WkS7JxtQVo4Hdu_L0KT_jwPg%40mail.gmail.com?utm_medium=email&utm_source=footer>.

--
Martin Kouba
Principal Software Engineer
Red Hat, Czech Republic

Reply all
Reply to author
Forward
0 new messages