Some Consul config enhancements

[Ivan St. Ivanov]

Hi folks,

Last week I started experimenting with the Consul config Quarkus extension.

I am missing some little features there:

1) I would like to add an optional refresh config property, which would basically contain a value in seconds that the config source is refreshed from Consul (if 0 or missing, don't try to refresh)

2) I'd also like to have a new value type (let's call it FOLDER for now). When a path is passed to it, that path won't be treated neither as property nor as raw value, but as a folder that contains raw values. And everything below that path will be scanned recursively and accumulated into the resulting config map, with probably all the folders along the way concatenated in the key separated by .

As you can see, these are just raw ideas and need to be further fleshed out. I just want to see whether in your opinion they make any sense at all.

If you think they do, then I will happily create the GH issues and try to contribute them.

Cheers,

Ivan

[Georgios Andrianakis]

Hi,

Thanks for your suggestions! I'll reply inline.

On Mon, Mar 29, 2021 at 2:04 PM Ivan St. Ivanov <ivan.st...@gmail.com> wrote:

Hi folks,

Last week I started experimenting with the Consul config Quarkus extension.

I am missing some little features there:

1) I would like to add an optional refresh config property, which would basically contain a value in seconds that the config source is refreshed from Consul (if 0 or missing, don't try to refresh)

We have discussed this before in various contexts and have agreed not to pursue this avenue.

2) I'd also like to have a new value type (let's call it FOLDER for now). When a path is passed to it, that path won't be treated neither as property nor as raw value, but as a folder that contains raw values. And everything below that path will be scanned recursively and accumulated into the resulting config map, with probably all the folders along the way concatenated in the key separated by .

I like this idea, please open a GH issue requesting it.

As you can see, these are just raw ideas and need to be further fleshed out. I just want to see whether in your opinion they make any sense at all.

If you think they do, then I will happily create the GH issues and try to contribute them.

That would be marvelous :)

Cheers,

Ivan

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CACYLA9HRhTpsZisQjNWuF-MGZUxWMvkyJJL_r5Nk4SijDEhD9A%40mail.gmail.com.

[Ivan St. Ivanov]

Thanks for the quick response Georgios!

I'll open the GH issue for the folder value type, sure.

As for the refresh, if you don't want it implemented in the extension, what is the proposed solution for applications that don't want to restart for every config change?

Cheers,

Ivan

[Georgios Andrianakis]

On Mon, Mar 29, 2021 at 2:54 PM Ivan St. Ivanov <ivan.st...@gmail.com> wrote:

Thanks for the quick response Georgios!

I'll open the GH issue for the folder value type, sure.

+1

As for the refresh, if you don't want it implemented in the extension, what is the proposed solution for applications that don't want to restart for every config change?

We have agreed that we don't want to have this functionality in Quarkus and that this responsibility for this should be delegated to the platform that runs the application.

[Roberto Cortez]

Hi Ivan,

I guess you could implement your own ConfigSource that is able to refresh the configuration. On the other hand, since you are using the extension you probably don’t want to have that extra work, considering that your configuration is coming from Consul.

The issue with dynamic configuration is that you are not able to distinguish what can be reloaded or not. For instance, you can’t expect Hibernate to change its configuration in-flight. Some configuration are just not safe to change dynamically. I suppose that you are interested on your own configuration, which you manage and you know it is safe, but we didn’t want to confuse users and allow configuration reloads on unsafe configuration.

Hope this helps!

Cheers,

Roberto

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CACYLA9HNB0Gr_kJphSS6XA8cdpz7GxAUsRdgpfQCff3-b_TWhQ%40mail.gmail.com.

[Ivan St. Ivanov]

Hi Roberto,

I fully understand this :)

I am just wondering what would be the best approach for both using the Consul extension (and not implementing my own config source provider) and at the same time refresh my config at a certain interval.

Thanks,

Ivan

[Roberto Cortez]

Unfortunately, for the Consul Config Source, the values are read at startup and it seems that the APIs are not available to access them to force a manual refresh.

What we can probably do is to improve the API to provide a way for users to be able to implement their refresh mechanism.

Would you like to try out a PR for it? :)

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CACYLA9GxVZ2G-2pVe4nYyQp-kgo%2BJzTZG8X_RyUy4XoFMoq6ow%40mail.gmail.com.

[Loïc MATHIEU]

Hi,

I'm too interested in a way to have a config property refreshed inside my apps.

I do understand that it's not desirable that a full config source would be refreshed

So maybe what we need to have is an ability to inject a Watch object that we can subscribe on to execute callback when it changes.

This would allow a more complex use case.

This will not be too trivial so people will not abuse it ;)

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/1D9488FE-F93A-4A5E-8DD5-23C391D35619%40yahoo.com.

[Loïc MATHIEU]

If the consul config Quarkus extension uses the official consul java client and provides a consul client inside the CDI container, we can inject it and uses the API like described here: https://github.com/rickfast/consul-client#example-5-subscribe-to-value-change

[Georgios Andrianakis]

It does not use the official consul client - this part of Quarkus needs to run before the CDI is fully initialized (i.e. in bootstrap config).

The implementation basically just does HTTP requests to Consul and converts the results

[Antonio Goncalves]

Hi all,

I was watching the Quarkus Insight episode with Phillip Kruger on OpenAPI[1]. There is this moment where he customizes the OpenAPI contract with properties in the application.properties. The properties are[2]:

mp.openapi.extensions.smallrye.info.title=ACME online store API
mp.openapi.extensions.smallrye.info.version=1.0.0
mp.openapi.extensions.smallrye.info.description=We make everything, and sell it online
mp.openapi.extensions.smallrye.info.contact.email=i...@acme.com
mp.openapi.extensions.smallrye.info.contact.name=ACME IT
mp.openapi.extensions.smallrye.info.contact.url=https://www.acme.com
mp.openapi.extensions.smallrye.info.license.name=Apache 2.0
mp.openapi.extensions.smallrye.info.license.url=http://www.apache.org/licenses/LICENSE-2.0.html

I didn't know about these properties so I looked in the Quarkus Configuration guide[3] but couldn't find them. So I thought, maybe in the Dev UI, but they are not there either. Dev UI only shows:

mp.openapi.extensions.smallrye.defaultProduces
mp.openapi.extensions.smallrye.defaultConsumes

It's not the first time that I look around for a MicroProfile property and can't find it. The only way is to look at the code[4].

It would be very helpful to have all the available properties on the Dev UI. The Quarkus properties, of course, but also the properties of the extensions that are being used in the project. 

WDYT ?

Antonio

[1] https://www.youtube.com/watch?v=8JsnjHQM36k

[2] https://github.com/phillip-kruger/openapi-example/blob/main/src/main/resources/application.properties#L14

[3] https://quarkus.io/guides/all-config

[4] https://github.com/smallrye/smallrye-open-api/blob/main/core/src/main/java/io/smallrye/openapi/api/constants/OpenApiConstants.java#L42

--

Antonio Goncalves 
Contractor, Java Champion and Pluralsight author

Blog | Twitter | LinkedIn | Author | Pluralsight | Devoxx France | Voxxed Microservices

[Ivan St. Ivanov]

I just created this issue for the recursive properties loading: https://github.com/quarkusio/quarkus/issues/16095

Maybe someone can assign it to me (I'm not sure I have the permissions to do it)?

I'd be happy to implement the refresh as well after that, although I am not sure I understand the approaches that Roberto and Loic proposed.

Cheers,

Ivan

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-kS7jjufaODjTMvc7O18UnusrC2bOF-YGppobV%2BHC6akg%40mail.gmail.com.

[Roberto Cortez]

Hi Antonio,

We should definitely improve this.

We do document and provide all properties known to us in the docs and the dev console. Unfortunately, we cannot guess configuration if it is now provided to us. I’m not sure if this is the case for the OpenAPI extension, but we will check.

Thanks!

Cheers,

Roberto

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAKawoOZHRdi-NfKAVCJ1pXMwpwhivGHDCbq79mEd0QJLatgh-Q%40mail.gmail.com.

[Phillip Kruger]

Hi Antonio.

We can maybe make aliases for these in Quarkus ?

In MP Open API, vendors are required to follow that pattern (see https://download.eclipse.org/microprofile/microprofile-open-api-1.1.2/microprofile-openapi-spec.html#_vendor_extensions)

@Roberto, w.d.y.t ? Is there a way to expose those in Quarkus documentation, or should I map them as 'quarkus.smallrye-openapi.*' properties ?

Phillip

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/87C6211E-3854-4EA4-B92D-53D9D77839CE%40yahoo.com.

[Phillip Kruger]

You can also try https://github.com/microprofile-extensions/config-ext/blob/main/configsource-consul/README.md

Not sure if this works (especially in native), but you can give it a try.

Cheers

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CACYLA9HovaZWw8Qc%3Dg5w9_8n8-zaHsEDLx%3DGqgnNT-eBMjy6ZQ%40mail.gmail.com.

[Roberto Cortez]

Everything that goes to Quarkus All Config is mapped with a ConfigRoot so we can also set the javadoc for the documentation. Producing a ConfigDescriptionBuildItem should also work. Right now, this is the way to expose them. 

I remember we had some conversations in the past where dependencies would provide a schema of their configuration so we could consume that and expose it in Quarkus, but we haven’t done any work on in yet. Maybe we need to look into it again.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAEyd7D4t90-vLPcdqbHC7Ta%2Bb-FKT6W5g84our1GHWvo%3DBgbbg%40mail.gmail.com.

[Phillip Kruger]

So what do you think, should I do a ConfigDescriptionBuildItem that describe the mp.* config items, or should I add new quarkus config that maps to the mp.* ?

[Knut Wannheden]

Hi,

On Mon, Mar 29, 2021 at 5:38 PM Phillip Kruger <phillip...@gmail.com> wrote:

So what do you think, should I do a ConfigDescriptionBuildItem that describe the mp.* config items, or should I add new quarkus config that maps to the mp.* ?

By "add new quarkus config that maps to the mp.*" I assume you mean you would like to define a new @ConfigRoot annotated class. IMHO that would be very nice but I think it would require https://github.com/quarkusio/quarkus/issues/9791: Add a new "namespace" attribute to the @ConfigRoot annotation and everything that comes with it.

I would still be very much interested in this and would also be willing to work on a PR for this. Would a PR for this have a chance of getting accepted in the near future?

Knut

 

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAEyd7D4-1Kq3_iRv8EBE41ONgs9qd2w9SLD6%3DVt_eT5HO%3DrkAw%40mail.gmail.com.

[Phillip Kruger]

No what I mean is to expose that same config in the Quarkus namespace, then in the extension map that to the mp namespace to the SmallRye can work. The nice thing about this is that the config is all quarkus.*

[Roberto Cortez]

I think we need to discuss this more broadly.

This came out before:

https://quarkusio.zulipchat.com/#narrow/stream/187038-dev/topic/Can.20smallrye.20properties.20be.20seen.20as.20quarkus.2E.20properties.20.3F/near/229877679

https://github.com/quarkusio/quarkus/issues/16099#issuecomment-809512681

Ideally, we should have a may to make this more transparent. The ConfigRoot vs ConfigDescriptionBuildItem is that in the ConfigRoot you use types and validate these, while the ConfigDescriptionBuildItem is only for the documentation. Also, we should keep all of these configs in the quarkus namespace for consistency, even if sometimes other properties do work, because they also use MP Config.

For now, I would advice to just add them as a ConfigRoot while we keep the discussion going.

Knut,

I was not aware of the linked issue (it was missing the config label), so let me have a look.

Cheers,

Roberto

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAEyd7D4S8H2u0RW1BaVnpxjrh1bw3v90XnPR0EDxxLRshveSmQ%40mail.gmail.com.

[Phillip Kruger]

I created https://github.com/quarkusio/quarkus/pull/16120 for this. Let me know what you think.

Cheers

[Ivan St. Ivanov]

Thanks Phillip!

The consul MicroProfile extension did the trick for me! It supports both - config refresh and recursive loading of properties.

The only inconvenience (although not a big one!) is that it doesn't know about Quarkus's application.properties. So I had to create META-INF/microprofile-config.properties.

But overall it does a great job for me!