How to deal with pass-through properties?

[Gunnar Morling]

Hi Quarksters,

As a follow-up to the initial Kafka Streams support I'm working on some related usability improvements [1]. One aspect of this is configuring KStreams through application.properties and I'm curious how this should look like.

I was thinking of supporting those options to be given like this in application.properties, using a common prefix (e.g. kafka.streams or kafka-streams, not sure what's recommended):

    kafka-streams.application.id=myapp
    kafka-streams.commit.interval.ms=20000
    kafka-streams.default.production.exception.handler=com.example.MyHandler

    ...

The options would be passed without the prefix to the KStreams engine.

Note there's many options for Kafka Streams itself, and also the underlying consumers and producers can be configured (e.g. consumer.send.buffer.bytes), so it's IMO not practical to have any hard-coded set of foreseen options in Quarkus.

I'd expect that it's fine for most options to require a rebuild when changing them, but some must be runtime configurable (most notably the Kafka bootstrap server(s) and the application id which is based on the specific instance's host and port and identifies the node in a KStreams cluster).

I already learned from Guillaume that I might need some kind of "config object", but details are blurry to me, esp. as it's about supporting an open range of pass-through options.

Thanks a lot for any advice,

--Gunnar

[1] https://github.com/quarkusio/quarkus/pull/2953

[Gunnar Morling]

So reading the extension guide, I found this:

    "The most useful configuration options should be exposed as quarkus.[extension]. instead of the natural namespace of the library. Less common properties can live in the library namespace."

Based on that I could see having just a few props prefixed with "quarkus.kafka-streams. ..." (and those would be runtime configurable), and the (open-ended) remainder under "kafka-streams. ..." (a rebuild would be required for changes). Would that make sense? It's a bit blurry where to draw the line, but it seems that'd be the prescribed way for Quarkus.

I'll proceed along these unless I hear about objections.

Thanks,

--Gunnar

[Stuart Douglas]

On Saturday, 29 June 2019 18:40:37 UTC+10, Gunnar Morling wrote:

So reading the extension guide, I found this:

    "The most useful configuration options should be exposed as quarkus.[extension]. instead of the natural namespace of the library. Less common properties can live in the library namespace."

Based on that I could see having just a few props prefixed with "quarkus.kafka-streams. ..." (and those would be runtime configurable), and the (open-ended) remainder under "kafka-streams. ..." (a rebuild would be required for changes). Would that make sense? It's a bit blurry where to draw the line, but it seems that'd be the prescribed way for Quarkus.

I'll proceed along these unless I hear about objections.

I think this makes sense. What do you mean by 'a rebuild would be required for changes'? Quarkus dev mode will pick up any changes to the file, so this should be handled automatically.

[Loïc MATHIEU]

Hello,

On Kafka, the configuration is managed via Properties objects that can be provided at client level, producer level and consumer level.

What we can do is to provide a way, in Quarkus, to create this Properties object from the config file and create separate config root for each that will be merged in a single Properties, for example:

quarkus.kafka.properties.bootstrap.servers=localhost:9092

quarkus.kafka.consumer.properties.client.id=consumer1

quarkus.kafka.consumer.properties.group.id=group

quarkus.kafka.producer.properties.acks=all

quarkus.kafka.producer.properties.buffer.memory=1024

quarkus.kafka.stream.<whatever>

Then, at runtime, properties are merged:

- For a producer: quarkus.kafka.properties.* and quarkus.kafka.producer.*

- For a consumer: quarkus.kafka.properties.* and quarkus.kafka.consumer.*

By this, when you are a consumer and a producer you can put properties in the global quarkus.kakfa.properties namespace to avoid duplicating them.

Regards,

Loïc

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
Visit this group at https://groups.google.com/group/quarkus-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/b9d5414c-6e5e-4008-9c92-4d5a67854dc6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Emmanuel Bernard]

Right so I think we have a bit fuzzy on what should land under quarkus.extension.* vs going outside.

Plus we do have exceptions like MicroProfile which defines some rules.

On top of that we said no no to pass through properties so far

So I was wondering if we could bring some rationality to this. Here is a proposal.

1. most common properties are exposed as quarkus.extension

    a. a benefit is that these properties will be documented, prepopulated in the application.properties and at some point autocompletable

2. as much as we can quarkus.extension.* properties should be build time and not runtime (that's the extension job)

3. for more complex properties or pass through properties, we encourage extension to use project.* properties (e.g. kafka-streams.*

4. [Open debate] some advanced properties can be build time as the extension developer permits but it's a harder effort potentially

Thoughts?

Emmanuel

PS: I'm sure the notion of Application Id would be generally useful for Quarkus if not already existing, we should just propagate that shared name or create a small extensions listening to a property and publishing it as a BuildItem.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
Visit this group at https://groups.google.com/group/quarkus-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CADGJaX8ETVOpPPhvf61YySj3-%3DbehV5Rng0%3D1AhMdWZ1_JdMjA%40mail.gmail.com.

[David Lloyd]

On Mon, Jul 1, 2019 at 8:59 AM Emmanuel Bernard <eber...@redhat.com> wrote:
>
> Right so I think we have a bit fuzzy on what should land under quarkus.extension.* vs going outside.
> Plus we do have exceptions like MicroProfile which defines some rules.
> On top of that we said no no to pass through properties so far
>
> So I was wondering if we could bring some rationality to this. Here is a proposal.
>
> 1. most common properties are exposed as quarkus.extension
> a. a benefit is that these properties will be documented, prepopulated in the application.properties and at some point autocompletable
> 2. as much as we can quarkus.extension.* properties should be build time and not runtime (that's the extension job)

This needs some nuance. We do want as much heavy lifting as possible
to be done during build time, and failing that, during static init;
but once that is done, we also should maximize the number of
properties that are run time configurable, because every property that
is run time configurable means one less reason to make the user
rebuild. So if the cost of having a given property be run time
configurable is minimal to none, it should be run time configurable.

In other words I would be cautious about documenting the assumption
that maximizing what is done at build time includes making config
properties build time only, because that is not always true. For a
concrete example, we could hard-code the thread pool sizes at build
time, but the amount of overhead of making them run time configurable
is almost nil and so it doesn't really make sense to force them into
build time.

> 3. for more complex properties or pass through properties, we encourage extension to use project.* properties (e.g. kafka-streams.*

I would strengthen the wording: I think the extension developer should
make every effort to use `quarkus.*`-namespaced properties for *all*
features. The only reason an extension should be using other
namespaces would be if they're facing a hard compatibility requirement
or a specification requirement IMO.

> To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANYWk7NjRjBN8TU4n2VfOW_3bWVqWAQY0GyattMHqC8mCUseKA%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.

--
- DML

[Gunnar Morling]

> What do you mean by 'a rebuild would be required for changes'?

> Quarkus dev mode will pick up any changes to the file, so this 

> should be handled automatically.

I didn't mean dev mode, but the final artifact you've built, i.e. the JAR or the native binary.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
Visit this group at https://groups.google.com/group/quarkus-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/b9d5414c-6e5e-4008-9c92-4d5a67854dc6%40googlegroups.com.

[Gunnar Morling]

Am Mo., 1. Juli 2019 um 18:19 Uhr schrieb David Lloyd <david...@redhat.com>:

On Mon, Jul 1, 2019 at 8:59 AM Emmanuel Bernard <eber...@redhat.com> wrote:
>
> Right so I think we have a bit fuzzy on what should land under quarkus.extension.* vs going outside.
> Plus we do have exceptions like MicroProfile which defines some rules.
> On top of that we said no no to pass through properties so far
>
> So I was wondering if we could bring some rationality to this. Here is a proposal.
>
> 1. most common properties are exposed as quarkus.extension
>     a. a benefit is that these properties will be documented, prepopulated in the application.properties and at some point autocompletable
> 2. as much as we can quarkus.extension.* properties should be build time and not runtime (that's the extension job)

This needs some nuance.  We do want as much heavy lifting as possible
to be done during build time, and failing that, during static init;
but once that is done, we also should maximize the number of
properties that are run time configurable, because every property that
is run time configurable means one less reason to make the user
rebuild.  So if the cost of having a given property be run time
configurable is minimal to none, it should be run time configurable.

Yes, I'd also lean towards that.

Esp. as it seems potentially confusing that some props would be runtime-configurable and some not. In the Kafka Streams example, none of the properties really apply at build time, while some only really can be given at runtime (most notably the application.server value which depends on the host name). But all the props are passed at once to Kafka Streams (i.e. during start-up), so there's no point really in fixing any props at build time.

In other words I would be cautious about documenting the assumption
that maximizing what is done at build time includes making config
properties build time only, because that is not always true.  For a
concrete example, we could hard-code the thread pool sizes at build
time, but the amount of overhead of making them run time configurable
is almost nil and so it doesn't really make sense to force them into
build time.

> 3. for more complex properties or pass through properties, we encourage extension to use project.* properties (e.g. kafka-streams.*

I would strengthen the wording: I think the extension developer should
make every effort to use `quarkus.*`-namespaced properties for *all*
features.  The only reason an extension should be using other
namespaces would be if they're facing a hard compatibility requirement
or a specification requirement IMO.

My problem with the quarkus.* namespace is that I'd have to pre-define all them (IIUC), which in case of KStreams would be a) lots of error-prone work and b) be some kind of catch-up game whenever upstream adds new props. Also naming style slightly differs, e.g. the expectation of Quarkus would be that it is quarkus.kafka-streams.application-id (dash), whereas the upstream prop is named applicationId. This might cause some friction e.g. when copying existing configuration files to Quarkus or just applying examples from upstream docs.

[Emmanuel Bernard]

On Mon, Jul 1, 2019 at 6:19 PM David Lloyd <david...@redhat.com> wrote:

On Mon, Jul 1, 2019 at 8:59 AM Emmanuel Bernard <eber...@redhat.com> wrote:

> 3. for more complex properties or pass through properties, we encourage extension to use project.* properties (e.g. kafka-streams.*

I would strengthen the wording: I think the extension developer should
make every effort to use `quarkus.*`-namespaced properties for *all*
features.  The only reason an extension should be using other
namespaces would be if they're facing a hard compatibility requirement
or a specification requirement IMO.

 

This is a deviation for our initial philosophy. Today we do not expose all 300 (arbitrary number) Hibernate properties, we try to make choices for the user. If we end up exposing 50-300 properties per extension, we are back in the space shuttle management analogy.

[David Lloyd]

Maybe. But having everything in config objects comes with a host of
benefits: better validation, type safety, documentation, default
values, one place to configure everything. The space shuttle problem
could be mitigated using other mechanisms. For example we could mark
configuration items or groups as "advanced", where they are not
included in generated configuration files, and would appear in
documentation in a different section. We could even have "hidden"
items which have source documentation but not published documentation
at all.

--
- DML

[Peter Palaga]

We are discussing the property naming policy in camel-quarkus as well.
Our current position is that we prefer configuration portability across
our big existing ecosystem (camel standalone, camel on spring boot,
camel-k, etc.). We would thus put only those propeties under the
quarkus.camel.* namespace, where we see a special reason for that - e.g.
that change of the given property requires a rebuild.

> 1. most common properties are exposed as quarkus.extension
> a. a benefit is that these properties will be documented, prepopulated in the application.properties and at some point autocompletable

I assume these benefits hold regardless of the given config's namespace;
those benefits are rather given by implementing the props via config
objects, right?

Thanks,

-- Peter