Kafka via annotation rather than config

[Stuart Douglas]

Hi Everyone,

Something that has been bugging me for a while is that if I am using Kafka I need to use more config than most other extensions, as I need to configure the connections in application.properties.

IMHO in a lot of cases this 'config' is not really something that will be overridden, but is a fundamental feature of the behaviour of the application (e.g. the serializer and deserializer types are not the sort of thing that you would expect to override, it's not really config, it's just part of the app).

To work around this I was thinking it could be useful to provide some extension annotations that allow for Kafka to be configured via annotation at the usage site, rather than in the config file.

Basically the annotations would look something like this (but with all the kafka options, not just the ones used in the quickstart):

https://github.com/quarkusio/quarkus/compare/main...stuartwdouglas:kafka-annotations?expand=1#diff-ddc2a4a4c65f127e000b34008b09a69a8a816d0fdfcce5ed26dcd12616d3a4e3R9

You could then just apply them at the usage site and use them to configure Kafka:

https://github.com/quarkusio/quarkus-quickstarts/commit/ea7477daafb03e60b4d92cd1eee3b353faf51252#diff-ecc24c31bb8a851dbe29295861101bdfa723ae1a3cc2df2ce820466d3e6a1ef4R27

Internally this should just output the relevant config, to still allow everything to be overridden via config as it can be now, but IMHO this is a much nicer and more user friendly way to wire up the endpoints.

Does this sound worthwhile? I think the implementation is pretty simple so I am happy to take it on if this is something that we think we want.

Stuart

[Georgios Andrianakis]

+1 from me.

I always find the Kafka config quite cumbersome

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cymm16F8id%3DY3zZZGL0ybA259o8EPYhpC7F-ksZ3iQo%3DQ%40mail.gmail.com.

[Emmanuel Bernard]

How much usage side == channel is the crux of the question as well as how much “usage site” is entirely captured by a build time discoverable construct. And how you can gracefully shift from usage site to channel config without triggering a lot of incomprehensible priority cases. 

--

[Ladislav Thon]

Serializers and deserializers should already be autodetected in like 99% of cases. (There's one possible improvement I'm aware of, tracked in https://github.com/quarkusio/quarkus/issues/18195.) There's no need for extra annotation -- this can be inferred directly from the @Incoming/@Outgoing method signature or @Channel injection point type.

When it comes to the other configs, I'm not sure all of them should be exposed via annotations, but it might be really nice for some of them.

LT

st 20. 10. 2021 v 8:49 odesílatel Emmanuel Bernard <eber...@redhat.com> napsal:

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANYWk7PWfNQZHWjK-iAUu%2B0Au9fJNQO7SCrkZC22ZvayLku1TQ%40mail.gmail.com.

[Loïc MATHIEU]

Currently, kafka configuration can be shared between channels, via quarkus.kafka.* properties.

The issue is also that Kafka configurations are:

- not exhaustive: you can pass arbitrary config to the serializers

- secured : password should not be displayed in source (they are handled specifically by the client)

- legion : will you create tens of annotations with tens of attributes in them ?

Maybe providing a named configuration bean instead would be more flexible. 

Annotations are for meta programmation so not designed for writing configuration with them IMHO.

But I agree that currently, configuring channels is painful so we can do better ;)

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALbocO%3DCgbTShWkkgvYOb56EzUGg1gVzbDpawbzJcxp-vXqUHA%40mail.gmail.com.

[clement escoffier]

-1 - I don’t want Kafka specific annotations. Kafka configuration are unbounded. You can’t list all parameters.

We have ongoing work to have no configuration required. It takes some time as we have to change things in reactive messaging.

Clement

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJLxjVHcGk3X7-rTpxfgzCqg5F4-vrY1txaWtuu44v6mzceOdw%40mail.gmail.com.

[Stephane Epardaud]

Definitely +1 on making config easier, I've always found it way too hard to configure.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAKW6ficwNqp8pcX4UKeeWk%3DxivNfm0dZ1YnsMTXCxziMjVxqXQ%40mail.gmail.com.

--

Stéphane Épardaud

[clement escoffier]

The initial work on the build time wiring has been merged.

So, no configuration required when you have a single connector on the class path. It will use the connector automatically.

Clement

[Stephane Epardaud]

Great news. But I still see lots of config that I didn't think I'd need in https://quarkus.io/version/main/guides/kafka-reactive-getting-started so perhaps the docs weren't updated?

--

Stéphane Épardaud

[Andy Damevin]

Should we update the codestart? https://github.com/quarkusio/quarkus/pull/21068

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAKU9E9vsKk3cwrMt2m_-%3DXuddg2zX741M3UUCovkjRdTxx%2Bdbw%40mail.gmail.com.

--

Andy Damevin

[clement escoffier]

Ah yes, we need to update the doc! 

Adding that to my list.

Clement