Roadmap Planning 2021

[Roberto Cortez]

Hi everyone,

We would like to start gathering some ideas to include in our Roadmap for 2021. These can include small things, like enhancements to the current projects, to bigger things like new projects and integrations with other APIs.

Fell free to add any ideas to this email thread.

Thanks!

Cheers,
Roberto

[Stuart Broad]

Hi Roberto,

I think it would be worth looking at trying to improve the performance of the kafka connectors. I have found that the method signatures that process messages sequentially are pretty slow even when the processing is quick e.g.:-

@Incoming void method(I p)

@Incoming Uni<?> method(Message<I> msg)

@Outgoing @Incoming Uni<Message<O>> method(Message<I> msg)

I'm not sure what can be done as I don't know too much about the source/sink code but perhaps messages could be read/ack'ed in batch? Maybe ack'ing individually might still be needed.

Less important but also an issue is the performance of the emitters.

For low throughput services we have found the above method signatures and emitters work just fine but for high throughput we start needing to do things like:-

1. Using a subscriber to request larger numbers of messages.
2. Using multiple channels on the same topic e.g. @Incoming("mychannel-in-1), @Incoming("mychannel-in-2) etc
3. Instead of using an emitter we use our own kafka producer

It feels a little like we need to twist the framework a bit to get the performance we need and I think this might hurt adoption as doing these above 3 things is not obvious (and maybe is not even the best thing to do).

Cheers,

Stuart

--
You received this message because you are subscribed to the Google Groups "SmallRye" group.
To unsubscribe from this group and stop receiving emails from it, send an email to smallrye+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/smallrye/3539F3D6-5F05-4A1D-B336-81AF7375846B%40yahoo.com.

[clement escoffier]

Hello,

Improving the Kafka support is on the roadmap. It's already work in progress I would say.

One of the problem is the overall serialization of the messages (which is derived from the Vert.x Kafka client), which makes concurrent processing convoluted. 

About acknowledgement, we are going to move to the throttled commit strategy as default in 2.5.0. It's going to provide quite a boost in term of acknowledgement management. We are still not there yet, many testing need to be done before we do the switch.

I'm more surprised by the emitter. Generally if behave well. One of the issue we have it that Kafka blocks after 5 in-flight writes. Because of this we limit the number of write to 5 (this is configurable). That may explain your observations. You can increase this number or avoid waiting for ack from the broker.

Clement

To view this discussion on the web visit https://groups.google.com/d/msgid/smallrye/CA%2B%3DWOw4bcAnu1YxSYNNq0COX_whHYhmjsOmevu%3DXfG3hy9hqMw%40mail.gmail.com.