Quarkus with Kafka & gRPC

[David Hoffer]

I'd like to prototype a new Quarkus app that uses Kafka as the internal message hub that will have potentially hundreds of topics that will receive & store incoming data.

Then to 'publish' that data to 'consumers' I would like to use gRPC for its more efficient protocol stack.  I can envision the 'consumers' getting data a couple of ways:

1. Batch requests.  Consumers would request batches of data from known Kafka offsets.

2. Streaming requests.  Consumers would receive streams of data from the Kafka topics either from a specified Kafka offset and/or from latest data in the topic.  E.g. if interested in the very latest messages they would stream to receive messages potentially forever.  

But in the streaming option, lots of questions arise such as flow control/etc.  What happens if client can't process at the rate new messages come in?  Who's job is it to manage this?  Should the client be required to scale out to handle what is provided?  Should the server slow down?  Or either/both?  How to manage all this?

Are there some examples of this sort of thing using Quarkus?  Any non Quarkus examples of this I could port to Quarkus perhaps?

If anyone has any pointers/suggestions on this please let me know.

-Dave

 

[clement escoffier]

Hello,

I don’t have pointers but gRPC streams are back-pressure aware and follow a protocol very close to reactive streams (which is the protocol used by reactive messaging).

So, if you forward requests from the gRPC streams to the Multi provided by Reactive Messaging, you should be good to go.

However, from what I can see, you may have to use the bare Kafka client, and then, you will need to implement the back-pressure yourself. It’s not an easy task. In the Reactive Messaging Kafka connector we track downstream requests and the size of a queue to pause/resume topic/partition consumption. Note that you must always call `poll`, pause just tells the broker that you are still there, but can’t consume yet. Also, if you follow that route, remember that all interactions with Kafka must be done from the same thread (the polling thread).

If your downstream consumers do not support back-pressure, the strategies are always the same: 

1. Buffer
2. Drop (latest or oldest)

Clement

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/63c6ad57-742a-4251-9f87-9b80747f3d3bn%40googlegroups.com.

[David Hoffer]

Why did you say "However, from what I can see, you may have to use the bare Kafka client, and then, you will need to implement the back-pressure yourself."?  Why can't I use the  Reactive Messaging Kafka connector?

-Dave