Kafka Streams and third party async APIs

[Martin Tyler]

I'd like to use a third party API within my Kafka Streams application. I need to make subscriptions to a service using this API - the subscriptions will be based on data incoming on a Kafka topic - and I will receive callbacks from this third party API which I want to forward on to a Kafka topic. I'm using the processor API.

So the input Kafka topic defines a set of data that I need to subscribe to and the async responses need to go back to Kafka.

I'm not sure if this is possible just in Kafka Streams though. The callback from the other API comes back on its own thread and calling context.forward() throws a NullPointer exception. I'm assuming you can only call context.forward() from within the Kafka Streams process() callback.

I can think of two solutions, neither of which are great:

1. Use punctuate() to send messages stored temporarily from the other api callback

2. Use the Producer API to send to the topic

Am I missing a better solution?

[Martin Tyler]

I've just realised that solution 1 won't work anyway - punctuate() doesn't get continuously called, it only gets called after a process() call.