Running Kafka Streams application on YARN

[peter...@gmail.com]

Hi,

Is there any documentation / recommendation for packaging and deploying Kafka Streams applications to a YARN cluster?

I'd heavily use the local state store and I'm not sure how it is related to the container allocation in YARN.

Any suggestion / best practices would be great.

Peter

[Guozhang Wang]

Hi Peter,

Integrating with Slider would be the easiest way to do this. Currently we do not have a demo example for it, but we will probably add one in the future if people found it not very straight forward. 

Guozhang

[peter...@gmail.com]

Thank you Guozhang, I'll take a look at Slider then.

Peter

[Guozhang Wang]

Hi Peter,

Any experience with YARN that you'd like to share with us?

Guozhang

[peter...@gmail.com]

Hi Guozhang,

Not yet but we're about do some investigation.

My only concern is how this would work for KTables. When containers are

allocated to run the Kafka Stream instances each time the local Rocksdb store needs to be built up

from scratch. I'm not quite sure how much is the overhead in this case.

Peter