nsqd on Kubernetes when autoscale is enabled

[harrison...@gmail.com]

Currently I co-located the nsqd and my message producer in a stateful set on kubernetes. However, if I enable the autoscale, kubernetes might create new pods or delete existing pods based on the usage. Say I have 1 pod (name `producer-1`) now and kubernetes decides to create a new pod (name `producer-2`) because of high cpu usage. Now I have 2 identical pods, each of them has own nsqd and message producer. 10 minutes later, the cpu usage goes down, and kubernetes decodes to delete `producer-2`. The problem is there might still be messages in the `nsqd` on `producer-2`. Although the messages are persisted, they won't be available until `producer-2` was re-created.

What would be the recommended way to deploy `nsqd` on kubernetes? Separating `nsqd` and message producers?