How to handle automated rolling restarts of nodes when using jenkins in kubernetes

[Aaron Mell]

Is there a good pattern for this? Right now we are converting jobs to run on k8 pods. One of those jobs applies terraform changes. When we update to a newer image for the node, terraform updates the autoscaling group and replaces the nodes. This causes the job to crash and hang in a bad state. My first inclination is to move the terraform change out of k8's and run on the master instead. However, we would like to eventually move the master into kubernetes as well. I am not sure how to best solve this problem.