CrashLoopBackOff - how to prevent a rolling update

[Norman Khine]

Hello, I have a pod with 3 kublets one of the kublet had an issue and I had a `CrashLoopBackOff` but the rolling update continued, so the application failed and became un available.

What is the correct way to prevent this from happening?

here is my yaml file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: app-dev
  labels:
    pod: app
    track: develop
  annotations:
    scheduler.alpha.kubernetes.io/affinity: >
      {
        "nodeAffinity": {
          "requiredDuringSchedulingIgnoredDuringExecution": {
            "nodeSelectorTerms": [
              {
                "matchExpressions": [
                  {
                    "key": "beta.kubernetes.io/instance-type",
                    "operator": "In",
                    "values": ["c4.large"]
                  }
                ]
              }
            ]
          }
        }
      }
spec:
  replicas: 3
  template:
    metadata:
      labels:
        pod: app
        track: develop
    spec:
      containers:
        - name: trint-api
          image: quay.io/nkhine/api:develop
          imagePullPolicy: Always
          ports:
            - containerPort: 3000
        - name: media
          image: quay.io/nkhine/media:develop
          imagePullPolicy: Always
          ports:
            - containerPort: 4000
        - name: phantom
          image: docker.io/wernight/phantomjs:2.1.1
          command: ["phantomjs", "--webdriver=8910", "--web-security=no", "--load-images=false", "--local-to-remote-url-access=yes"]
          ports:
            - containerPort: 8910
          resources:
            requests:
              memory: 1000
      restartPolicy: Always
      imagePullSecrets:
        # we download these from quay.io account
        - name: my-pull-secret

in my case the :media kublet had an issue and it failed to start.

any advise is much appreciated

[Rodrigo Campos]

On Mon, Nov 14, 2016 at 09:57:20AM -0800, Norman Khine wrote:
> Hello, I have a pod with 3 kublets one of the kublet had an issue and I had
> a `CrashLoopBackOff` but the rolling update continued, so the application
> failed and became un available.
> What is the correct way to prevent this from happening?

Add a rediness and liveness check. That way, the deployment will continue only
if the new pods are up and ready.

[Timo Reimann]

You possibly also want to set http://kubernetes.io/docs/user-guide/deployments/#min-ready-seconds to something larger then zero. This can prevent cases where your container starts up fine but then crashes shortly afterwards. One example may be that it ran out of memory while populating some internal data structure from a database.