Re: [kubernetes/kubernetes] kubectl can't delete deployment (#51764)

[k8s-ci-robot]

@dfang: Reiterating the mentions to trigger a notification:
@kubernetes/sig-cli-bugs.

In response to this:

@kubernetes/sig-cli-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Mengqi Yu]

I suspect it may be blcoked by finalizers somehow.
Can you verify if it has non empty finalizers field after you deleting it.

[fang duan]

@mengqiy

Hi, what do you mean by finalizers ?
how to verify it ?

thanks !

[fang duan]

kubectl get deployment tiller-deploy -n kube-system -o yaml | grep finalizers
returns nothing

and

kubectl get deployment tiller-deploy -n kube-system -o yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  creationTimestamp: 2017-09-01T06:03:00Z
  generation: 2
  labels:
    app: helm
    name: tiller
  name: tiller-deploy
  namespace: kube-system
  resourceVersion: "94"
  selfLink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/tiller-deploy
  uid: 356aeec5-8edb-11e7-a65d-9e0bc4f67857
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helm
      name: tiller
  strategy:
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: helm
        name: tiller
    spec:
      containers:
      - env:
        - name: TILLER_NAMESPACE
          value: kube-system
        image: gcr.io/kubernetes-helm/tiller:v2.6.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /liveness
            port: 44135
            scheme: HTTP
          initialDelaySeconds: 1
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: tiller
        ports:
        - containerPort: 44134
          name: tiller
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readiness
            port: 44135
            scheme: HTTP
          initialDelaySeconds: 1
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        beta.kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status: {}

[Mengqi Yu]

search finalizer in https://kubernetes.io/docs/api-reference/v1.7/. You will see what it is.

Graceful deletion should deletionTimestamp field. But it doesn't appear in the manifest.

Can you verify if kubectl is sending the right request by:

kubectl delete deployment tiller-deploy -v8

Look for DELETE
Paste here the what endpoint the request goes and the request body.

[fang duan]

thanks for your response.
no DELETE request in logs ...
let me know if you need any more details

> kubectl get all --namespace=kube-system
NAME                CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
svc/tiller-deploy   10.0.0.24    <none>        44134/TCP   18h

NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/tiller-deploy   0         0         0            0           18h

~
> kubectl delete deployment tiller-deploy -v8 --namespace=kube-system
I0902 09:02:33.111091   62292 loader.go:357] Config loaded from file /Users/mj23/.kube/config
I0902 09:02:33.112130   62292 cached_discovery.go:119] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/servergroups.json
I0902 09:02:33.112283   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apiregistration.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.112443   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/extensions/v1beta1/serverresources.json
I0902 09:02:33.112517   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apps/v1beta1/serverresources.json
I0902 09:02:33.112559   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authentication.k8s.io/v1/serverresources.json
I0902 09:02:33.112599   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authentication.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.112691   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authorization.k8s.io/v1/serverresources.json
I0902 09:02:33.112750   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authorization.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.112791   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/autoscaling/v1/serverresources.json
I0902 09:02:33.112837   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/autoscaling/v2alpha1/serverresources.json
I0902 09:02:33.112879   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/batch/v1/serverresources.json
I0902 09:02:33.112928   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/batch/v2alpha1/serverresources.json
I0902 09:02:33.112973   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/certificates.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.119753   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/networking.k8s.io/v1/serverresources.json
I0902 09:02:33.119828   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/policy/v1beta1/serverresources.json
I0902 09:02:33.119890   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/rbac.authorization.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.119964   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/rbac.authorization.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.120010   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/settings.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.120056   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/storage.k8s.io/v1/serverresources.json
I0902 09:02:33.120112   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/storage.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.120161   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/admissionregistration.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.120208   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apiextensions.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.120424   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/v1/serverresources.json
I0902 09:02:33.121051   62292 cached_discovery.go:119] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/servergroups.json
I0902 09:02:33.121127   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apiregistration.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.121305   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/extensions/v1beta1/serverresources.json
I0902 09:02:33.121461   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apps/v1beta1/serverresources.json
I0902 09:02:33.121547   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authentication.k8s.io/v1/serverresources.json
I0902 09:02:33.121618   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authentication.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.121700   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authorization.k8s.io/v1/serverresources.json
I0902 09:02:33.121754   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authorization.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.121833   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/autoscaling/v1/serverresources.json
I0902 09:02:33.121926   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/autoscaling/v2alpha1/serverresources.json
I0902 09:02:33.121991   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/batch/v1/serverresources.json
I0902 09:02:33.122054   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/batch/v2alpha1/serverresources.json
I0902 09:02:33.122160   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/certificates.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.122250   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/networking.k8s.io/v1/serverresources.json
I0902 09:02:33.122311   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/policy/v1beta1/serverresources.json
I0902 09:02:33.122387   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/rbac.authorization.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.122467   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/rbac.authorization.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.122518   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/settings.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.122566   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/storage.k8s.io/v1/serverresources.json
I0902 09:02:33.122611   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/storage.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.122670   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/admissionregistration.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.122729   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apiextensions.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.123030   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/v1/serverresources.json
I0902 09:02:33.123671   62292 cached_discovery.go:119] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/servergroups.json
I0902 09:02:33.123727   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apiregistration.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.123840   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/extensions/v1beta1/serverresources.json
I0902 09:02:33.123938   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apps/v1beta1/serverresources.json
I0902 09:02:33.123993   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authentication.k8s.io/v1/serverresources.json
I0902 09:02:33.124059   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authentication.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.124126   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authorization.k8s.io/v1/serverresources.json
I0902 09:02:33.124202   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authorization.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.124276   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/autoscaling/v1/serverresources.json
I0902 09:02:33.124336   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/autoscaling/v2alpha1/serverresources.json
I0902 09:02:33.124378   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/batch/v1/serverresources.json
I0902 09:02:33.124431   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/batch/v2alpha1/serverresources.json
I0902 09:02:33.124482   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/certificates.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.124541   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/networking.k8s.io/v1/serverresources.json
I0902 09:02:33.124586   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/policy/v1beta1/serverresources.json
I0902 09:02:33.124641   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/rbac.authorization.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.124895   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/rbac.authorization.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.125008   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/settings.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.125109   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/storage.k8s.io/v1/serverresources.json
I0902 09:02:33.125203   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/storage.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.125330   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/admissionregistration.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.125462   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apiextensions.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.125866   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/v1/serverresources.json
I0902 09:02:33.126065   62292 cached_discovery.go:119] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/servergroups.json
I0902 09:02:33.126116   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apiregistration.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.126243   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/extensions/v1beta1/serverresources.json
I0902 09:02:33.126312   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apps/v1beta1/serverresources.json
I0902 09:02:33.126350   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authentication.k8s.io/v1/serverresources.json
I0902 09:02:33.126438   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authentication.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.126482   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authorization.k8s.io/v1/serverresources.json
I0902 09:02:33.126522   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authorization.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.126564   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/autoscaling/v1/serverresources.json
I0902 09:02:33.126604   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/autoscaling/v2alpha1/serverresources.json
I0902 09:02:33.126642   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/batch/v1/serverresources.json
I0902 09:02:33.126695   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/batch/v2alpha1/serverresources.json
I0902 09:02:33.126741   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/certificates.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.126776   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/networking.k8s.io/v1/serverresources.json
I0902 09:02:33.126815   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/policy/v1beta1/serverresources.json
I0902 09:02:33.126864   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/rbac.authorization.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.126915   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/rbac.authorization.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.126950   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/settings.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.126986   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/storage.k8s.io/v1/serverresources.json
I0902 09:02:33.127020   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/storage.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.127061   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/admissionregistration.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.127103   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apiextensions.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.127286   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/v1/serverresources.json
I0902 09:02:33.127423   62292 cached_discovery.go:119] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/servergroups.json
I0902 09:02:33.127463   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apiregistration.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.127564   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/extensions/v1beta1/serverresources.json
I0902 09:02:33.127632   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apps/v1beta1/serverresources.json
I0902 09:02:33.127667   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authentication.k8s.io/v1/serverresources.json
I0902 09:02:33.127700   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authentication.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.127740   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authorization.k8s.io/v1/serverresources.json
I0902 09:02:33.127784   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/authorization.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.127826   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/autoscaling/v1/serverresources.json
I0902 09:02:33.127866   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/autoscaling/v2alpha1/serverresources.json
I0902 09:02:33.127904   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/batch/v1/serverresources.json
I0902 09:02:33.127951   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/batch/v2alpha1/serverresources.json
I0902 09:02:33.127996   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/certificates.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.128032   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/networking.k8s.io/v1/serverresources.json
I0902 09:02:33.128073   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/policy/v1beta1/serverresources.json
I0902 09:02:33.128120   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/rbac.authorization.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.128168   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/rbac.authorization.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.128202   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/settings.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.128238   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/storage.k8s.io/v1/serverresources.json
I0902 09:02:33.128277   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/storage.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.128323   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/admissionregistration.k8s.io/v1alpha1/serverresources.json
I0902 09:02:33.128362   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/apiextensions.k8s.io/v1beta1/serverresources.json
I0902 09:02:33.128537   62292 cached_discovery.go:72] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/v1/serverresources.json
I0902 09:02:33.129600   62292 cached_discovery.go:119] returning cached discovery info from /Users/mj23/.kube/cache/discovery/192.168.64.12_8443/servergroups.json
I0902 09:02:33.142553   62292 round_trippers.go:383] GET https://192.168.64.12:8443/apis/extensions/v1beta1/namespaces/kube-system/deployments/tiller-deploy
I0902 09:02:33.142576   62292 round_trippers.go:390] Request Headers:
I0902 09:02:33.142584   62292 round_trippers.go:393]     Accept: application/json, */*
I0902 09:02:33.142592   62292 round_trippers.go:393]     User-Agent: kubectl/v1.7.3 (darwin/amd64) kubernetes/2c2fe6e
I0902 09:02:33.154171   62292 round_trippers.go:408] Response Status: 200 OK in 11 milliseconds
I0902 09:02:33.154189   62292 round_trippers.go:411] Response Headers:
I0902 09:02:33.154195   62292 round_trippers.go:414]     Content-Type: application/json
I0902 09:02:33.154200   62292 round_trippers.go:414]     Content-Length: 1528
I0902 09:02:33.154205   62292 round_trippers.go:414]     Date: Fri, 01 Sep 2017 10:59:05 GMT
I0902 09:02:33.154237   62292 request.go:991] Response Body: {"kind":"Deployment","apiVersion":"extensions/v1beta1","metadata":{"name":"tiller-deploy","namespace":"kube-system","selfLink":"/apis/extensions/v1beta1/namespaces/kube-system/deployments/tiller-deploy","uid":"356aeec5-8edb-11e7-a65d-9e0bc4f67857","resourceVersion":"103","generation":3,"creationTimestamp":"2017-09-01T06:03:00Z","labels":{"app":"helm","name":"tiller"}},"spec":{"replicas":0,"selector":{"matchLabels":{"app":"helm","name":"tiller"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"helm","name":"tiller"}},"spec":{"containers":[{"name":"tiller","image":"gcr.io/kubernetes-helm/tiller:v2.6.0","ports":[{"name":"tiller","containerPort":44134,"protocol":"TCP"}],"env":[{"name":"TILLER_NAMESPACE","value":"kube-system"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/liveness","port":44135,"scheme":"HTTP"},"initialDelaySeconds":1,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/readiness","port":44135,"scheme":"HTTP"},"initialDelaySeconds":1,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"beta.kubernetes.io/os":"linux"},"securityContext":{},"schedulerName":"default-scheduler"}},"strategy":{"type":"Recreate"},"revisionHistoryLimit":0,"paused":true},"status":{}}
I0902 09:02:33.156022   62292 request.go:991] Request Body: {"kind":"Deployment","apiVersion":"extensions/v1beta1","metadata":{"name":"tiller-deploy","namespace":"kube-system","selfLink":"/apis/extensions/v1beta1/namespaces/kube-system/deployments/tiller-deploy","uid":"356aeec5-8edb-11e7-a65d-9e0bc4f67857","resourceVersion":"103","generation":3,"creationTimestamp":"2017-09-01T06:03:00Z","labels":{"app":"helm","name":"tiller"}},"spec":{"replicas":0,"selector":{"matchLabels":{"app":"helm","name":"tiller"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"helm","name":"tiller"}},"spec":{"containers":[{"name":"tiller","image":"gcr.io/kubernetes-helm/tiller:v2.6.0","ports":[{"name":"tiller","containerPort":44134,"protocol":"TCP"}],"env":[{"name":"TILLER_NAMESPACE","value":"kube-system"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/liveness","port":44135,"scheme":"HTTP"},"initialDelaySeconds":1,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/readiness","port":44135,"scheme":"HTTP"},"initialDelaySeconds":1,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"beta.kubernetes.io/os":"linux"},"securityContext":{},"schedulerName":"default-scheduler"}},"strategy":{"type":"Recreate"},"revisionHistoryLimit":0,"paused":true},"status":{}}
I0902 09:02:33.156099   62292 round_trippers.go:383] PUT https://192.168.64.12:8443/apis/extensions/v1beta1/namespaces/kube-system/deployments/tiller-deploy
I0902 09:02:33.156108   62292 round_trippers.go:390] Request Headers:
I0902 09:02:33.156113   62292 round_trippers.go:393]     User-Agent: kubectl/v1.7.3 (darwin/amd64) kubernetes/2c2fe6e
I0902 09:02:33.156117   62292 round_trippers.go:393]     Content-Type: application/json
I0902 09:02:33.156120   62292 round_trippers.go:393]     Accept: application/json, */*
I0902 09:02:33.160614   62292 round_trippers.go:408] Response Status: 200 OK in 4 milliseconds
I0902 09:02:33.160629   62292 round_trippers.go:411] Response Headers:
I0902 09:02:33.160635   62292 round_trippers.go:414]     Date: Fri, 01 Sep 2017 10:59:05 GMT
I0902 09:02:33.160640   62292 round_trippers.go:414]     Content-Type: application/json
I0902 09:02:33.160645   62292 round_trippers.go:414]     Content-Length: 1528
I0902 09:02:33.160714   62292 request.go:991] Response Body: {"kind":"Deployment","apiVersion":"extensions/v1beta1","metadata":{"name":"tiller-deploy","namespace":"kube-system","selfLink":"/apis/extensions/v1beta1/namespaces/kube-system/deployments/tiller-deploy","uid":"356aeec5-8edb-11e7-a65d-9e0bc4f67857","resourceVersion":"103","generation":3,"creationTimestamp":"2017-09-01T06:03:00Z","labels":{"app":"helm","name":"tiller"}},"spec":{"replicas":0,"selector":{"matchLabels":{"app":"helm","name":"tiller"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"helm","name":"tiller"}},"spec":{"containers":[{"name":"tiller","image":"gcr.io/kubernetes-helm/tiller:v2.6.0","ports":[{"name":"tiller","containerPort":44134,"protocol":"TCP"}],"env":[{"name":"TILLER_NAMESPACE","value":"kube-system"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/liveness","port":44135,"scheme":"HTTP"},"initialDelaySeconds":1,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/readiness","port":44135,"scheme":"HTTP"},"initialDelaySeconds":1,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"beta.kubernetes.io/os":"linux"},"securityContext":{},"schedulerName":"default-scheduler"}},"strategy":{"type":"Recreate"},"revisionHistoryLimit":0,"paused":true},"status":{}}
I0902 09:02:34.161833   62292 round_trippers.go:383] GET https://192.168.64.12:8443/apis/extensions/v1beta1/namespaces/kube-system/deployments/tiller-deploy
I0902 09:02:34.161870   62292 round_trippers.go:390] Request Headers:
I0902 09:02:34.161881   62292 round_trippers.go:393]     Accept: application/json, */*
I0902 09:02:34.161903   62292 round_trippers.go:393]     User-Agent: kubectl/v1.7.3 (darwin/amd64) kubernetes/2c2fe6e
I0902 09:02:34.164653   62292 round_trippers.go:408] Response Status: 200 OK in 2 milliseconds
I0902 09:02:34.164677   62292 round_trippers.go:411] Response Headers:
I0902 09:02:34.164684   62292 round_trippers.go:414]     Content-Type: application/json
I0902 09:02:34.164690   62292 round_trippers.go:414]     Content-Length: 1528
I0902 09:02:34.164695   62292 round_trippers.go:414]     Date: Fri, 01 Sep 2017 10:59:06 GMT
I0902 09:02:34.164738   62292 request.go:991] Response Body: {"kind":"Deployment","apiVersion":"extensions/v1beta1","metadata":{"name":"tiller-deploy","namespace":"kube-system","selfLink":"/apis/extensions/v1beta1/namespaces/kube-system/deployments/tiller-deploy","uid":"356aeec5-8edb-11e7-a65d-9e0bc4f67857","resourceVersion":"103","generation":3,"creationTimestamp":"2017-09-01T06:03:00Z","labels":{"app":"helm","name":"tiller"}},"spec":{"replicas":0,"selector":{"matchLabels":{"app":"helm","name":"tiller"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"helm","name":"tiller"}},"spec":{"containers":[{"name":"tiller","image":"gcr.io/kubernetes-helm/tiller:v2.6.0","ports":[{"name":"tiller","containerPort":44134,"protocol":"TCP"}],"env":[{"name":"TILLER_NAMESPACE","value":"kube-system"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/liveness","port":44135,"scheme":"HTTP"},"initialDelaySeconds":1,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/readiness","port":44135,"scheme":"HTTP"},"initialDelaySeconds":1,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"beta.kubernetes.io/os":"linux"},"securityContext":{},"schedulerName":"default-scheduler"}},"strategy":{"type":"Recreate"},"revisionHistoryLimit":0,"paused":true},"status":{}}
I0902 09:02:35.162217   62292 round_trippers.go:383] GET https://192.168.64.12:8443/apis/extensions/v1beta1/namespaces/kube-system/deployments/tiller-deploy
I0902 09:02:35.162248   62292 round_trippers.go:390] Request Headers:
I0902 09:02:35.162256   62292 round_trippers.go:393]     Accept: application/json, */*
I0902 09:02:35.162263   62292 round_trippers.go:393]     User-Agent: kubectl/v1.7.3 (darwin/amd64) kubernetes/2c2fe6e
I0902 09:02:35.165270   62292 round_trippers.go:408] Response Status: 200 OK in 2 milliseconds
I0902 09:02:35.165305   62292 round_trippers.go:411] Response Headers:
I0902 09:02:35.165312   62292 round_trippers.go:414]     Content-Type: application/json
I0902 09:02:35.165318   62292 round_trippers.go:414]     Content-Length: 1528
I0902 09:02:35.165324   62292 round_trippers.go:414]     Date: Fri, 01 Sep 2017 10:59:07 GMT
I0902 09:02:35.165362   62292 request.go:991] Response Body: {"kind":"Deployment","apiVersion":"extensions/v1beta1","metadata":{"name":"tiller-deploy","namespace":"kube-system","selfLink":"/apis/extensions/v1beta1/namespaces/kube-system/deployments/tiller-deploy","uid":"356aeec5-8edb-11e7-a65d-9e0bc4f67857","resourceVersion":"103","generation":3,"creationTimestamp":"2017-09-01T06:03:00Z","labels":{"app":"helm","name":"tiller"}},"spec":{"replicas":0,"selector":{"matchLabels":{"app":"helm","name":"tiller"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"helm","name":"tiller"}},"spec":{"containers":[{"name":"tiller","image":"gcr.io/kubernetes-helm/tiller:v2.6.0","ports":[{"name":"tiller","containerPort":44134,"protocol":"TCP"}],"env":[{"name":"TILLER_NAMESPACE","value":"kube-system"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/liveness","port":44135,"scheme":"HTTP"},"initialDelaySeconds":1,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/readiness","port":44135,"scheme":"HTTP"},"initialDelaySeconds":1,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"beta.kubernetes.io/os":"linux"},"securityContext":{},"schedulerName":"default-scheduler"}},"strategy":{"type":"Recreate"},"revisionHistoryLimit":0,"paused":true},"status":{}}

ps: i had to press CTRL + C to stop it ...

[Di Xu]

@dfang Have you tried kubectl delete deployment tiller-deploy -n kube-system --now?

From the log, it seems kubectl is waiting for the controller's status (like observedGeneration, replicas etc) to update.

[fang duan]

@dixudx no, i tried --force, now i upgraded my kubernetes, helm, i can kubectl delete the deployment...

[naikabhay]

Hello.

I am using 1.8.4 and still I am getting the same error while deleting the deployment.

[Blade]

+1

[Mark Gerrits]

Same issue here. I can not delete anything except a pod. Deleting a Deployment will just hang forever and no DELETE call to be found in de verbose logging. I am using 1.8.1 and also tried 1.8.2

[Jacqueline Outka]

I had this issue on AKS and resolved by upgrading the cluster from 1.7.7 to 1.8.2 (as a note for anyone who happens to find themselves in the same situation).

[Daniel Albuschat]

I'm seeing this issue with k8s 1.9.1 on a Raspberry Pi 3, too. Deleting one of the deployments spits out "timed out waiting for the condition", and the deployment is still there.

On this cluster, I experience that the apiserver vanishes from time to time - I have to systemctl restart kubelet in those cases to get it up again. Maybe it's related.

The cluster was initialized with kubeadm (v1.9.1, too).

[Duncan Gauld]

I'm also seeing this with a kubeadm 1.9.2 cluster:

$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

websocket-echo.yml (uses a local image):

apiVersion: v1
kind: Service
metadata:
  name: websocket-echo-svc
  labels:
    app: websocket-echo
spec:
  type: ClusterIP
  ports:
  - port: 80
    protocol: TCP
    name: http
  selector:
    app: websocket-echo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: websocket-echo-dep
  labels:
    app: websocket-echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: websocket-echo
  template:
    metadata:
      labels:
        app: websocket-echo
    spec:
      containers:
      - image: websocket-echo
        imagePullPolicy: Never
        name: websocket-echo
        ports:
        - containerPort: 80

Created with:

$ kubectl apply -f websocket-echo.yml

The service and deployment are created, but no pods seem to be ever created.

$ kubectl delete -f websocket-echo.yml
service "websocket-echo-svc" deleted
error: error when stopping "websocket-echo.yml": timed out waiting for the condition

I wonder if it has something to do with the following strange output (desired == current == 0). Expected: desired == 1 as per spec.replicas: 1.

$ kubectl get all
NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/websocket-echo-dep     0         0         0            0           21m

[Valentin Ouvrard]

Got the same problem with own deployed cluster on v1.9.2..

Any way to delete the deploy via ETCDv3 and etcdctl ?

Thanks !

[Jan Weidner]

Same problem here, is there a workaround?

[Daniel Albuschat]

I'm on 1.9.3 now and haven't had this problem on the Raspberry Pi 3 cluster any more. I'm deleting and re-creating deployments on this cluster regularly, since I'm using it for experiments. I didn't experience the apiserver dying anymore, either.

[Keyvan Hedayati]

I have same issue here, beside deleting deployment scaling doesn't work either

[John Stange]

I'm getting this as well, just in trying to clean up after a test run in my first Kubernetes foray. I've been following https://opencredo.com/kubernetes-aws-terraform-ansible-3/ and modernizing as I go for current versions of docker, k8s, etc. My Kubernetes bits are all 1.9.3.

$ kubectl delete deployment nginx
error: timed out waiting for the condition

Futzing around with --force and various timeout options doesn't seem to change the behavior. If I run with --cascade=false (picked up from kubernetes/federation#215), it says that it succeeded, but nothing actually changes:

$ kubectl delete deployment nginx --cascade=false
deployment "nginx" deleted
$ kubectl get deployments
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx     0         2         2            2           23h
nginx2    0         0         0            0           16h

(I attempted at one point to add a second deployment to see if it would take over available slots, hence the nginx2)

If I try to nuke everything from orbit, I hang forever:

$ kubectl delete all --all --include-uninitialized
<drool>

Nuke from orbit, but with cascade=false, and we still sit on a throne of lies:

$ kubectl delete all --all --include-uninitialized --cascade=false
deployment "nginx" deleted
deployment "nginx2" deleted
replicaset "nginx-7587c6fdb6" deleted
deployment "nginx" deleted
deployment "nginx2" deleted
replicaset "nginx-7587c6fdb6" deleted
pod "nginx-7587c6fdb6-f5qrs" deleted
pod "nginx-7587c6fdb6-rr8s5" deleted
service "kubernetes" deleted
service "nginx2" deleted
$ kubectl get all
NAME            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/nginx    0         2         2            2           1d
deploy/nginx2   0         0         0            0           16h

NAME                  DESIRED   CURRENT   READY     AGE
rs/nginx-7587c6fdb6   0         2         2         1d

NAME            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/nginx    0         2         2            2           1d
deploy/nginx2   0         0         0            0           16h

NAME                  DESIRED   CURRENT   READY     AGE
rs/nginx-7587c6fdb6   0         2         2         1d

NAME                        READY     STATUS        RESTARTS   AGE
po/nginx-7587c6fdb6-f5qrs   0/1       Terminating   2          1d
po/nginx-7587c6fdb6-rr8s5   0/1       Terminating   2          18h

NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
svc/kubernetes   ClusterIP   10.1.100.1   <none>        443/TCP   14s

"Terminating" eh... let's see what the logs on my controller hosts have... whoa, chatty:

I'm still getting oriented here, and don't know what the heck I'm doing, esp. re: routing and the overlay network for this AWS scenario. Instinct suspects that I've fouled something at the network level and the tools are responding gracelessly to an inability to communicate. Which is purely speculative for now. Some logs I'll investigate later, which may or may not related to my inability to delete things:

Mar  8 16:30:27 ip-10-0-2-242 kube-controller-manager: I0308 16:30:27.819380   21546 range_allocator.go:108] Node ip-10-1-1-96.us-gov-west-1.compute.internal has CIDR 10.1.0.0/24, occupying it in CIDR map
Mar  8 16:30:27 ip-10-0-2-242 kube-controller-manager: E0308 16:30:27.819402   21546 controllermanager.go:486] Error starting "node"
Mar  8 16:30:27 ip-10-0-2-242 kube-controller-manager: F0308 16:30:27.819411   21546 controllermanager.go:155] error starting controllers: failed to mark cidr as occupied: cidr 10.1.0.0/24 is out the range of cluster cidr 10.1.101.0/24

[John Stange]

I fixed my dopey network configuration, and suddenly things started working. So I think what we have here is a fussy utility with confusing error messaging, rather than a proper bug.

If I had to speculate I'd guess that my controllers have been starting up, flailing over their configs, giving up, then restarting in a loop. Alternately, they might be running but unable to communicate with things pinned to IPs that aren't in a real IP space.

Either way, I'd expect kubectl to nudge me in that direction when a deletion times out ("communication with X.X.X.X timed out, go look at your controller logs you goon").

[Song]

Restart k8s master can solve this problem.

[xuji-cny]

I got as some problem when delete deployment and I fixed it
just restart controller manager, then it works!

[Ahmed Itman]

Did anyone solve this issue ?

[Jan Hudec]

Seeing this in 1.10.0 with a stateful set. No finalizers. Restart didn't seem to help, the stateful set is still there though all the related other objects are gone. Restarting the server (it is single-node test environment, so it restarts all of it) didn't help.

[slve]

I also had this issue and solved for my case #60957 (comment)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.

/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[Kubernetes Prow Robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Kubernetes Prow Robot]

Closed #51764.