Ingress stuck in create when specifying kubernetes.io/ingress.class: nginx

662 views
Skip to first unread message

cpretzer

unread,
Sep 7, 2017, 1:52:29 AM9/7/17
to kubernet...@googlegroups.com
Hi,

I've been working on creating an ingress controller on GKE and am having some trouble creating an ingress controller with the annotation kubernetes.io/ingress.class: "nginx".

I'll provide code and examples below, but the short version is that the ingress is created successfully when there is no annotation, but adding the lines below to my ingress YAML results in the ingress never being created.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: nginx-ingress
  annotations:
    # This tells to only use the Nginx Ingress Controller
    # and avoids the creation on a Global LoadBalancer on GKE.
...

I have confirmed that removing the annotation results in the ingress being successfully created.

Here is the successful ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: nginx-ingress
spec:
  rules:
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80


and here is the failing definition; the only difference is that the failing definition has the kubernetes.io/ingress.class annotation

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
  namespace: nginx-ingress
  annotations:
    # This tells to only use the Nginx Ingress Controller
    # and avoids the creation on a Global LoadBalancer on GKE.
spec:
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80


When reproducing this, I have used both the kubernetes ingress controller from the official repository: https://github.com/kubernetes/ingress/tree/master/controllers/nginx as well as the NGINX repository: https://github.com/nginxinc/kubernetes-ingress/

Here is the YAML for the deployment for the Kubernetes version:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  labels:
    k8s-app: nginx-ingress-controller
  namespace: nginx-ingress
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress-controller
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      # hostNetwork makes it possible to use ipv6 and to preserve the source IP correctly regardless of docker configuration
      # however, it is not a hard dependency of the nginx-ingress-controller itself and it may cause issues if port 10254 already is taken on the host
      # that said, since hostPort is broken on CNI (https://github.com/kubernetes/kubernetes/issues/31307) we have to use hostNetwork where CNI is used
      # like with kubeadm
      # hostNetwork: true
      terminationGracePeriodSeconds: 60
      containers:
        name: nginx-ingress-controller
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/coffee-svc

Several hours after applying the YAML for the gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.12 image, this is the output for the command: kubectl describe ing -n nginx-ingress test-ingress:

21:12:30:complete-example$ kubectl describe ing -n nginx-ingress test-ingress
Name: test-ingress
Namespace: nginx-ingress
Address: <IP REMOVED>
Default backend: default-http-backend:80 (10.24.2.5:8080)
Rules:
  Host Path Backends
  ---- ---- --------
    /tea tea-svc:80 (<none>)
    /coffee coffee-svc:80 (<none>)
Annotations:
Events: <none>


Here is the YAML for the image from the NGINX repository: 

apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-plus-ingress-rc
  labels:
    app: nginx-plus-ingress
  namespace: nginx-ingress
spec:
  replicas: 1
  selector:
    app: nginx-plus-ingress
  template:
    metadata:
      labels:
        app: nginx-plus-ingress
    spec:
      containers:
        imagePullPolicy: Always
        name: nginx-plus-ingress
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        - containerPort: 8080
          hostPort: 8080
        readinessProbe:
          httpGet:
            scheme: HTTPS
            path: /heartbeat
            port: 443
            httpHeaders:
              - name: Host
                value: test.ingress.example.com
          periodSeconds: 20
          timeoutSeconds: 20
          successThreshold: 1
          failureThreshold: 10
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        # Uncomment the lines below to enable extensive logging and/or customization of
        # NGINX configuration with configmaps
        args:
          - -nginx-plus
          - -v=2
         #- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
          - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret


After deploying the nginx-plus-ingress:1.0.0-beta0 image, the output for kubectl describe ing -n nginx-ingress test-ingress 

22:47:16:complete-example$ kubectl describe ing -n nginx-ingress test-ingress
Name: test-ingress
Namespace: nginx-ingress
Address:
Default backend: default-http-backend:80 (10.24.2.5:8080)
Rules:
  Host Path Backends
  ---- ---- --------
    /tea tea-svc:80 (<none>)
    /coffee coffee-svc:80 (<none>)
Annotations:
Events:
  FirstSeen LastSeen Count From SubObjectPath Type Reason Message
  --------- -------- ----- ---- ------------- -------- ------ -------
  6m 6m 1 nginx-ingress-controller Normal AddedOrUpdated Configuration for nginx-ingress/cafe-ingress was added or updated

It shows only 6 minutes in this example, but I've left it for 20 or 30 minutes with no update. One difference is that no address is assigned in this case.

Regardless of which ingress controller image I use, the ingress never starts. The entry in the Discovery & Load Balancing section of the GKE interface looks like this:

Can anyone tell me how to debug the process that creates the ingress? I can't find any useful logs using kubectl or in GKE/GCE.

It seems like there's a simple setting that I'm missing in order to make this work, and I've looked through the docs to no avail.

cpretzer

unread,
Sep 7, 2017, 12:51:07 PM9/7/17
to kubernet...@googlegroups.com
When I started testing the Kubernetes version in the cluster was 1.6.9, I have since upgraded to 1.7.4, and the behavior is reproducible in both versions.

For the sake of being thorough, here are the YAML files for the services in the example:

coffee-rc.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  name: coffee-rc
  namespace: nginx-ingress
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: coffee
    spec:
      containers:
      - name: coffee
        image: nginxdemos/hello
        ports:
        - containerPort: 80

coffee-svc.yaml
apiVersion: v1
kind: Service
metadata:
 name: coffee-svc
 labels:
   app: coffee
 namespace: nginx-ingress
spec:
 type: NodePort
 ports:
 - port: 80
   targetPort: 80
   protocol: TCP
   name: http
 selector:
   app: coffee


tea-rc.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  name: tea-rc
  namespace: nginx-ingress
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: tea
    spec:
      containers:
      - name: tea
        image: nginxdemos/hello
        ports:
        - containerPort: 80


tea-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: tea-svc
  labels:
    app: tea
  namespace: nginx-ingress
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: tea



dvrede...@sbevision.com

unread,
Jan 16, 2018, 9:05:52 AM1/16/18
to Kubernetes user discussion and Q&A
I'm experiencing the same thing. Did you ever get past this?

Charles Pretzer

unread,
Feb 8, 2018, 3:35:03 PM2/8/18
to kubernet...@googlegroups.com
I didn't, but I think I was probably doing something wrong.

It's my understanding that the annotation is meant to be used when
there are multiple ingress controllers in use.

I was hoping that this thread would provide pointers on how to debug
the Ingress resource.

On Tue, Jan 16, 2018 at 6:05 AM, <dvrede...@sbevision.com> wrote:
> I'm experiencing the same thing. Did you ever get past this?
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/6-Fi1Kk7cfY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.
> To post to this group, send email to kubernet...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.



--
Charles Pretzer
Technical Architect, Professional Services
Mobile: 4152500816
San Francisco, CA
www.nginx.com
Reply all
Reply to author
Forward
0 new messages