Multiple nginx ingress controllers.

matt....@gmail.com

unread,

Apr 9, 2017, 10:57:44 AM4/9/17

to Kubernetes user discussion and Q&A

If anyone can point me to documentation on this I would be much appreciative. How can I have two or more nginx ingress controllers running in the same cluster?

This is the use case:
I have on AWS an nginx ingress controller where I spin up the controller with annotations to front the ELB with a particular cert. This means I want ( I think ) the ELB to terminate the TLS and everything behind it to be unencrypted HTTP.

That being said I have certain application running in the cluster, such as Kibana as an example that I want to just run as HTTPS. Easy enough to do with a normal setup, however if I have the ELB terminating the TLS this presents a problem.

Another use case would be if I wanted to setup one ingress controller to be internal only meaning having a list of white listed IP's. Another controller could be open to the world.

Does this configuration currently exist?

Thanks,
M

Brandon Philips

unread,

Apr 9, 2017, 11:46:22 PM4/9/17

to Kubernetes user discussion and Q&A

You are looking for the Ingress Class annotation. See: https://github.com/kubernetes/ingress/tree/master/docs/faq#how-do-i-disable-an-ingress-controller

Cheers,

Brandon

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

Matt Snoby

unread,

Apr 10, 2017, 8:28:19 AM4/10/17

to kubernet...@googlegroups.com

Brandon,

Thanks for taking time to reply. The links you sent unfortunately links to a dead url. However I”m not talking about running a traefik ingress controller and a nginx ingress controller. I’m talking about running TWO nginx ingress controllers at the same time.

Matt

You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/FFt3z5XKP6E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.

Guangya Liu

unread,

Apr 10, 2017, 8:47:56 AM4/10/17

to kubernet...@googlegroups.com

Does this help? https://github.com/gyliu513/ingress/tree/4674c0ec691c25b1640064d772c3a74e4337a56b/examples/daemonset/nginx#nginx-ingress-daemonset , this can enable running multiple nginx ingress controllers in one cluster with daemonset.

On Mon, Apr 10, 2017 at 8:28 PM, Matt Snoby <matt....@icloud.com> wrote:

Brandon,
Thanks for taking time to reply. The links you sent unfortunately links to a dead url. However I”m not talking about running a traefik ingress controller and a nginx ingress controller. I’m talking about running TWO nginx ingress controllers at the same time.

Matt

On Apr 9, 2017, at 11:46 PM, Brandon Philips <brandon...@coreos.com> wrote:

You are looking for the Ingress Class annotation. See: https://github.com/kubernetes/ingress/tree/master/docs/faq#how-do-i-disable-an-ingress-controller

Cheers,

Brandon

On Sun, Apr 9, 2017 at 7:57 AM <matt....@gmail.com> wrote:

If anyone can point me to documentation on this I would be much appreciative. How can I have two or more nginx ingress controllers running in the same cluster?

This is the use case:
I have on AWS an nginx ingress controller where I spin up the controller with annotations to front the ELB with a particular cert. This means I want ( I think ) the ELB to terminate the TLS and everything behind it to be unencrypted HTTP.

That being said I have certain application running in the cluster, such as Kibana as an example that I want to just run as HTTPS. Easy enough to do with a normal setup, however if I have the ELB terminating the TLS this presents a problem.

Another use case would be if I wanted to setup one ingress controller to be internal only meaning having a list of white listed IP's. Another controller could be open to the world.

Does this configuration currently exist?

Thanks,
M

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/FFt3z5XKP6E/unsubscribe.

To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

Matt Snoby

unread,

Apr 10, 2017, 9:13:10 AM4/10/17

to kubernet...@googlegroups.com

Is there a way, with annotations, to target a certain NGINX controller?

Not sure what use case there could be to run a daemonset.

Matt

To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

Guangya Liu

unread,

Apr 10, 2017, 9:53:43 AM4/10/17

to kubernet...@googlegroups.com

I'm not sure if there are ways to enable you target a certain NGINX controller, perhaps you can get more comments from @aledbf (https://github.com/aledbf) for this.

One question is why not use one ingress controller with different endpoints? Such as foo.bar.com/v1 and foo.bar.com/v2?

Matt

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe...@googlegroups.com.

To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/FFt3z5XKP6E/unsubscribe.

To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsubscribe...@googlegroups.com.

To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/FFt3z5XKP6E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

Guangya Liu

unread,

Apr 10, 2017, 9:58:27 AM4/10/17

to kubernet...@googlegroups.com

The use case of daemonset with nginx ingress controller is usually for some HA purpose, we can set up a HAProxy before all of the ingress controllers to make sure even if one ingress controller goes down, the service can still be accessed.

Giancarlo Rubio

unread,

Apr 11, 2017, 3:36:59 AM4/11/17

to Kubernetes user discussion and Q&A, matt....@gmail.com

On Sunday, 9 April 2017 16:57:44 UTC+2, matt....@gmail.com wrote:

> If anyone can point me to documentation on this I would be much appreciative. How can I have two or more nginx ingress controllers running in the same cluster?

>

> This is the use case:

> I have on AWS an nginx ingress controller where I spin up the controller with annotations to front the ELB with a particular cert. This means I want ( I think ) the ELB to terminate the TLS and everything behind it to be unencrypted HTTP.

>

> That being said I have certain application running in the cluster, such as Kibana as an example that I want to just run as HTTPS. Easy enough to do with a normal setup, however if I have the ELB terminating the TLS this presents a problem.

Why it's a problem? Have you enabled the proxy_protocol for tls termination?

https://github.com/kubernetes/ingress/tree/098bcb5e656356d9d0ab0f6e592dbad764dc32e3/controllers/nginx/examples/proxy-protocol

https://github.com/kubernetes/ingress/search?utf8=✓&q=proxy_protocol&type=

> Another use case would be if I wanted to setup one ingress controller to be internal only meaning having a list of white listed IP's. Another controller could be open to the world.

>

I have the same scenario

this is the internal ingress

--default-backend-service=kube-system/default-http-backend

--tcp-services-configmap=kube-system/tcp-configmap

--configmap=kube-system/nginx-load-balancer-conf

--healthz-port=10001

--ingress-class=intern

--election-id=intern

the annotation for this ingress should be "kubernetes.io/ingress.class: intern"

the external ingress

--default-backend-service=kube-system/default-http-backend

--tcp-services-configmap=kube-system/tcp-configmap

--configmap=kube-system/nginx-load-balancer-conf

--healthz-port=10002

--ingress-class=extern

--election-id=intern

Beware that both use different ports for the health check, so you can isolate both from elb

whitelist ip => https://github.com/kubernetes/ingress/blob/7ca7652ab26e1a5775f3066f53f28d5ea5eb3bb7/controllers/nginx/configuration.md#whitelist-source-range

Matt Snoby

unread,

Apr 11, 2017, 8:40:36 AM4/11/17

to Giancarlo Rubio, Kubernetes user discussion and Q&A

Giancarlo,

This is exactly what I was looking for. Yes we do have proxy protocol enabled. I did not know about the “ingress-class” command line argument. I also did not know about the election-id argument. Is there where all that magic is kept or is there another location?

https://github.com/kubernetes/ingress/blob/12a0373d2e52ac19ec80c80d0c00abf2b0975612/core/pkg/ingress/controller/launch.go

I assume in your internal / external configuration each ingress controller spins up it’s own ELB?

Or it shares the same ELB and that’s why you have your health checks on different ports?

Do you have any ingress annotations set for the ELB?

Thanks,

Matt

Giancarlo Rubio

unread,

Apr 13, 2017, 3:41:07 AM4/13/17

to Kubernetes user discussion and Q&A, Matt Snoby

On Tue, 11 Apr 2017 at 14:40, Matt Snoby <matt....@icloud.com> wrote:

Giancarlo,
This is exactly what I was looking for. Yes we do have proxy protocol enabled. I did not know about the “ingress-class” command line argument. I also did not know about the election-id argument. Is there where all that magic is kept or is there another location?
https://github.com/kubernetes/ingress/blob/12a0373d2e52ac19ec80c80d0c00abf2b0975612/core/pkg/ingress/controller/launch.go

The ingres class is just a argument, see this example https://github.com/kubernetes/ingress/blob/master/examples/customization/custom-vts-metrics/nginx/nginx-ingress-controller.yaml#L53

I assume in your internal / external configuration each ingress controller spins up it’s own ELB?

Yes, I have 2 ELB (internal/external) with all the workers attached in both. I'm using different nodePort for each ingress to avoid elb attach the incorrect ingress.

Or it shares the same ELB and that’s why you have your health checks on different ports?
Do you have any ingress annotations set for the ELB?

Yes, I annotated my ingress with kubernetes.io/ingress.class: "internal" or external

Reply all

Reply to author

Forward