Consul Connect Sidecar Proxies - Ingress?

[Daniel Middleton]

Hey! I'm trying to understand the topology around Consul Connect Sidecar Proxies (with Nomad). It's my understanding that for every deployed Task Group, a separate Sidecar Proxy is deployed (which allows for Intentions, etc). This suggests each proxy listens on a different port as there could be multiple per host. In my current setup on Rancher, the way it works is that there's one proxy/LB per host listening on a static port, and Host headers are used to route traffic to the relevant "Task" (Docker containers in my case). In front of this is, for example, an AWS ELB which forwards to the proxy/LB port on any of the hosts. How would Consul Connect/Nomad handle this?

What I'm used to:

User (request https://someservice.someco.tld) ---> DNS ---> ELB ---> Nodes (all run Proxy/LB on 443 -> read Host header) ---> someservice container

What I don't get with Consul's proxies, is that because there's multiple per host (one per Task Group) I don't have a single proxy port to point the ELB at across the nodes.

Thanks!

[pba...@hashicorp.com]

Hey Daniel,

Yes you're right in your understanding. Currently Connect support only the sidecar model which typically requires a proxy per distinct identity,

It's possible if you have say VMs which run separate services but that are all one logical identity to share a proxy and there might be reasons to do that.

That said, Connect is not built internally assuming side-car is the only valid model. We intend to support Ingress/Edge router, centralised gateway and several other patterns that are important in different places in the datacenter or different deployment models.

Right now the built in proxy only supports sidecar mode but soon support for Envoy and other proxies will make it possible to do more sophisticated things.

One issue thought is that, right now we are only proxying at L4 which means we can't support multiple identities based on hostname or path routing rules which is what your current setup likely does. It is theoretically possible to do this even at L4 using SNI in the TLS handshake to choose the correct certificate to present and then route the connection but we've not got an example of that up yet.

When we have Envoy support, it's at least possible in theory to setup SNI to allow a single listener to present different certificates. We might not support that on day 1 but we'll need to for the Gateway and Ingress use cases we plan to build tooling around.

Hope this is helpful context!

[Daniel Middleton]

Hey! Thank you for your reply, I’m glad I’ve not completely misunderstood the current implementation.

So, just to wrap this up (until we have Envoy, etc), what would you personally do to expose, for example a simple Nginx/Apache service (Task Group) to an end user? Or am I right in thinking that due to your points around the built-in proxy being L4-only, it’s not yet possible other than to use unique ports and distribute them out the end users manually.

Thanks again.

-- 
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
Community chat: https://gitter.im/hashicorp-consul/Lobby
--- 
You received this message because you are subscribed to a topic in the Google Groups "Consul" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/consul-tool/QhevZvVBETM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/d7eae69e-010c-4e3b-b1e8-12bd9586d2e9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Justin DynamicD]

This might be a bit nutz (read: I've not tried this yet) but if we accept what is essentially a proxy endpoint per identity for communication outside your service mesh, could you not do something similar to the following:

1.  Dedicate 2 systems to be "gateway proxies" (publish services on unique ports as discussed), make sure they also register in Consul with a tag when they do.

2.  Run Nginx/HAProxy on 443 at layer 7 and use consul-template to build it's config file only for published services that are tagged to match.  You can route layer 7  to a layer 4 endpoint just fine.

Basically it's two sets of proxies ... and thus a bit messy, but should work.  Everything passes through your "proxy gateway" on random/unique ports at layer 4 and then you have a simpler layer 7 proxy to map 443 for services that support it (it just points at first set of proxies).  Yeah Envoy would make this easier.  But it should get you off the ground.

[Daniel Middleton]

Hi Justin!

I did consider something along those lines, although I’d run an identical “gateway proxy” on each Nomad Client host, which in my case would allow an ELB to target the entire Nomad Client ASG while maintaining consistency and scalability. Perhaps the “gateway proxy” itself could be a Nomad Task exposing the same static port on each host. It seems we’re the real guinea pigs for Nomad Ingress!

To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/c932852e-cf94-4374-8133-8f8fab3a91a2%40googlegroups.com.

[Justin DynamicD]

Hmmm that should work actually.  I currently run a pair of HAProxy servers with consul-template and it works great, but I've not toggled the connect sidecars on yet.  As I think about it though, as long as the sidecars are what register with the tag things should just ... continue to work.  You're right ... there's no reason to dedicate another set of layer 4 proxies ... I was going a bit too far.  

The main reason I use HAProxy instead an ELB is I simply couldn't find an elegant way to dynamically configure the ELB with consul data.  A Nomad ASG is a nonissue from a configuration standpoint because Consul keeps track of all of that for me.  I just have two Proxies (in separate AZs) on fixed IPs so I could point wildcard dns resolution to them and a matching wildcard cert.

But yeah ... you COULD replace that with an ELB perhaps,  and use a nomad service definition so your proxy was running on every host.  But configuring custom load balance options would get ugly fast (if a service couldn't use round robbin and needed sourceip sessions how do coordinate that between ELB AND the proxy service?).  Right now I use the KV store to hold those options, but I've not thought about how that applies/conflicts with consul connect.  The connect proxy is just TLS encrypting, isn't it?  It's not actually doing any masking of available endpoints and just letting consul expose all valid ones, right?  Or is it?

Oh yes ... guinea pig indeed ...

[Daniel Middleton]

Thanks for sharing your implementation, certainly a lot to think about.

However...if you have a minute to review the attached diagram(s)...could it really be this simple of am I being naive?

The first one is what I think is the simplest form of ingress using something like Envoy (as a Nomad Task).

The second is just the same zoomed out for the sake of completeness.

I'd be really grateful if you have any points to offer on them.

Ingress:

Whole shebang: