Envoy on a Docker Swarm and Configuration

[Ashley Aitken]

Hi,

Thanks to Lyft for open sourcing Envoy, it seems like a great solution to a lot of problems.

With my limited knowledge I am trying to understand:

1. Will it be difficult to run Envoy in a Docker Swarm, where Envoy would be a global service deployed on each host (probably in a Docker container)?  I assume not, but it would be good to know if anyone has done this and how it went.

2. I was really excited about Enjoy until I started to read about the configuration being complex and the recommendation using a configuration generator (always a bad sign IMHO :-).  Is it really this complex? The achilles heal of Envoy?

I would very much like to use Envoy but I would hope for mostly "configuration by default" and minimal configuration needed to get started.  

Thanks in advance for any thoughts.

Cheers,

Ashley.

PS It seems like me that it would be great to have service discovery built-in to Envoy, at least as a default.  Nodes and services could just register with the service etc.  Setting up a SDS is another task keeping me from enjoying Envoy :-)

[Ashley Aitken]

PPS  Sorry, I had forgot that you did also open source your Discovery Service, and I can understand why it is best to have this as a separate service, so to speak.  But still, couldn't the services just register with the local Envoy proxy?

[Harvey Tuch]

On 2), configuration isn't that complicated, you can start with something minimal example like https://github.com/lyft/envoy/blob/master/configs/google_com_proxy.json and then build up the config based on https://lyft.github.io/envoy/docs/configuration/configuration.html. The provided template driven config generation is based on the Lyft setup, which reflects an advanced deployment demonstrating the full set of capabilities, and where you want a separation of concerns, e.g. having route config specified/generated in a different place to the listener spec.

Maybe it would be helpful to have some intermediate config, more capable than the simple proxy example but without the templating.

--
You received this message because you are subscribed to the Google Groups "Envoy Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/159ea555-9483-410a-8a0c-2490e59d7998%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ashley Aitken]

Thank you Harvey, that is good to hear.  I skipped over the smoke test but will look more closely.  I really just want a default configuration of pretty much everything (of course, besides our specific details).  

Can you (or someone) please tell me if the configuration is a global configuration (i.e. specified in one place for entire mesh, perhaps with different config for different proxies) or on each Envoy proxy.

[Harvey Tuch]

The configuration is currently per-proxy, since it is sourced from the local filesystem (via the -c flag). There has been discussion about moving some of this to a network/RPC model in the future, where a configuration server may be used, but there is no design for this yet. 

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/73bd7a25-48ca-4e59-bc00-05eb688cb4a4%40googlegroups.com.

[Matt Klein]

The configuration is currently per-proxy, since it is sourced from the local filesystem (via the -c flag). There has been discussion about moving some of this to a network/RPC model in the future, where a configuration server may be used, but there is no design for this yet. 

This is not entirely true. We do allow a portion of the configuration to be provided by a remote server at this point (the portion that is most likely to frequently change). See here for an overview of static and dynamic configuration options.

I was really excited about Enjoy until I started to read about the configuration being complex and the recommendation using a configuration generator (always a bad sign IMHO :-).  Is it really this complex? The achilles heal of Envoy?

Unfortunately, what Envoy does is extremely complicated. There is no reasonable way to have a default configuration that will work for everyone. We have tried to build a system of layered complexity. I.e., simple deployments can use simple configurations that are static. More complex deployments can use additional knobs which may include dynamic configuration servers.

At this point, Envoy is being used across a variety of different infrastructures which range from fully custom to k8s. It's my opinion that for the average consumer, Envoy is best used via an orchestration layer that is specific to the infrastructure in question. For example, the Istio system is going to make it trivial to run Envoy in k8s. Hopefully someone will come along and make it easy to use in docker swarm, etc.

Thanks,

Matt

To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.

To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/73bd7a25-48ca-4e59-bc00-05eb688cb4a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Envoy Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.

To post to this group, send email to envoy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/CAA4W8ZmvC6LXsNXSAWdXqu1fXN5Bo8UqQXcc7G%2BZLbBDarbG%3Dw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Matt Klein

Software Engineer

mkl...@lyft.com / 206.327.4515

[gilli...@gmail.com]

Thanks very much for you reply Matt.

On Tuesday, 7 March 2017 01:12:07 UTC+8, Matt Klein wrote:

I was really excited about Enjoy until I started to read about the configuration being complex and the recommendation using a configuration generator (always a bad sign IMHO :-).  Is it really this complex? The achilles heal of Envoy?

Unfortunately, what Envoy does is extremely complicated. There is no reasonable way to have a default configuration that will work for everyone. We have tried to build a system of layered complexity. I.e., simple deployments can use simple configurations that are static. More complex deployments can use additional knobs which may include dynamic configuration servers.

I guess a few more example configurations could be useful.  

At this point, Envoy is being used across a variety of different infrastructures which range from fully custom to k8s. It's my opinion that for the average consumer, Envoy is best used via an orchestration layer that is specific to the infrastructure in question. For example, the Istio system is going to make it trivial to run Envoy in k8s. Hopefully someone will come along and make it easy to use in docker swarm, etc.

Sure, that would be fantastic.  Docker Swarm mode already provides some very nice (easy to start with) facilities, like overlay networks with automatically setup DNS server for service discovery etc.  I think Docker Swarm and Envoy could be a great match.  

Thanks also again to Harvey.

 

On Mon, Mar 6, 2017 at 8:12 AM, 'Harvey Tuch' via Envoy Users <envoy...@googlegroups.com> wrote:

The configuration is currently per-proxy, since it is sourced from the local filesystem (via the -c flag). 

 

So in a quite simple implementation of just service-to-service proxying could the configuration for each node be the same?  

I assume one may need a different configuration for a front proxy etc.

Cheers,

Ashley.

 

[Matt Klein]

So in a quite simple implementation of just service-to-service proxying could the configuration for each node be the same?  

Yes, this is possible, but it really depends on the details of your infrastructure.  

I assume one may need a different configuration for a front proxy etc.

Most likely.

I would encourage you to start with the reference configurations. The use of Jinja is mainly to make them easier to read and avoid duplication. The reality is that in almost any reasonably complex Envoy deployment, the configurations are going to need to be machine generated to some extent (whether by using SDS/CDS/RDS or via a complex static config). The reference configurations demonstrate most Envoy features and are a rough approximation of how we currently run Envoy at Lyft (we will be moving to a centralized configuration service that provides CDS/RDS in Q2).

--

You received this message because you are subscribed to the Google Groups "Envoy Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/60e109db-512a-465c-a5a4-d4fa18949a36%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.