Linkerd

534 views
Skip to first unread message

Artem Miniailo

unread,
Feb 23, 2017, 4:02:38 AM2/23/17
to Nomad
Can I use Linkerd as proxy with Nomad.
My final goal is to connect Linkerd with Zipkin to have the distributed tracing
Thanks in advance

Michael Schurter

unread,
Feb 23, 2017, 1:17:28 PM2/23/17
to Artem Miniailo, Nomad
I'd love if anyone currently using linkerd would chime in, but it appears like a good match to me -- especially with linkerd's Consul integration.

It'd be great to get a Nomad Getting Started Guide alongside their k8s and docker ones. The configuration seems pretty straightforward if the Consul discovery works well (or you have another disco mechanism internally).

I'd love to hear how it goes and if there's any way we could improve the integration.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/57145c97-4d04-4950-a0ae-8d0c0d05cbad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Gal Ben Haim

unread,
Mar 21, 2017, 4:00:45 PM3/21/17
to Nomad
I have this setup.. what I did:

- create a new bridged docker network on all nomad client hosts, let's say `docker network create nomad`
- have port 53/udp forwarded to consul (using dnsmasq) - you can also do this by running consul dns on port 53
- add localhost added to /etc/resolv.conf (i'm using systemd-resolved for this)

and this is an example job file -
job "{{env "ENV"}}-files" {
    datacenters = ["dc1"]
    type = "service"

    constraint {
        attribute = "${meta.env}"
        value = "{{env "ENV"}}"
    }

    # Configure the job to do rolling updates
    update {
        # Stagger updates every 10 seconds
        stagger = "1m"

        # Update a single task at a time
        max_parallel = 1
    }

    group "files" {
        count = {{with env "ENV" | printf "%s/services/files/count"}}{{or (key .) 1}}{{end}}

        restart {
            attempts = 10
            interval = "30s"
            delay = "1s"
            mode = "delay"
        }

        # Define a task to run
        task "files" {
            driver = "docker"

            config {
                image = "my/file-server:{{with env "ENV" | printf "%s/services/files/tag"}}{{key .}}{{end}}"

                auth = {
                    username = "{{key "dockerhub/username"}}"
                    email = "{{key "dockerhub/email"}}"
                    password = "{{key "dockerhub/password"}}"
                }

                dns_servers = ["172.17.0.1", "8.8.8.8"]
                network_mode = "nomad"
            }

            env {
                OAUTH2_URL = "http://linkerd-${NOMAD_ALLOC_ID}:5000"
                OAUTH2_INTERNAL_URL = "http://linkerd-${NOMAD_ALLOC_ID}:6000"

                {{range $key := tree "globals/services/files/env/"}}
                {{.Key}} = "{{.Value}}"{{end}}

                {{with env "ENV" | printf "%s/services/files/env/"}}
                {{range $key := tree .}}
                {{.Key}} = "{{.Value}}"{{end}}{{end}}
            }

            resources {
                cpu = 500 # 500 Mhz
                memory = 256 # 256MB
                network {
                    mbits = 1
                }
            }
        }

        task "log-forwarder" {
            driver = "exec"
            config {
                command = "/bin/bash"
                args = ["-c", "$NOMAD_TASK_DIR/remote_syslog/remote_syslog -D --hostname=$(echo $NOMAD_ALLOC_ID | cut -d- -f1) -d localhost -p 514 --tcp files=$NOMAD_ALLOC_DIR/logs/files*"]
            }

            artifact {
            }

            resources {
                cpu = 20
                memory = 20
                network {
                    mbits = 1
                }
            }
        }

        task "linkerd" {
            driver = "docker"

            config {
                image = "buoyantio/linkerd:0.9.1"
                args = ["/local/linkerd.yml"]
                network_mode = "nomad"

                port_map {
                    http = 3000
                    linkerd = 8080
                }

                labels = {
                    SERVICE_IGNORE = "true"
                }
            }

            template {
                left_delimiter = "[["
                right_delimiter = "]]"
                data = <<EOF
                    admin:
                        port: 9990

                    namers:
                    - kind: io.l5d.consul
                      host: 172.17.0.1
                      port: 8500
                      useHealthCheck: true
                      setHost: true
                      consistencyMode: stale

                    routers:
                    - protocol: http
                      label: linkerd
                      dtab:
                        /svc => /$/inet/127.1/9990
                      servers:
                        - port: 8080
                          ip: 0.0.0.0

                    - protocol: http
                      label: ingress
                      dtab:
                        /svc => /$/inet/files-[[env "NOMAD_ALLOC_ID"]]/3000
                      servers:
                        - port: 3000
                          ip: 0.0.0.0

                    - protocol: http
                      label: egress-oauth2-3000
                      dtab:
                        /svc => /#/io.l5d.consul/dc1/{{env "ENV"}}-oauth2-3000
                      servers:
                        - port: 5000
                          ip: 0.0.0.0

                    - protocol: http
                      label: egress-oauth2-9000
                      dtab:
                        /svc => /#/io.l5d.consul/dc1/{{env "ENV"}}-oauth2-9000
                      servers:
                        - port: 6000
                          ip: 0.0.0.0
                    
                    telemetry:
                    - kind: io.l5d.prometheus
                EOF

                destination = "local/linkerd.yml"
            }

            service {
                name = "{{env "ENV"}}-files"
                port = "http"
                check {
                    name = "http ping"
                    type = "http"
                    path = "/ping"
                    interval = "10s"
                    timeout = "2s"
                }
            }

            service {
                name = "{{env "ENV"}}-files-linkerd"
                port = "linkerd"
                check {
                    name = "http ping"
                    type = "http"
                    path = "/"
                    interval = "10s"
                    timeout = "2s"
                }
            }

            resources {
                cpu = 200
                memory = 200
                network {
                    port "http" {}
                    port "linkerd" {}
                }
            }
        }
    }
}

Chris Boulton

unread,
Mar 23, 2017, 6:27:26 PM3/23/17
to Nomad
We're actually going through this exercise right now, so I'd love to share what we're thinking for deployment, and also help anyone out who is getting stuck.

Our environment consists of the following right now: Nomad, Consul, Vault (no need to talk about this here), and linkerd. We've been heavy users of Consul for a while, and the other three are relatively new additions into our environment.

We have a lot of services we'll be pushing into our Nomad environment soon, but we'll continue to have to support applications outside of it. These services can talk gRPC (HTTP 2) and HTTP 1 (mostly JSON REST APIs). Both of these play into our design decisions for how we've tied Nomad/linkerd/Consul together:
  • We need to be able to run linkerd on hosts that don't have Docker/Nomad
  • Our services don't necessarily (and shouldn't have to know) how to talk TLS, but we would like to secure transport between hosts
  • HTTP 1.1 services will route traffic based on the HTTP Host header, gRPC services based on the gRPC path
  • All our services (the Nomad and outside Nomad) services register services in Consul using Registrator and we want to use this same lookup information to route via linkerd
  • We don't want to limit ourselves in a way that we can't do green/blue or canary deploys of services where a small amount of traffic is sent one path, and the rest by way of the normal service path
  • We didn't want to use a transparent proxy approach with linkerd, and instead favoured explicit configuration of services to use the linkerd endpoints
To that end (mainly to cover the first point), we deploy linkerd on all hosts that will communicate with our services outside of Docker (we have it packaged as a Debian package). This is the "per-host" deployment scenario linkerd talk about here: https://linkerd.io/in-depth/deployment/. If you are a Nomad only environment, you could accomplish the same thing by deploying linkerd as a Nomad service, so you end up with one instance on every Nomad host.

To secure communications into and between hosts, we use the "linker to linker" configuration pattern discussed on that same page. To talk from service A to service B, you enter service A via linkerd, it reaches out to linkerd (TLS added here), which routes to the destination linkerd (TLS removed here), and then to the destination service. This results in two routers in linkerd: an outbound router for traffic leaving a service to hit another service, and an ingress router for traffic received from other linkerd instances.

To handle both HTTP 1.1 and gRPC services, we define routers for each. This is where the behaviour is different.

For a HTTP 1.1 service, we expect that the HTTP host changes depending upon the service you want to talk to. For example, if you wanted to talk to our auth service ("auth.service.consul") via the linkerd path, you should request it via http://auth.linkerd:4140. To handle this DNS resolution, we run a local Unbound instance on every host in our environment. It's configured to resolve "*.linkerd" to a dummy link-local interface (169.254.1.1), so containers are able to resolve that name. linkerd when it sees a request like this is configured to strip ".linkerd" from the hostname, and ask Consul for a list of backend services. The request is then forwarded to the destination linkerd, which then maps it to a locally running instance.

For a gRPC service, we'll be handling things a little differently. Right now it's a fixed list in our prototype environment, but let me describe how we want it to work: In this instance, we'll be running linkerd's namerd, as well as an internal service one of our engineers has built called "Meshroute". Meshroute looks at our Consul services, and discovers any service with a "grpc:xyz" tag. xyz is extracted, and is used to build a routing table of gRPC service name/path to Consul service name. This dtab routing table is stored in Consul's K/V store, and namerd is configured to go look it up. It looks something like this:

  /svc/bigcommerce.rpc.storeconfig.StoreConfig  => /#/io.l5d.consul/.local/storeconfig;

In this case, all our services are configured with a single RPC host: http://linkerd:4142. The path based routing provided by linkerd takes care of getting you to the right spot. In the future we can use this same tag extraction process to provide advanced behaviour like traffic splitting, weighting, etc for services that share the same gRPC tag, by building a different dtab configuration and pushing it into Consul for namerd to consume.

The important thing to understand about "linker to linker" configurations is that you need to use both the "port" and "localhost" transformers. When traffic leaves a linkerd instance, it should hit the linkerd instance running on the destination host instead of the service directly. To do this, you apply a port transformation on the looked up hosts which transforms the port to the linkerd port. On the receiving side, you do the same Consul service lookup again but you need to filter it down to the service instances running on that host. To do this you use the "localhost" transformer which will basically strip namer lookup result down to only the services that have an IP address that exists on that host.

If you want to see how all this fits together, here's a linkerd configuration which we have in our environment right now: https://gist.github.com/chrisboulton/8832db12e97f77f818ac629f81378274

Note: because of how we do the *.linkerd routing, and how our services expect a HTTP host header to be set that matches the consul service (*.service.consul), you'll see two namers in there: one that overrides the HTTP host based on the Consul service name, and one that doesn't. The "consul_to_linker" service doesn't mangle the host header, and is applied only on outbound requests to other linkerd instances. When we finally route to the destination service, we want the host header overwritten with the Consul service name.

Chris


On Thursday, February 23, 2017 at 10:17:28 AM UTC-8, Michael Schurter wrote:
I'd love if anyone currently using linkerd would chime in, but it appears like a good match to me -- especially with linkerd's Consul integration.

It'd be great to get a Nomad Getting Started Guide alongside their k8s and docker ones. The configuration seems pretty straightforward if the Consul discovery works well (or you have another disco mechanism internally).

I'd love to hear how it goes and if there's any way we could improve the integration.
On Thu, Feb 23, 2017 at 1:02 AM, Artem Miniailo <artem.m...@gmail.com> wrote:
Can I use Linkerd as proxy with Nomad.
My final goal is to connect Linkerd with Zipkin to have the distributed tracing
Thanks in advance

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.

Alex Dadgar

unread,
Mar 26, 2017, 6:21:24 PM3/26/17
to Nomad, Chris Boulton
Awesome write ups Chris and Gal! Sure it will help a lot of people!

Thanks,
Alex Dadgar
Reply all
Reply to author
Forward
0 new messages