Re: linkerd routing to endpoints external to k8s cluster

468 views
Skip to first unread message

Kevin Lingerfelt

unread,
Apr 10, 2017, 10:22:36 PM4/10/17
to Thomas, Betson, linkerd-users
(moving this thread to linkerd-users and bcc'ing the individuals who were previously cc'd on it)

Hey Betson,

This is a good question. It does indeed look like the transformers you're using are filtering out the external address. Those transformers are intended for internal routing within a Kubernetes cluster, so I'm not surprised that they're a bit unwieldy when trying to send traffic externally.

I came up with a configuration that should work for you -- it's in the attached linkerd.yml file. Basically, you'd still need an external router that sends traffic externally, without any transformers, and you also need to modify your outgoing and incoming routers to forward external traffic to the external router. So the outgoing router's dtab changes to:

/external   => /#/io.l5d.k8s/default/incoming/l5d;
/srv        => /#/io.l5d.k8s/default/http;
/host       => /srv;
/svc        => /host | /external;
/host/world => /srv/world-v1;

That forces all destinations not found in service discovery to forward to the incoming router. The incoming router's dtab changes to:

/external   => /#/io.l5d.k8s/default/external/l5d;
/srv        => /#/io.l5d.k8s/default/http;
/host       => /srv;
/svc        => /host | /external;
/host/world => /srv/world-v1;

And that forces all destinations not found in service discovery to forward to the external router, which is able to then route the request externally without interference from transformers.

While this approach works, it's not great. It requires 3 router hops for external traffic, which is wasteful. It would be great if the outgoing router could just route externally immediately, without the extra hops. To do that though, I think we'd need some sort of dtab convention that would allow you to create a dtab entry that disables the transformer for the remainder of the resolution. Something like:

/external => /%/none/$/io.buoyant.rinet;

But I haven't given that too much thought. Anyway, hope this helps in the meantime.

Kevin

On Mon, Apr 10, 2017 at 5:41 PM, Thomas, Betson <Betson...@charter.com> wrote:
@linkerd:
I was working with @esbie regarding this issue on slack today and we've run into a bit of a roadblock. She suggested that it appears the daemonset transformer is filtering out the relevant IPs.

I've upgraded to 1.0.0-rc2 to match the version of the example that was provided, but I'm still not able to proxy to external endpoints from the main outgoing router. We also tried setting up a dedicated external router without a transformer specified to which the outgoing router will forward to. curling directly to that external router successfully fetches www.google.com on 1.0.0-rc2 (but fails on 0.9.1). However, attempting to route through the main outgoing router for the same http://www.google.com call results in a failure as before. 
Note: linkerd is installed as a k8s daemonset

Is there a dtab config that will successfully fallback to this external router through the primary outgoing router? Alternatively, is there a way to distinguish external calls from internal calls so that we can route external calls exclusively to the dedicated external router and avoid the primary router entirely, even on the app side, considering the Downward API configuration?

Please find attached both versions of the yaml file, with and without the dedicated external router. Thanks

curl to dedicated external router on 0.9.1
http_proxy=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}"):4142 curl -vL http://www.google.com
* Rebuilt URL to: http://www.google.com/
*   Trying 52.38.172.180...
* TCP_NODELAY set
> User-Agent: curl/7.51.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 502 Bad Gateway
< l5d-err: Unable+to+match+%2F%24%2Fio.buoyant.rinet%2F80%2Fwww.google.com+with+available+names%3A+%2F%24%2Fio.buoyant.rinet%2F443%2F%7Bhostname%7D
< Content-Type: text/plain
< Content-Length: 110
< 
* Curl_http_done: called premature == 0
Unable to match /$/io.buoyant.rinet/80/www.google.com with available names: /$/io.buoyant.rinet/443/{hostname}

curl to outgoing router on 1.0.0-rc2
http_proxy=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}"):4140 curl -vL http://www.google.com
* Rebuilt URL to: http://www.google.com/
*   Trying 52.88.203.51...
* TCP_NODELAY set
> User-Agent: curl/7.51.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 502 Bad Gateway
< l5d-err: No+hosts+are+available+for+%2Fsvc%2Fwww.google.com%2C+Dtab.base%3D%5B%2Fsvc%3D%3E%2F%24%2Finet%2Flocalhost%2F4142%3B%2Fsrv%3D%3E%2F%23%2Fio.l5d.k8s%2Fdefault%2Fhttp+%26+%2F%23%2Fio.l5d.k8s%2Fzipkin%2Fhttp+%7C+%2F%23%2Fio.l5d.k8s%2Fhello%2Fhttp%3B%2Fsrv%3D%3E%2F%23%2Fio.l5d.k8s%2Fcnet%2Fhttp%3B%2Fhost%3D%3E%2Fsrv%3B%2Fsvc%3D%3E%2Fhost%3B%2Fzpk%3D%3E%2Fhost%2Fiam-red%3B%2Fhlo%3D%3E%2Fhost%2Fhello%5D%2C+Dtab.local%3D%5B%5D.+Remote+Info%3A+Not+Available
< Content-Type: text/plain
< Content-Length: 311
< 
* Curl_http_done: called premature == 0
No hosts are available for /svc/www.google.com, Dtab.base=[/svc=>/$/inet/localhost/4142;/srv=>/#/io.l5d.k8s/default/http & /#/io.l5d.k8s/zipkin/http | /#/io.l5d.k8s/hello/http;/srv=>/#/io.l5d.k8s/cnet/http;/host=>/srv;/svc=>/host;/zpk=>/host/iam-red;/hlo=>/host/hello], Dtab.local=[]. Remote Info: Not Available

curl to external router on 1.0.0-rc2
http_proxy=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}"):4142 curl -vL http://www.google.com
* Rebuilt URL to: http://www.google.com/
*   Trying 54.213.38.78...
* TCP_NODELAY set
> User-Agent: curl/7.51.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
< Date: Tue, 11 Apr 2017 00:40:02 GMT
< Expires: -1
< Cache-Control: private, max-age=0
< Content-Type: text/html; charset=ISO-8859-1
< P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
< Server: gws
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< Set-Cookie: NID=101=CiYnsTdoV7QAmvnD-4l6i_hHaX86tQ5RqN9jXMFyJOgRAe7mwIq-dKJzulmV8W8Aq_CD2-tB6vT7GZ6qQ3eKzplzTR0OVJ_ZahtcYgBKYX9ojG2ro9IZWP2VeTFyUb0BM1iu22udA5cVH95G; expires=Wed, 11-Oct-2017 00:40:02 GMT; path=/; domain=.google.com; HttpOnly
< Accept-Ranges: none
< Vary: Accept-Encoding
< Via: 1.1 linkerd
< Transfer-Encoding: chunked
< 
<!doctype html>
[truncated]

-Betson


On Apr 7, 2017, at 5:14 PM, Thomas, Betson <Betson...@charter.com> wrote:

Thanks, that helps clarify. Yes, we're looking to route to the real google.com as an analog to a number of different public and internal, legacy services that we need access to outside the k8s cluster.

I updated the dtab with the entries to the outgoing router and I'm still not able to route. Note that you may want to consider versioning the examples. Looks like the client section structure is different in 0.9.1: https://linkerd.io/config/0.9.1/linkerd/index.html#client-tls

I've attached the full yaml file. 

Here is the curl output:
http_proxy=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}"):4140 curl -v http://www.google.com
* Rebuilt URL to: http://www.google.com/
*   Trying 52.35.78.67...
* TCP_NODELAY set
> User-Agent: curl/7.51.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 502 Bad Gateway
< l5d-err: No+hosts+are+available+for+%2Fsvc%2Fwww.google.com%2C+Dtab.base%3D%5B%2Fph%3D%3E%2F%24%2Fio.buoyant.rinet%3B%2Fsvc%3D%3E%2Fph%2F80%3B%2Fsvc%3D%3E%2F%24%2Fio.buoyant.porthostPfx%2Fph%3B%2Fsrv%3D%3E%2F%23%2Fio.l5d.k8s%2Fdefault%2Fhttp+%26+%2F%23%2Fio.l5d.k8s%2Fzipkin%2Fhttp+%7C+%2F%23%2Fio.l5d.k8s%2Fhello%2Fhttp%3B%2Fsrv%3D%3E%2F%23%2Fio.l5d.k8s%2Fcnet%2Fhttp%3B%2Fhost%3D%3E%2Fsrv%3B%2Fsvc%3D%3E%2Fhost%3B%2Fzpk%3D%3E%2Fhost%2Fiam-red%3B%2Fhlo%3D%3E%2Fhost%2Fhello%5D%2C+Dtab.local%3D%5B%5D.+Remote+Info%3A+Not+Available
< Content-Type: text/plain
< Content-Length: 355
< 
* Curl_http_done: called premature == 0
No hosts are available for /svc/www.google.com, Dtab.base=[/ph=>/$/io.buoyant.rinet;/svc=>/ph/80;/svc=>/$/io.buoyant.porthostPfx/ph;/srv=>/#/io.l5d.k8s/default/http & /#/io.l5d.k8s/zipkin/http | /#/io.l5d.k8s/hello/http;/srv=>/#/io.l5d.k8s/cnet/http;/host=>/srv;/svc=>/host;/zpk=>/host/iam-red;/hlo=>/host/hello], Dtab.local=[]. Remote Info: Not Available

https_proxy=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}"):4140 curl -v https://www.google.com
* Rebuilt URL to: https://www.google.com/
*   Trying 52.35.78.67...
* TCP_NODELAY set
* Establish HTTP proxy tunnel to www.google.com:443
> CONNECT www.google.com:443 HTTP/1.1
> User-Agent: curl/7.51.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 502 Bad Gateway
< l5d-err: No+hosts+are+available+for+%2Fsvc%2Fwww.google.com%3A443%2C+Dtab.base%3D%5B%2Fph%3D%3E%2F%24%2Fio.buoyant.rinet%3B%2Fsvc%3D%3E%2Fph%2F80%3B%2Fsvc%3D%3E%2F%24%2Fio.buoyant.porthostPfx%2Fph%3B%2Fsrv%3D%3E%2F%23%2Fio.l5d.k8s%2Fdefault%2Fhttp+%26+%2F%23%2Fio.l5d.k8s%2Fzipkin%2Fhttp+%7C+%2F%23%2Fio.l5d.k8s%2Fhello%2Fhttp%3B%2Fsrv%3D%3E%2F%23%2Fio.l5d.k8s%2Fcnet%2Fhttp%3B%2Fhost%3D%3E%2Fsrv%3B%2Fsvc%3D%3E%2Fhost%3B%2Fzpk%3D%3E%2Fhost%2Fiam-red%3B%2Fhlo%3D%3E%2Fhost%2Fhello%5D%2C+Dtab.local%3D%5B%5D.+Remote+Info%3A+Not+Available
< Content-Type: text/plain
< Content-Length: 359
< 
* Received HTTP code 502 from proxy after CONNECT
* Curl_http_done: called premature == 0
* Closing connection 0
curl: (56) Received HTTP code 502 from proxy after CONNECT

-Betson

<proxy-cnet-linkerd.yml>

On Apr 7, 2017, at 1:46 PM, Alex Leong <al...@buoyant.io> wrote:

There's an example dtab that does this here: https://github.com/linkerd/linkerd/blob/master/linkerd/examples/proxy.yaml

You're also correct that https calls from the app will bypass the http_proxy variable.  You can set the https_proxy variable to capture https calls from the app.

On Fri, Apr 7, 2017 at 10:40 AM, Oliver Gould <v...@buoyant.io> wrote:
Hi Betson,

So, it looks like the relevant part of your dtab is:

  /srv=>/#/io.l5d.k8s/default/http & /#/io.l5d.k8s/zipkin/http | /#/io.l5d.k8s/hello/http;
  /host=>/srv;
  /svc=>/host;
To which Kubernetes services do you want requests to /svc/www.google.com to be routed?

As it is, /svc/www.google.com is rewritten as /host/www.google.com is rewritten as as /#/io.l5d.k8s/default/http/www.google.com & /#/io.l5d.k8s/zipkin/http/www.google.com | /#/io.l5d.k8s/hello/http/www.google.com. This final resolution looks for a service called www.google.com in the default namespace on the http port unioned with the service called www.google.com in the zipkin namespace on the http port. If neither of those services exist, it tries to find the service called www.google.com in the hello namespace on the http port.

I don't believe that it's possible to have a kubernetes service called www.google.com, however.  So, this route can be created in one of two ways:

1. Add an explicit dtab rule like /host/www.google.com => /srv/google (which will cause you to look for a service called google instead of www.google.com).
2. You can use a rewriting namer to programatticaly turn a name like /host/www.google.com into, for example, /srv/com/google/www, and use dtabs to extract the name further.

If you actually want to contact the real www.google.com, you can use the /$/inet or /$/io.buoyant.rinet namers to cause linkerd to resolve the dns name.  This could be added as an additional fallback so that this dns lookup is only performed when the service isn't in kubernetes.

Hope this helps!

On Fri, Apr 7, 2017 at 7:31 AM Thomas, Betson <Betson...@charter.com> wrote:
Hello,

I posted this in the slack chat, but it looks like there have been a few posts afterwards and want to make sure this doesn't get buried.

We are having some issues being able to route to endpoints external to the k8s cluster via linkerd. curl to http://www.google.com proxied through linkerd fails, but the same curl to the HTTPS endpoint does route. I'm not sure if this is because the HTTPS call is bypassing linkerd. Do we need to configure anything specifically for this to work in dtabs or otherwise? 

linkerd is installed as a k8s daemonset. Sample curl:

http_proxy=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}"):4140 curl -v http://www.google.com
* Rebuilt URL to: http://www.google.com/
*   Trying [...]...
* TCP_NODELAY set
* Connected to [...] port 4140 (#0)
> User-Agent: curl/7.51.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 502 Bad Gateway
< l5d-err: No+hosts+are+available+for+%2Fsvc%2Fwww.google.com%2C+Dtab.base%3D%5B%2Fsrv%3D%3E%2F%23%2Fio.l5d.k8s%2Fdefault%2Fhttp+%26+%2F%23%2Fio.l5d.k8s%2Fzipkin%2Fhttp+%7C+%2F%23%2Fio.l5d.k8s%2Fhello%2Fhttp%3B%2Fsrv%3D%3E%2F%23%2Fio.l5d.k8s%2Fcnet%2Fhttp%3B%2Fhost%3D%3E%2Fsrv%3B%2Fsvc%3D%3E%2Fhost%3B%2Fzpk%3D%3E%2Fhost%2Fiam-red%3B%2Fhlo%3D%3E%2Fhost%2Fhello%5D%2C+Dtab.local%3D%5B%5D.+Remote+Info%3A+Not+Available
< Content-Type: text/plain
< Content-Length: 282
< 
* Curl_http_done: called premature == 0
* Connection #0 to host [...] left intact
No hosts are available for /svc/www.google.com, Dtab.base=[/srv=>/#/io.l5d.k8s/default/http & /#/io.l5d.k8s/zipkin/http | /#/io.l5d.k8s/hello/http;/host=>/srv;/svc=>/host;/zpk=>/host/iam-red;/hlo=>/host/hello], Dtab.local=[]. Remote Info: Not Available

Thanks

-Betson
The contents of this e-mail message and
any attachments are intended solely for the
addressee(s) and may contain confidential
and/or legally privileged information. If you
are not the intended recipient of this message
or if this message has been addressed to you
in error, please immediately alert the sender
by reply e-mail and then delete this message
and any attachments. If you are not the
intended recipient, you are notified that
any use, dissemination, distribution, copying,
or storage of this message or any attachment
is strictly prohibited.



The contents of this e-mail message and
any attachments are intended solely for the
addressee(s) and may contain confidential
and/or legally privileged information. If you
are not the intended recipient of this message
or if this message has been addressed to you
in error, please immediately alert the sender
by reply e-mail and then delete this message
and any attachments. If you are not the
intended recipient, you are notified that
any use, dissemination, distribution, copying,
or storage of this message or any attachment
is strictly prohibited.

linkerd.yml

Kevin Lingerfelt

unread,
Apr 11, 2017, 7:48:03 PM4/11/17
to Thomas, Betson, linkerd-users
Hey Betson,

Alex pointed out that the io.l5d.k8s.daemonset transformer can also be applied to a namer, and doesn't have to be present on the router's interpreter. In that case you could move the outgoing router's transformer to a separate namer, and use the outgoing router to route external requests directly. I've attached an updated linkerd.yml that has this setup. This is a big improvement over the previous setup that I proposed.

In this setup you no longer need the external router. You would need to define two namers -- one with the io.l5d.k8s.daemonset transformer applied for the outgoing router, and one without the transformer for the internal router. You can distinguish the two namers by using the prefix attribute:

namers:
- kind: io.l5d.k8s
  experimental: true
  host: localhost
  port: 8001
- kind: io.l5d.k8s
  experimental: true
  prefix: /daemonset
  host: localhost
  port: 8001
  transformers:
  - kind: io.l5d.k8s.daemonset
    namespace: default
    port: incoming
    service: l5d

And then your outgoing router config changes to:

routers:
- protocol: http
  label: outgoing
  dtab: |
    /ph         => /$/io.buoyant.rinet;
    /external   => /$/io.buoyant.porthostPfx/ph | /ph/80;
    /srv        => /#/daemonset/default/http;
    /host       => /srv;
    /svc        => /host | /external;
    /host/world => /srv/world-v1;
  servers:
  - port: 4140
    ip: 0.0.0.0
  client:
    kind: io.l5d.static
    configs:
    - prefix: /$/io.buoyant.rinet/443/{hostname}
      tls:
        commonName: "{hostname}"
  responseClassifier:
    kind: io.l5d.retryableRead5XX

Updated dtab, no transformer, added client section. The incoming router doesn't need to change in this setup. Hope that helps.

Kevin

On Tue, Apr 11, 2017 at 2:06 PM, Thomas, Betson <Betson...@charter.com> wrote:
Thanks Kevin. That worked for us. Any idea on rough timeline for official 1.0 release and if you think the dtab spec for optimized routing may be included?

-Betson

<linkerd.yml>

linkerd.yml

Kevin Lingerfelt

unread,
Apr 14, 2017, 8:52:37 PM4/14/17
to Thomas, Betson, linkerd-users
Hey Betson,

Glad it's working! As for linkerd 1.0, we don't have an official date yet but it's close; rc3 will be out next week. In the meantime you can track progress on the 1.0 milestone in github here:


Kevin

On Fri, Apr 14, 2017 at 4:26 PM, Thomas, Betson <Betson...@charter.com> wrote:
Kevin,

Appreciate the followup. Just tested this out and it looks good! 
Any rough timeline on official 1.0 release?

Thanks!

-Betson

Betson Thomas

unread,
May 11, 2017, 7:53:00 PM5/11/17
to linkerd-users
Thanks Kevin. We are finding that connecting to external HTTPS endpoints returns an HTTP 405 error with this configuration. This is the behavior for both 1.0.0-rc2 as well as the latest 1.0.0 release. (Standard HTTP is functioning fine)

-Betson

Kevin Lingerfelt

unread,
May 11, 2017, 8:12:52 PM5/11/17
to Betson Thomas, linkerd-users
Hey Betson,

Interesting. Coincidentally, Alex just added an egress k8s config example in the linkerd-examples repo, which also handles TLS. Can you compare your setup to the config here:


I think that should hopefully fix it -- let us know how it goes.

Kevin


--
You received this message because you are subscribed to the Google Groups "linkerd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to linkerd-users+unsubscribe@googlegroups.com.
To post to this group, send email to linker...@googlegroups.com.
Visit this group at https://groups.google.com/group/linkerd-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/linkerd-users/9142a4e0-e184-464e-aa32-896cbb857efc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Betson Thomas

unread,
May 12, 2017, 1:59:00 AM5/12/17
to linkerd-users
Hey Kevin,

I loaded the linked sample egress yaml without changes and I'm seeing the same issue. The HTTP version of the following resolves and responds fine.

https_proxy=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}"):4140 curl -vL https://www.google.com

* Rebuilt URL to: https://www.google.com/
* Trying ......
* TCP_NODELAY set
* Connected to redacted.elb.amazonaws.com (...) port 4140 (#0)

* Establish HTTP proxy tunnel to www.google.com:443
> CONNECT www.google.com:443 HTTP/1.1
> Host: www.google.com:443
> User-Agent: curl/7.51.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 405 Method Not Allowed
< Content-Type: text/html; charset=UTF-8
< Referrer-Policy: no-referrer
< Content-Length: 1592
< Date: Fri, 12 May 2017 05:36:41 GMT
< Alt-Svc: quic=":443"; ma=2592000; v="37,36,35"
< l5d-success-class: 1.0
< Via: 1.1 linkerd
<
* Received HTTP code 405 from proxy after CONNECT

* Curl_http_done: called premature == 0
* Closing connection 0
curl: (56) Received HTTP code 405 from proxy after CONNECT

-Betson
To unsubscribe from this group and stop receiving emails from it, send an email to linkerd-user...@googlegroups.com.

Kevin Lingerfelt

unread,
May 12, 2017, 2:04:05 PM5/12/17
to Betson Thomas, linkerd-users
Hi Betson,

Thanks for sending the full curl output. It appears that https_proxy attempts to send a CONNECT request, which is not supported by linkerd. More info about that here:


In my testing of HTTPS traffic, I've been using:

http_proxy=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}"):4140 curl -vL http://www.google.com:443

And that routes successfully.

Kevin

To unsubscribe from this group and stop receiving emails from it, send an email to linkerd-users+unsubscribe@googlegroups.com.

To post to this group, send email to linker...@googlegroups.com.
Visit this group at https://groups.google.com/group/linkerd-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/linkerd-users/0bd745bb-4f62-455b-a5d6-e42218c0e577%40googlegroups.com.

Betson Thomas

unread,
May 16, 2017, 12:58:52 AM5/16/17
to linkerd-users
Got it, thanks Kevin

-Betson
Reply all
Reply to author
Forward
0 new messages