Using go-kit with linkerd or envoy

[Nathan Murthy]

Hi folks,

I like that go-kit puts good thought into layering transport and instrumentation around core business logic. With that in mind, is anyone using go-kit with transparent proxies like linkerd or envoy? For those who are, what best practices have you adopted for reconciling things like distributed tracing, service metrics, circuit breaking, context propagation, etc. in go-kit with similar features offered by linkerd and envoy?

Kind regards,

Nathan

[Yuri Shkuro]

Hi Nathan,

You may want to treat some of the benefits provided by service mesh proxies as complimentary to what your own service is doing, not a replacement. For example, take distributed tracing and context propagation: despite the claims otherwise, the service meshes do not actually solve either of those unless your application, at minimum, has some form of in-process context propagation to preserve the tracing headers. Enabling additional instrumentation in your app does not conflict with monitoring provided by the proxies, but allows you to do more things, like trace or monitor internal parts of the application.

Circuit breaking is a more complex feature which can benefit from external help, since generally you cannot do it based on the information present in a single instance of client or server.

--Yuri

[Nathan Murthy]

Hi Yuri,

In what ways do you draw that complementarity within your software org? For instance, I can't force every developer across the whole company to implement in-process context propagation in each of our apps, but it's much easier to standardize on common deployment infrastructure that comes already equipped with some kind of tracing instrumentation. Curious to hear more of your thoughts.

- Nathan

[Yuri Shkuro]

Context propagation doesn't come for free, but can be easier in some language than in others. Some commercial vendors offer monkey-patching solutions (java-agent approach is among them), but it won't work in Go. The service mesh proxies certainly don't help there; Envoy docs explicitly say "your application must forward some headers", which is just another form of in-process context propagation. I recently gave a talk about this at Monitorama https://vimeo.com/221070602, including how we approached it at Uber, where we have 4+ programming languages in active use.

[Peter Bourgon]

You describe necessary complexity — you do in fact have to require (or
perhaps better said: encourage) all developers to do certain things in
their services. But this is no different (in kind) than other types of
things you require of services: that they respond to SIGINT, that they
don't leak memory, that they log according to some accepted format,
that they expose a certain core set of metrics — and that they accept,
amend, and forward context propagation headers correctly. Yuri's talk
describes some good strategies at an organization level. More
concretely, a lot of these things are made easier by creating and
maintaining per-language "companykit" repos that provide the necessary
scaffolding. Go kit is explicitly designed to work well in that
environment.

> --
> You received this message because you are subscribed to the Google Groups
> "Go kit" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-kit+un...@googlegroups.com.
> To post to this group, send email to go-...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-kit/bdfd6e2d-f8ea-4591-99ad-6ed517c9bc4a%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Nathan Murthy]

Yuri, thanks. Really great talk. I'll share it with some folks at work. That's pretty cool you guys have a Distributed Tracing team. I'm also going to have a thorough look at Jaeger. 

Peter, very good points. The complexity largely becomes organizational rather than technical.