Go “service container” thoughts

[Zellyn]

Hi folks,

At Square, we have a collection of shared frameworks and library code, loosely referred to as the “service container” in each language. For example, our framework code in Go includes:

• a lightweight "module" system for dependency injection
• config: a way of layering yaml files named for environments. For instance, common.yaml gets loaded first, and production.yaml shadows over it, and production-east-coast.yaml gets applied last.
• logging: code for sending log lines over kafka to our logging aggregator
• networking: our stubby-alike protobuf RPC mechanism, naming and discovery, etc. Gradually migrating to "just use grpc and envoy"
• datasource config: a convention for configuring database sources via the yaml configs

People probably also think of our service container as including slightly less core things too:

• sharded jobqueues
• feature flags (backed by zookeeper)
• cron jobs (again configurable via yaml)

This stuff has gone a while with less-than-generous attention and upkeep, and the extent of Square-specific code makes it more difficult for new developers to come up to speed.

What I'm trying to figure out is: what are folks at other companies using to create a shared set of conventions and out-of-the-box setup for their Go applications? We can knock some pieces out one by one (eg. replace our Module system with Wire), but it would be nice to find an existing lightweight “service container” that other companies are using.

I suppose my main question is this: is the kind of flexibility required for a shared, open-source “service container” antithetical to the kind of simple, opinionated, internal-convention-obeying “service container” you'd want for your company? To put it another way, it has never felt like using gokit would be appropriate, because it obeys none of our conventions.

Thoughts, comments, experience to relate?

Yours,

Zellyn

[Peter Bourgon]

I've repeatedly built "service container" or "corp kits" like the one
you're describing, as well as helped a number of orgs improve their
own versions of the thing in a consultant capacity. I think everyone
has the same reaction that you do: why are we duplicating all this
effort? But I can say with some breadth of experience that the
commonalities are fewer than they may appear. Once you take your e.g.
log shipper library, which is likely quite purpose-built not only for
your chosen MQ but the peculiarities of its runtime environment(s),
and provided switches and knobs to account for reasonable alternatives
to make it worth open-sourcing, you've rounded off a sufficient number
of corners to make it significantly less useful in your original
environment. The utility of these kits or containers is generally
proportional to how zero-conf they are: just use these pre-ordained
things, and you'll be good to go. Provide enough knobs to make them
generally useful, and the magic is replaced by config; new users,
faced with a sprawling YAML config and compiling in support for things
they'll never need, wonder why they can't eliminate all this cruft by
making a purpose-built jig for their specific conditions, and you're
back to square one. Advice from someone who's been down the path
enough times: accept as invariant that this code will never see the
light of open-source day, and hyper-specialize to your programming
environment, optimizing for ease-of-use. It's a better use of time and
energy.

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Zellyn]

Peter, thanks for that thoughtful and clear answer. It articulates very clearly the tussle I've been feeling, and is immensely helpful!

Happy New Year!

Zellyn