logging in Kubernetes: introducing contextual logging

[Patrick Ohly]

Hello all!

You may have heard about the on-going effort to improve logging in
Kubernetes by switching log entries from printf-style strings to a
format that is easier to parse automatically ("structured logging",
https://kubernetes.io/blog/2020/09/04/kubernetes-1-19-introducing-structured-logs/).

Right now, logging is always done though the klog package and a global
logger instance defined there. There was a discussion around allowing
arbitrary logging implementations already when the structured logging
KEP was initially proposed. That aspect just got taken out of that KEP
to keep it simpler.

Now would be a good time to address that gap because if we can come to
an agreement on how to do it, further code conversions as part of the
structured logging migration can also directly remove the hard klog
dependency, thus killing two birds with one stone, eh, pull request,
which will reduce work for the Kubernetes maintainers.

Therefore I have collected use cases and prepared a KEP for "contextual
logging". In a nutshell, contextual logging gives the caller of a
function full control over all logging inside that function. By
leveraging features from go-logr (https://github.com/go-logr/logr#readme),
it becomes possible to attach additional values and a prefix to each log
entry and to control verbosity on a per-call-chain basis. When running
unit tests in parallel, the log entries of different test cases don't
get mixed up and can be printed by "go test" for failed tests (and only
for those).

We already discussed this in the Structured Logging WG and in SIG
Instrumentation. SIG Scheduling is also supporting this proposal because
it addresses several of the shortcomings that they have identified in
the current kube-scheduler log output.

But like structured logging, this will affect the entire Kubernetes code
base and associated projects. We want to raise awareness early and
solicit feedback from everyone before moving forward with the
KEP. Please have a look and comment on
https://github.com/kubernetes/enhancements/pull/3078.

The "alternatives" section discusses why other options like configuring
a logger per component don't satisfy the use cases and provides further
explanations for some design decisions. Under "code examples" are a link
to a prototype implementation and some examples from that prototype.

--
Best Regards

Patrick Ohly
Cloud Software Architect