Proposal: Use Signal to dynamically modify log levels or enable pprof

[Aimuz]

A premature proposal

Switching Log Levels via Signal

kill -SIGUSR1 pid

Each time it is executed, the log level will be adjusted to one level up to the maximum value, and then back to the minimum value

kill -SIGUSR2 pid

Each execution will toggle between pprof enabled and pprof disabled states. Or pprof will be written to a file.

This proposal may be helpful in troubleshooting problems, avoiding program restarts and maximizing the availability of information on site.

[Aimuz]

kill -s SIGUSR1 / kill -s SIGUSR2

https://www.gnu.org/software/libc/manual/html_node/Miscellaneous-Signals.html

[Aldo Culquicondor]

Are you targeting a specific binary or do you want to target all k8s binaries?

Note that kube-scheduler already responds to SIGUSR2 to print a dump of its cache into the log.

We have an HTTP endpoint for all binaries that you can use to grab a pprof (kube-scheduler code). It's not expensive to keep that endpoint up all the time.

The ability to increase the log level at runtime sounds desirable to me.

Aldo

--
You received this message because you are subscribed to the Google Groups "dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@kubernetes.io.
To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/ced8a41d-443e-4e29-96c3-421486c57dc1n%40kubernetes.io.

[Bowei Du]

Use of signals is usually not preferred vs a standard HTTP flag changing endpoint:

- Does not allow for passing additional information or extension/change later on.

- May already be used for other purposes.

Thanks,

Bowei

To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAJi5r2p4HK9_8-CT8U7ip21pJz%2BTYNFGcSqx-dJsd_WgS_%2BOOw%40mail.gmail.com.

[Antoine Pelisse]

Aimuz, why don't you keep the http profile endpoint open all the time?

If you think that might be a security concern, then you might be interested in that change: 

https://github.com/kubernetes/kubernetes/pull/114191

Thanks,

Antoine

To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAGv8vBpJpm5g1Yx_6hgiA9itaQ51rb6S6bH8FYP6ASavboF2sw%40mail.gmail.com.

[Aimuz]

I'd like to add to all the binaries and then leave the feature on without changing the configuration, so that I can avoid problems that can't be tracked after a reboot. If you can get pprof to turn on by default is one way

Rebooting would solve 90% of the problem.

So does adjusting the log format.

As for security, I didn't consider it.

If kube-scheduler already uses SIGUSR2, can we add SIGUSR1 to all binaries to adjust the log output level? He would be really useful.

Thanks!
Aimuz

[Antoine Pelisse]

I think you can adjust the log-level through the debug endpoint too (when profiling is enabled)

https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/config.go#L915

I haven't tried that though.