SIG-Scalability PSA: Golang 1.8 performance regression.

[Tim St. Clair]

==== PSA ====

We ran out of time on the community meeting, but we wanted to make
certain folks are aware that there is a golang 1.8 performance
regression:

https://github.com/kubernetes/kubernetes/issues/45216

We should have feedback from the golang team if they will have a patch
release by next week. Current upstream builds are using a patched
version of golang 1.8.

Lastly, Wojtek will send out a follow on summary in the coming day(s).

==== END PSA ====
--
Cheers,
Timothy St. Clair

[Wojciech Tyczynski]

 Hi,

  To elaborate more on the regression Tim mentioned above:

1. The regression in in the http2 server implementation and it manifests generally only

when there are a lot of open connections to it. Our experiments and observations show

that this starts being a problem when the cluster has few hundreds of node.

if you are more interested in exact details of the regression, you can take a look into:

https://github.com/kubernetes/kubernetes/issues/45216

In particular, apiserver in 5000-node cluster is using ~3x more CPU, so the difference in

large scale is enormous.

We can't revert back to 1.7.5 version, because this would block some of the important

efforts, e.g. cert rotation.

2. Fortunately, we have a very small patch that mitigates the regressions.

3. We are still waiting for the decision from Go team, whether they will create Go patch

release containing this fix (if not, they will solve it in 1.9 release).

4. In the meantime, all the Kubernetes builds we are doing now are using a patched

version of Go - see https://github.com/kubernetes/kubernetes/pull/45879 for how exactly

we did it.

5. Depending on whether Go patch release will be build, we will:

- bump to new patch release of Go if it will appear

- stick to patched version of Go (and then 1.7 release will be built with that).

 If you are building Kubernetes yourself, you should consider patching Go version

you are using too.

 Thanks

wojtek

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CALM%2Bqp9UA3GLkmQtO5CFd6eVihWJUau%3DXRqikexCSka%3DT4bo7w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Wojciech Tyczynski]

 Hi,

  We have just upgraded Kubernetes head to be built using Go 1.8.3 and got rid of the patch.

This has solved our performance issues.

 Thanks

wojtek

On Fri, May 19, 2017 at 6:38 PM, Brad Fitzpatrick <brad...@google.com> wrote:

We'll put the workaround performance fix in Go 1.8.3 soon. Go 1.8.2 will be a security-only release released probably the same day (or a day before) Go 1.8.3.

(The workaround fix is burning an extra goroutine and its 2KB per connection to relay the channel notification. We have a better fix without the goroutine for Go 1.9)

To post to this group, send email to kuberne...@googlegroups.com.