Hi folks,We've been working hard to reduce the performance impact from serverside apply so we beta / turn on by default:* wrapped the fieldset (i.e. in something like a RawExtension) to make decoding it optional* ~90% speedup (10x) in writing/reading the fieldsets* collapsed writes by same users into the same entries (i.e., not split by time)* ~50% speedup (2x) in the update & apply operationsWe're still working to get final numbers with all these fixes in place at the same time. However, it's currently looking like > 75% of the latency increase is simply due to the increased size of the objects and not due to additional code (we measured controlled for size by making big annotations).
So, we need to make some plans:
1. What is our latency budget? We need to know this ASAP.
If we are not meeting the target once all our optimizations are in, then we have some options for reducing the object size, which will be required for making additional significant improvement:2. How to balance size with API usability?
2a. gzip existing fieldset and call it a day.-> Not a great API experience for users.-> We can add a prefix byte to permit versioning.-> If we go to beta, we'd have to lize with this format forever.
2b. Invent a more concise format. This will be somewhat difficult while leaving it readable.-> Same re: living with this forever & versioning.-> I've got an improvement here, probably the best possible while maintaining JSON structure
2c. Hybrid approach: gzip or other binary format unless user requests it in pretty form (via header or something)-> No precedent in the API-> Possible format: omit it unless user asks for it-> Whatever you get from GET must go back to PUT without adverse consequences
--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3bauUYysXs%3DDFHy2Jr-K1J%2B%2Bdyp%3D%2Bjf8bkSntAdPUZJG-A%40mail.gmail.com.
On Tue, Aug 20, 2019 at 8:33 PM 'Daniel Smith' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com> wrote:Hi folks,We've been working hard to reduce the performance impact from serverside apply so we beta / turn on by default:* wrapped the fieldset (i.e. in something like a RawExtension) to make decoding it optional* ~90% speedup (10x) in writing/reading the fieldsets* collapsed writes by same users into the same entries (i.e., not split by time)* ~50% speedup (2x) in the update & apply operationsWe're still working to get final numbers with all these fixes in place at the same time. However, it's currently looking like > 75% of the latency increase is simply due to the increased size of the objects and not due to additional code (we measured controlled for size by making big annotations).That's cool!. And that's actually matching what we're observing (and it with couple upcoming improvements like this it should be even better).So I'm not too worried about that processing part - the thing that seem to be the most limiting now is etcd + what is below it (kernel, storage, ...).We're currently (literally as we speak, though it's really tricky so it will take some time) in the process of investigating what exactly is the most limiting factor.Another interesting point is that we were investigating significant differences between kubemark and gce testsin latencies. And apparently the biggest difference was in size of Node object (in kubemark it doesn'tcontain any images) - and by adding a single annotation with 3KB of random data (to compensate for differentobject size), is actually bringing the result to something much more similar.So, we need to make some plans:1. What is our latency budget? We need to know this ASAP.It's a very tricky question - I'm not sure we can actually give a very satisfying answer for that.
We have a lot of slack on low percentiles (e.g. 50th percentile is really low for api calls).On high percentiles (99th, which is the most interesting from SLO POV) we're actually not that far.With the old implementation (99th percentile over test) we have quite a significant slack (like ~40-50% for the worst calls),but in the new implementation that we're trying to switch (which is aggregating how the SLO is actually saying into 5min windows),we're really on the fence (in fact we're frequently not meeting it for some resources (like PUT leases)).The other aspect is that even if we would have XX% of slack, how much can actually offer for this feature is not obvious.Recently we switched on mTLS between etcd and apiserver and that also has eaten some non-negiligble part of the slack.And I bet there will be more such features in the future that are pretty important and will require sacrificing some latency.I would probably feel safe if you increases latencies by 50ms, but it's really hard to say if it's really max we can afford.Also see my comments below.If we are not meeting the target once all our optimizations are in, then we have some options for reducing the object size, which will be required for making additional significant improvement:2. How to balance size with API usability?What I'm most worried about is that the size of object with this change is increasing dramatically -- by just a bit less than 2x - IIRC it was something like 70%, please correct me if I'm wrong.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3bY2f3RuYTEw4YfJVFs7Q2C_nWKM_3d-vpjj%2B%2BHLN4gxRg%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3bZR%2BgYeot%3DfE7N-%2B%3DhsuScPzv9vP0WufF3cx5ArvR%3DqOQ%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAFS1MjK2zNrPi5Dxr5wOFU%3DRjUQMURq1bHL3hPC%2BBBTTrOHgVQ%40mail.gmail.com.
I agree we should be able to turn it on for everything before GA.I've thought of another way to reduce the performance impact while still letting users gain experience (we've been working for too long without user reports): turn it on for an object only once APPLY has been used for the object.Today, our format gzip'd is still a ~30% size increase on a proto object, which has an as-yet-unquantified impact on latency. We can likely do a bit better than gzip if we custom write something. As a working assumption, I think we should assume that this will be at least a 20% size increase.Will we be able to afford that?
I think that turning on only after using apply once will result in losing information about which fields have been set by which actors in the past. I think a timeline like this could work:
- releaseA - beta1 on by default for most kube-apiserver resources, but skipped for resources of your choice
- releaseB - beta2 on by default for all kube-apiserver resources, no exceptions
- releaseC - GA, on for all kube-apiserver, no exceptions
My only concern is then a state of perma-beta if we get stuck between beta1 and beta2. Adding Jordan specifically since he's been dealing with that this year.
Making the "kubectl apply" flow work well is the primary purpose of Server-Side Apply.
For beta1, will all resources (including pods, nodes, and endpoints), have apply enabled? You have " The apply verb is enabled everywhere (well, for builtins and CRs) in all betas, we don't need to adjust the discovery docs specially." under "for everything", but you also explicitly say under "beta3 and/or GA" that "We'll default on for all resources".
--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAFS1Mj%2B1guSUGQOWfDxmf4Bt%3DmFHPOo%2BXK4iNv7DGDCh2xn1oA%40mail.gmail.com.