metrics name changes

[Jordan Liggitt]

There's a KEP and PR improving metrics reporting, and some of the improvements involve renaming existing metrics.

There was discussion about impact to existing consumers and efforts to leave existing metrics in place for a deprecation period, which seems good, but I wasn't sure where metrics fell under the deprecation policy.

Are metrics an API? Are there currently any guarantees around them? https://github.com/kubernetes/kubernetes/pull/74418#discussion_r259713158 indicated they are not considered stable currently, but I wasn't sure if that was just for the metrics being modified, or for metrics in general.

[Pat Christopher]

metrics in general are not considered stable and there are no guarantees around them.  

it is an on-going concern for sig-instrumentation to try and provide a set of core metrics that we can offer guarantees around.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-instrumentation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instru...@googlegroups.com.
To post to this group, send email to kubernetes-sig-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-instrumentation/CAMBP-pL7qPYSS6DTGhTKkoRrdHVGahAngygxGsjNOQ0Gk1TUGw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Frederic Branczyk]

Typically there is no deprecation needed, the only reason we thought it would be good for this effort is because it’s such a large scale change. Instead of a few metrics changing as they’ve done in most releases, we’re changing a very significant amount of metrics exposed by Kubernetes.

Metrics as a whole will never have a single stability guarantee as many metrics are specific to their surrounding code, those metrics should always be allowed to change, just as their surrounding code changes. Once this KEP is finished, we want to start the conversation of which metrics _could_ have a stability guarantee and in detail discuss what that would mean and what it would look like. In my mind this would be a very carefully selected list of metrics of things that will ever change such as API and scheduling latency metrics (just examples let’s have the actual discussion about this then).

[Brian Grant]

Fixing the SIG Instrumentation email

On Mon, Mar 4, 2019 at 8:23 PM Brian Grant <brian...@google.com> wrote:

+Vishnu Kannan +John Belamaric 

I completely agree that metrics need significant improvement, and would like to incorporate metrics into the maturity (alpha/beta/GA) criteria.

Currently I think metrics fall into the "other" category, which has a 1-year deprecation period:

https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-a-feature-or-behavior

Information/configuration intended for direct human admin consumption/control (e.g., contents of events and logs) has fewer guarantees than behaviors people are expected to build tooling around. I think metrics fall into that latter category. 

However, one challenge is that monitoring systems covering multiple clusters, especially large numbers of clusters, need to collect metrics from clusters of multiple versions, probably all supported versions. Hence, a deprecation period shorter than the support period is problematic.

Whatever we decide, for clarity, we should add metrics, logs, and Event contents to the deprecation policy.

On Fri, Mar 1, 2019 at 9:45 AM Clayton Coleman <ccol...@redhat.com> wrote:

I'm ok if we think we want more than one release - here's the rules I've personally been involved in enforcing along with Frederic in the PRs he's copied me on (but there may be others)

1. New metrics are added before old metrics are removed for at least 1 release

2. Old metrics are marked deprecated in their metrics description at the same time as new metrics are added

3. Metrics that cause significant performance problems can be removed without 1 release notice, but they do require a release note and a justification, and that justification is either significant performance regression at scale, unbounded cardinality, or issues that can't be addressed by filtering at the receiver end

We should however add these rules to the API doc - are there any volunteers from sig-instrumentation to open the PR?  We can then discuss in that and have a record.

On Fri, Mar 1, 2019 at 12:36 PM Jordan Liggitt <lig...@google.com> wrote:

Great, thanks for confirming.

On Fri, Mar 1, 2019 at 12:35 PM Clayton Coleman <ccol...@redhat.com> wrote:

Our stance historically has been one release, and we have summarily removed metrics that cause problems without that notice.  We have treated cadvisor metrics more like an API, but still not with the same guarantees as our core APIs.

On Fri, Mar 1, 2019 at 12:34 PM Clayton Coleman <ccol...@redhat.com> wrote:

Which was the guidance we gave to the instrumentation team already and how they've been executing.

On Fri, Mar 1, 2019 at 12:23 PM 'Daniel Smith' via kubernetes-api-reviewers <kubernetes-a...@googlegroups.com> wrote:

Even in the absence of given guarantees, for metrics that are actually currently useful I think--if we want to think of ourselves as good people--one release of deprecation period is mandatory (otherwise how can a rollout be possible without breaking monitoring?) and two would be nice.

On Fri, Mar 1, 2019 at 7:34 AM <qsj.d...@gmail.com> wrote:

+1

Definitely, we need a guarantees for metrics, it's a long-term goal.

This KEP is one MUST step to this goal. Only when metrics naming follow guideline and best practices, we can move on to the next step.

--
You received this message because you are subscribed to the Google Groups "kubernetes-api-reviewers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-api-rev...@googlegroups.com.
To post to this group, send email to kubernetes-a...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-api-reviewers/5d953c15-8b47-4059-a319-07d01c1cb7e8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-api-reviewers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-api-rev...@googlegroups.com.
To post to this group, send email to kubernetes-a...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-api-reviewers/CAB_J3bayFio%3D_%2Bi_P2oyK%3DHfcqEhhXKNy6Hp61uSMuz27NThNQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-api-reviewers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-api-rev...@googlegroups.com.
To post to this group, send email to kubernetes-a...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-api-reviewers/CAH16ShLOWPhQUbq%2B1hZY4x9J8u%3DYvM8tJ%2B2ZmWeP0SYNcbzkYQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-api-reviewers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-api-rev...@googlegroups.com.
To post to this group, send email to kubernetes-a...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-api-reviewers/CAH16ShLYFn4BrzAEjzy5JDyDAgwON%3DJUiGfiH9%3DjrZTbN5zg7g%40mail.gmail.com.

[Tim Hockin]

I agree.  Is it a practical problem to keep old and new?

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-api-reviewers/CAKCBhs7JrHPPXGxvmxVTShE5UqhnqLp57e3h%2Bi1DEVq8YGmFPA%40mail.gmail.com.

[Brian Grant]

On Mon, Mar 4, 2019 at 8:33 PM Tim Hockin <tho...@google.com> wrote:

I agree.  Is it a practical problem to keep old and new?

Depends on whether it's just a name change or differently computed information.

[Vishnu Kannan]

+1 on Brian's point on needing a year long depreciation period (as much as possible) to ensure we give users adequate time to test & identify metric changes and switch to new metrics. 

I assume the deprecation policy for alpha and beta features will apply to their corresponding metrics as well?

We also need to provide a way to test and ensure newly introduced metrics that are not related to new features are stable in production prior to requiring a one year depreciation period. That is, we need graduation phases (alpha, beta & GA) for metrics as well and policies around how that will be communicated in a standard way.

I agree events should probably follow the same policy. For logs, I'd recommend having a relaxed policy around the severity level since I have seen in practice that our logs are very spammy and severity levels need some adjustment. It is possible for users to have consumed logs to generate metrics and so we should consider log text similar to metrics APIs.

On Mon, Mar 4, 2019, 8:57 PM Brian Grant <brian...@google.com> wrote:

On Mon, Mar 4, 2019 at 8:33 PM Tim Hockin <tho...@google.com> wrote:

I agree.  Is it a practical problem to keep old and new?

Depends on whether it's just a name change or differently computed information.

 

On Mon, Mar 4, 2019, 8:25 PM 'Brian Grant' via kubernetes-api-reviewers <kubernetes-a...@googlegroups.com> wrote:

Fixing the SIG Instrumentation email

On Mon, Mar 4, 2019 at 8:23 PM Brian Grant <brian...@google.com> wrote:

+Vishnu Kannan +John Belamaric 

I completely agree that metrics need significant improvement, and would like to incorporate metrics into the maturity (alpha/beta/GA) criteria.

Currently I think metrics fall into the "other" category, which has a 1-year deprecation period:

https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-a-feature-or-behavior

Information/configuration intended for direct human admin consumption/control (e.g., contents of events and logs) has fewer guarantees than behaviors people are expected to build tooling around. I think metrics fall into that latter category. 

However, one challenge is that monitoring systems covering multiple clusters, especially large numbers of clusters, need to collect metrics from clusters of multiple versions, probably all supported versions. Hence, a deprecation period shorter than the support period is problematic.

Even with a deprecation period, assuming these systems continue to support newer versions, they will have to deal with metric changes right either via rewriting metrics in their monitoring backend (for renames) or adding version specific policies to dashboards, alerts and SLOs right?

[Frederic Branczyk]

We recently had this discussion on the sig-instrumentation mailing list I believe originating from the same source. Generally I'm all for having more stability around metrics, but I think we can only do this by hand picking those metrics. Many metrics currently instrumented in Kubernetes are specific to their surrounding code as the metrics describe what this code is doing and how it is performing, those metrics should be allowed and encouraged to be changed as the surrounding code changes. That said, there are a number of metrics that I feel are eligible for stability agreements, like API requests/error count and latency, scheduling latency, essentially anything that people are likely to build their SLAs around.

Due to the above I don't think we can have a blanket statement for metrics stability. The KEP is meant as a one time cleanup, to start a serious discussion about possibly creating guarantees around some metrics. Given that in the past we have not cared about metrics stability guarantees at all (we have broken metrics many times without any deprecation), I think the KEP is already handling this more appropriate then we did previously. I'm not generally opposed to leaving both in place for 1 year, I just wonder why start now, and not after we've actually established guarantees around metrics, and which.

>> Is it a practical problem to keep old and new?

> Depends on whether it's just a name change or differently computed information.

Correct, although I can just speak for Prometheus, where no it's not a problem. Dropping metrics at ingestion time is a commonly used Prometheus feature and is a good fit here. The only implication keeping both has on Prometheus users is that there will be unnecessary bytes transferred in the deprecation period.

You received this message because you are subscribed to a topic in the Google Groups "kubernetes-sig-instrumentation" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-sig-instrumentation/XbElxDtww0Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-sig-instru...@googlegroups.com.

To post to this group, send email to kubernetes-sig-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-instrumentation/CAA_vbqRv7jRVpGno%2BotFm8cC5WpFaC2hU%3D6mXwLaDM%2B8LFrXvg%40mail.gmail.com.

[Clayton Coleman]

Put another way, we have a lot of metrics that are like unit tests.  They are not conformance tests, and if the code they measure is inaccurate or changes or if the consumer of those metrics is primarily the engineers who own that component they probably shouldn't be under the same API guarantees as our long term tests.

[Tim Hockin]

On Tue, Mar 5, 2019 at 8:45 AM Frederic Branczyk <fbra...@gmail.com> wrote:
>
> We recently had this discussion on the sig-instrumentation mailing list I believe originating from the same source. Generally I'm all for having more stability around metrics, but I think we can only do this by hand picking those metrics. Many metrics currently instrumented in Kubernetes are specific to their surrounding code as the metrics describe what this code is doing and how it is performing, those metrics should be allowed and encouraged to be changed as the surrounding code changes. That said, there are a number of metrics that I feel are eligible for stability agreements, like API requests/error count and latency, scheduling latency, essentially anything that people are likely to build their SLAs around.

Perhaps we should be phrasing metrics more genetically, then? Or do
we have any way to denote which metrics are of which variety
(supported vs implementaion-coupled)

[Daniel Smith]

I think log messages should not be under any sort of deprecation period. We should not have to wait a year or even a release to fix a log message. I think people scanning log messages for monitoring should know that isn't a good thing to do and hence already understand that we don't make promises. IMO we are really not doing a good job on the metrics front if there's more than a few people finding that necessary.

I also don't know if I buy the "monitor multiple installations simultaneously" argument--does monitoring software not permit metric names to be changed?

I think a metric deprecation policy needs to be explicit that the metric deprecation period doesn't block code refactoring / improvements. e.g., if we had a hot loop that we published latency for, and find a way to not need the loop any more, I think it is actually reasonable for the policy to mandate either a) just remove the metric completely without warning or b) keep the metric for <time period>, but set it to 0 on start up and never touch it afterwards. It would not be reasonable to require something like generating a value that looks roughly the same as prior versions, even though that's the only thing that could 100% protect users from actually needed to make a monitoring configuration change.

[Frederic Branczyk]

> Perhaps we should be phrasing metrics more genetically, then?  Or do

> we have any way to denote which metrics are of which variety
> (supported vs implementaion-coupled)

We haven't made a decision how this would work. Two solutions that have been brought up so far:

1) Have a separate metrics endpoint that describes the stability. For example "/metrics/v1".

2) Just document which metrics have an explicit stability agreement.

I personally prefer 2, because having multiple endpoints to scrape on kubelets means we're multiplying kubelet count by number of endpoints we add, which in low numbers is fine, but the higher node count we go the worse this gets (and we already have 3 metrics endpoints in 1.13 on kubelets, admittedly 1 was removed in 1.14, for KEP unrelated reasons).

> keep the metric for <time period>, but set it to 0 on start up and never touch it afterwards

cAdvisor has tried this, and it only causes more confusion, when metrics are there but set to 0 and you don't understand why, it seems like a bug (this is a great example that has confused myself greatly until finding the issue https://github.com/google/cadvisor/issues/1925)

> It would not be reasonable to require something like generating a value that looks roughly the same as prior versions, even though that's the only thing that could 100% protect users from actually needed to make a monitoring configuration change.

I think as an operator of clusters I prefer a hard change in my monitoring rather than a subtle one that makes me think everything is ok, when it's not.

---

I feel we're all pretty much agreeing. Is there a concrete ask in the scope of this KEP? Otherwise I'd say we have the discussion around metrics stability as planned, when the metrics overhaul KEP is finished, targeting the 1.15 release.

[John Belamaric]

I think it makes sense of the component authors to identify a subset of metrics that are considered stable, and maybe mark them in the help text similar to how the KEP is doing it for deprecated metrics. This gives the authors more control over the ones they want to support. Since metrics may be transitively included from dependencies, complying with the deprecation policy could be really challenging. For example, the prometheus go instrumentation library exports a bunch of golang metrics. I wouldn't want to be forced into keeping an older version of that library because some metric changes in there.

Informational logs should not be part of the API surface. I could see an argument for errors and maybe warnings having some stability guarantees, with the caveat that if it's no longer relevant it will just go away. Even with errors, I am not sure there is value unless the log has some structure that can be relied upon.

[Vishnu Kannan]

On Tue, Mar 5, 2019 at 10:08 AM John Belamaric <jbela...@google.com> wrote:

I think it makes sense of the component authors to identify a subset of metrics that are considered stable, and maybe mark them in the help text similar to how the KEP is doing it for deprecated metrics. This gives the authors more control over the ones they want to support. Since metrics may be transitively included from dependencies, complying with the deprecation policy could be really challenging. For example, the prometheus go instrumentation library exports a bunch of golang metrics. I wouldn't want to be forced into keeping an older version of that library because some metric changes in there.

Informational logs should not be part of the API surface. I could see an argument for errors and maybe warnings having some stability guarantees, with the caveat that if it's no longer relevant it will just go away. Even with errors, I am not sure there is value unless the log has some structure that can be relied upon.

It would be useful to collect some data from current users on the expectations around logs including using it as a source for metrics (which is a common well established industry practice AFAIK).

  

On Tue, Mar 5, 2019 at 9:20 AM Frederic Branczyk <fbra...@gmail.com> wrote:

> Perhaps we should be phrasing metrics more genetically, then?  Or do

> we have any way to denote which metrics are of which variety
> (supported vs implementaion-coupled)

We haven't made a decision how this would work. Two solutions that have been brought up so far:

1) Have a separate metrics endpoint that describes the stability. For example "/metrics/v1".

2) Just document which metrics have an explicit stability agreement.

I personally prefer 2, because having multiple endpoints to scrape on kubelets means we're multiplying kubelet count by number of endpoints we add, which in low numbers is fine, but the higher node count we go the worse this gets (and we already have 3 metrics endpoints in 1.13 on kubelets, admittedly 1 was removed in 1.14, for KEP unrelated reasons).

+1 on not preferring `1` coz it also complicates integrations with other monitoring systems. 

Can the stability guarantees and versioning (if necessary) become metric labels (ideally first class fields in the prom exposition format) instead to make it easier to discover and filter non-stable ones from SLOs?

> keep the metric for <time period>, but set it to 0 on start up and never touch it afterwards

cAdvisor has tried this, and it only causes more confusion, when metrics are there but set to 0 and you don't understand why, it seems like a bug (this is a great example that has confused myself greatly until finding the issue https://github.com/google/cadvisor/issues/1925)

> It would not be reasonable to require something like generating a value that looks roughly the same as prior versions, even though that's the only thing that could 100% protect users from actually needed to make a monitoring configuration change.

I think as an operator of clusters I prefer a hard change in my monitoring rather than a subtle one that makes me think everything is ok, when it's not.

---

I feel we're all pretty much agreeing. Is there a concrete ask in the scope of this KEP? Otherwise I'd say we have the discussion around metrics stability as planned, when the metrics overhaul KEP is finished, targeting the 1.15 release.

Assuming no metrics are being removed in v1.14, IIUC, the concrete ask is to finalize the support policy around metrics prior to removing existing ones. 

[Han Kang]

I'm all in favor of stable APIs and non-breaking monitoring changes but as possibly low-ish hanging fruit, I would personally love it if each release with some notes saying which metrics are included with the corresponding binary. 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-instrumentation/CAA_vbqQRDnQVuTyZnavaKrwrZN2MdwJ0CjJGRNjNLa%2Bii14B%3DQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

- Han

[Daniel Smith]

Did this email thread reach closure? It doesn't appear so, I read Brian & Tim as advocating for longer deprecation periods than some others.

This PR removes many metrics which have been deprecated for 1 release: https://github.com/kubernetes/kubernetes/pull/76496

In my earlier emails in this thread I claim that 1 release deprecation should be the minimum, but now that I see the metrics listed out I'm a little nervous about removing them all, and I'm no longer sure that 1 release is a sufficient amount of warning.

[John Belamaric]

Forcing users to adapt in a one release timeframe does seem aggressive. Also, if you are monitoring a fleet of many clusters, a single release deprecation would mean you are likely to have a lot of clusters that run a version that does not have the new metrics, along with those that have both and just the new ones. This complicates monitoring. If we follow the 1 year policy, then any clusters without the new metrics will have fallen out of support before we remove those old ones.

What's the rush?

[Mike Spreitzer]

I also think one release is rather agressive.  AFAIK it is pretty standard in Kube, but given the frequency of Kube releases it is aggressive.  For a point of comparison, in IBM software products the general rule is backward compatibility for two major releases.  And those happen less frequently than Kube releases.  And I know some shops do not pick up every single Kube release.  I think 1 year is much more reasonable.

Regards,
Mike

"'John Belamaric' via kubernetes-sig-instrumentation" <kubernetes-sig-...@googlegroups.com> wrote on 05/30/2019 12:09:37 PM:

> From: "'John Belamaric' via kubernetes-sig-instrumentation"
> <kubernetes-sig-...@googlegroups.com>
> To: Daniel Smith <dbs...@google.com>
> Cc: Han Kang <han...@google.com>, Vishnu Kannan
> <vis...@google.com>, Frederic Branczyk <fbra...@gmail.com>, Tim
> Hockin <tho...@google.com>, Brian Grant <brian...@google.com>,
> Clayton Coleman <ccol...@redhat.com>, Jordan Liggitt
> <lig...@google.com>, Daniel Qian <qsj.d...@gmail.com>,
> kubernetes-api-reviewers <kubernetes-api-
> revi...@googlegroups.com>, kubernetes-sig-instrumentation
> <kubernetes-sig-...@googlegroups.com>
> Date: 05/30/2019 12:10 PM
> Subject: [EXTERNAL] Re: metrics name changes

> keeping an older version of that library because some metric changesin there.

> instrum...@googlegroups.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> There's a KEP and PR improving metrics reporting, and
> some of the improvements involve renaming existing metrics.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> There was discussion about impact to existing
> consumers and efforts to leave existing metrics in place for a
> deprecation period, which seems good, but I wasn't sure where
> metrics fell under the deprecation policy.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Are metrics an API? Are there currently any
> guarantees around them?

https://github.com/kubernetes/kubernetes/

> pull/74418#discussion_r259713158 indicated they are not considered
> stable currently, but I wasn't sure if that was just for the metrics
> being modified, or for metrics in general.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> You received this message because you are subscribed
> to the Google Groups "kubernetes-sig-instrumentation" group.
> >>>>>>>>>>>>>> To unsubscribe from this group and stop receiving

> emails from it, send an email to kubernetes-sig-instrumentation
> +unsub...@googlegroups.com.

> >>>>>>>>>>>>>> To post to this group, send email to kubernetes-sig-

> instrum...@googlegroups.com.

> >>>>>>>>>>>>>> To view this discussion on the web visit https://
> groups.google.com/d/msgid/kubernetes-sig-instrumentation/CAMBP-
> pL7qPYSS6DTGhTKkoRrdHVGahAngygxGsjNOQ0Gk1TUGw%40mail.gmail.com.
> >>>>>>>>>>>>>> For more options, visit

https://groups.google.com/d/optout.

> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> You received this message because you are subscribed to
> the Google Groups "kubernetes-api-reviewers" group.
> >>>>>>>>>>>> To unsubscribe from this group and stop receiving

> emails from it, send an email to kubernetes-api-reviewers
> +unsub...@googlegroups.com.
> >>>>>>>>>>>> To post to this group, send email to kubernetes-api-
> revi...@googlegroups.com.

> >>>>>>>>>>>> To view this discussion on the web visit https://

> groups.google.com/d/msgid/kubernetes-api-reviewers/
> 5d953c15-8b47-4059-a319-07d01c1cb7e8%40googlegroups.com.
> >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.

> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> You received this message because you are subscribed to
> the Google Groups "kubernetes-api-reviewers" group.
> >>>>>>>>>>> To unsubscribe from this group and stop receiving emails

> from it, send an email to kubernetes-api-reviewers
> +unsub...@googlegroups.com.
> >>>>>>>>>>> To post to this group, send email to kubernetes-api-
> revi...@googlegroups.com.

> >>>>>>>>>>> To view this discussion on the web visit https://

> groups.google.com/d/msgid/kubernetes-api-reviewers/
> CAB_J3bayFio%3D_%2Bi_P2oyK%3DHfcqEhhXKNy6Hp61uSMuz27NThNQ%40mail.gmail.com.
> >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.

> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> You received this message because you are subscribed to
> the Google Groups "kubernetes-api-reviewers" group.
> >>>>>>>>> To unsubscribe from this group and stop receiving emails

> from it, send an email to kubernetes-api-reviewers
> +unsub...@googlegroups.com.
> >>>>>>>>> To post to this group, send email to kubernetes-api-
> revi...@googlegroups.com.

> >>>>>>>>> To view this discussion on the web visit https://

> groups.google.com/d/msgid/kubernetes-api-reviewers/
> CAH16ShLOWPhQUbq%2B1hZY4x9J8u%3DYvM8tJ%2B2ZmWeP0SYNcbzkYQ%40mail.gmail.com.
> >>>>>>>>> For more options, visit https://groups.google.com/d/optout.

> >>>>>>>
> >>>>>>> --
> >>>>>>> You received this message because you are subscribed to the
> Google Groups "kubernetes-api-reviewers" group.
> >>>>>>> To unsubscribe from this group and stop receiving emails

> from it, send an email to kubernetes-api-reviewers
> +unsub...@googlegroups.com.
> >>>>>>> To post to this group, send email to kubernetes-api-
> revi...@googlegroups.com.

> >>>>>>> To view this discussion on the web visit https://

> groups.google.com/d/msgid/kubernetes-api-reviewers/
> CAH16ShLYFn4BrzAEjzy5JDyDAgwON%3DJUiGfiH9%3DjrZTbN5zg7g%40mail.gmail.com.
> >>>>>>> For more options, visit https://groups.google.com/d/optout.

> >>>>>
> >>>>> --
> >>>>> You received this message because you are subscribed to the
> Google Groups "kubernetes-api-reviewers" group.
> >>>>> To unsubscribe from this group and stop receiving emails from
> it, send an email to kubernetes-api-rev...@googlegroups.com.

> >>>>> To post to this group, send email to kubernetes-api-
> revi...@googlegroups.com.

> >>>>> To view this discussion on the web visit https://

> groups.google.com/d/msgid/kubernetes-api-reviewers/

> CAKCBhs7JrHPPXGxvmxVTShE5UqhnqLp57e3h%2Bi1DEVq8YGmFPA%40mail.gmail.com.
> >>>>> For more options, visit

https://groups.google.com/d/optout.

> >>
> >> --
> >> You received this message because you are subscribed to a topic
> in the Google Groups "kubernetes-sig-instrumentation" group.
> >> To unsubscribe from this topic, visit

https://groups.google.com/

> d/topic/kubernetes-sig-instrumentation/XbElxDtww0Y/unsubscribe.
> >> To unsubscribe from this group and all its topics, send an email to
> kubernetes-sig-instru...@googlegroups.com.
> >> To post to this group, send email to kubernetes-sig-

> instrum...@googlegroups.com.

> >> To view this discussion on the web visit https://
> groups.google.com/d/msgid/kubernetes-sig-instrumentation/

> CAA_vbqRv7jRVpGno%2BotFm8cC5WpFaC2hU%3D6mXwLaDM%2B8LFrXvg%40mail.gmail.com.
> >> For more options, visit

https://groups.google.com/d/optout.

> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-instrumentation" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kubernetes-sig-instru...@googlegroups.com.
> To post to this group, send email to kubernetes-sig-

> instrum...@googlegroups.com.

> To view this discussion on the web visit

https://groups.google.com/
> d/msgid/kubernetes-sig-instrumentation/

> CAA_vbqQRDnQVuTyZnavaKrwrZN2MdwJ0CjJGRNjNLa%2Bii14B%3DQ%40mail.gmail.com.
> For more options, visit

https://groups.google.com/d/optout.
>
> --
> - Han

> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-instrumentation" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kubernetes-sig-instru...@googlegroups.com.
> To post to this group, send email to kubernetes-sig-

> instrum...@googlegroups.com.

> To view this discussion on the web visit

https://groups.google.com/
> d/msgid/kubernetes-sig-instrumentation/CAC_Rkjw2-
> jOZPsfSptSCf4%2Bo2JYsx2FmhEUS%2B%2B4gB08QUSw5kw%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Frederic Branczyk]

The PR didn't make it into code freeze anyways. We're preparing for better communication and further SIG approval for 1.16. We're working on a plan for this and will communicate this in the upcoming weeks. For 1.15 the deprecated metrics won't be removed, however they likely will be for 1.16.

Best,

Frederic