Filter the cadvisor metrics

[wangpeng007]

cAdvisor returns metrics for a lot of the running processes on the system its running on. This results in a lot of extra metrics that cAdvisor exposes, and for the most these are metrics we do not care about. 

Is there any API or configuration to filter the cadvisor metrics?

[wangpeng007]

Is there any suggestion?

[Matthias Rampke]

What issues agree these causing? They are returned pretty quickly, can you just fetch them all and filter in whatever collects them?

A server side filter API could become quite complex trying to cover all possible ways to filter, while it's rather easy to filter for any particular use case on the client side.

/MR

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Justin W. Flory]

On 07/26/2017 02:19 AM, Matthias Rampke wrote:
> What issues agree these causing? They are returned pretty quickly, can
> you just fetch them all and filter in whatever collects them?
>
> A server side filter API could become quite complex trying to cover all
> possible ways to filter, while it's rather easy to filter for any
> particular use case on the client side.
>
> /MR
>
>
> On Wed, Jul 26, 2017, 05:39 wangpeng007 <huihui...@gmail.com
> <mailto:huihui...@gmail.com>> wrote:
>
> Is there any suggestion?
>
>
> On Tuesday, July 18, 2017 at 4:06:57 PM UTC+8, wangpeng007 wrote:
>
> cAdvisor returns metrics for a lot of the running processes on
> the system its running on. This results in a lot of extra
> metrics that cAdvisor exposes, and for the most these are
> metrics we do not care about.
> Is there any API or configuration to filter the cadvisor metrics?
>

Admittedly, I'm pretty novice to all of this, but if you're taking these
metrics and putting them into a time-series database (depending on what
you're using), you could set up opt-in filters to only collect the
measurements you actually want or need.

If you're using InfluxDB / Kapacitor specifically, you could also write
a script that aggregates some of the measurements together into smaller
ones, and then write those ones out to the database for storing.

Not sure if this really answers your question, but if the reason you're
wanting to filter out some of the metrics is disk space, you probably
want to look into a way to only collect the measurements you want / need
on an opt-in basis, so the rest are discarded.

--
Cheers,
Justin W. Flory
jfl...@gmail.com

[Ben Kochie]

Yes, that is correct.

Similar techniques apply for Prometheus.  There is a set of rewrite rules that can be used to filter metrics on ingestion, and recording rules to aggregate data.

As Matthias says, this kind of policy stuff generally better left to the service reading the data, because it knows what is necessary or not.

There are some use case for the metrics request to include a filter/collector pattern, to help target the data returned.  But there isn't a good universal pattern for this yet.  

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[wangpeng007]

Thanks for all your guys answers!

I have taken your advise and grabe all metrics from cadvisor and filter it by service!

Thanks again!

I still have a small question:

As containers has many status like:

• Up 1 day (paused)
  
• Restarting (123) 1 day ago
  
• Up 1 day
  
• Removal in Progress
  
• Dead
  
• Created
  
• Exited (123) 1 day ago
  
• (empty string)
  

Which metrics in cadvisor show this?

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

[Ben Kochie]

On Thu, Jul 27, 2017 at 8:35 AM, wangpeng007 <huihui...@gmail.com> wrote:

Thanks for all your guys answers!

I have taken your advise and grabe all metrics from cadvisor and filter it by service!

Thanks again!

I still have a small question:

As containers has many status like:

• Up 1 day (paused)
  
• Restarting (123) 1 day ago
  
• Up 1 day
  
• Removal in Progress
  
• Dead
  
• Created
  
• Exited (123) 1 day ago
  
• (empty string)
  

Which metrics in cadvisor show this?

Those look like event logs, not metrics.  They also look like Kubernetes specifc state changes, which will not be in cAdvisor, but in the API or Kubelet.  cAdvisor is for tracking the container use metrics.