Metric seggregation by projectID for stackdriver exporter

[Dinesh Koritela]

Hello All,

Can anyone suggest how can i seggregate the metrics coming out of stackdriver exporter from multiple ProjectIDs with multiple APIs (compute, container 7 etc).

Also i am not seeing the projectID label getting pouplated when scraped using the prometheus scrape job.Any help here?

Can we use relabel config section and drop all metrics using the projectID?

[Brian Candler]

Ref: https://github.com/prometheus-community/stackdriver_exporter

According to this, the metrics should have a project_id label.  So my first suggestion is to take Prometheus out of the equation, and talk directly to the exporter using curl.  You will then see the exact metrics as returned to you - and you can show some examples here.

> Can we use relabel config section and drop all metrics using the projectID?

Yes - although it would be more efficient not to collect the unwanted metrics in the first place (i.e. configure the exporter with flag "google.project-id" which according to the docs is a comma-separated list of project IDs to collect)

Otherwise, to drop all metrics with project_id="abc123" or project_id="def456" you'd do something like this:

metric_relabel_configs:

- source_labels: [project_id]

  regex: 'abc123|def456'

  action: drop

Or to drop all metrics except those with project_id="ghi789" or no project_id at all:

metric_relabel_configs:

- source_labels: [project_id]

  regex: 'ghi789|'

  action: keep

Note that:

* "relabel_configs" is used to adjust the results from service discovery _before_ scraping, and therefore can select which targets are to be scraped. It can also set labels which will be added to *all* metrics collected from that target.

* "metric_relabel_configs" is used to process the received metrics _after_ scraping - and therefore can selectively drop or change individual metrics. 

If you are talking to multiple instances of stackdriver_exporter, and you are using static_configs or file_sd_configs, then that gives another way to set labels which will be added to *all* metrics returned from each target. e.g.

- targets:

    - 1.2.3.4:9255

    - 5.6.7.8:9255

  labels:

    env: prod

- targets:

    - 9.10.11.12:9255

  labels:

    env: test

This sets env="prod" on all metrics from the first two exporters, and env="test" on all metrics from the third exporter.

[Dinesh Koritela]

Thank you for the swift response Brian, 

Tried both the actions with the metric_relabel_configs and it worked good.

metric_relabel_configs:

- source_labels: [project_id]

  regex: 'abc123|def456'

  action: drop