sample with repeated timestamp but different value (cadvisor)

[seth.r...@verizon.com]

I'm getting the below errors and I'm trying to figure out how to get rid of them ... 

Aug 04 17:28:14 promon0101 prometheus[3555]: time="2017-08-04T17:28:14Z" level=debug msg="Sample discarded" error="sample with repeated timestamp but different value" sample=container_fs_writes_total{datacenter="dallas", device="/dev/mapper/vg_root-lv_var_log_audit", env="dallas_non_prod", id="/", instance="192.168.23.1_42999", job="cadvisor_manager_dallas", role="swarm_manager"} => 17312172 @[1501867694.541] source="scrape.go:527"
Aug 04 17:28:14 promon0101 prometheus[3555]: time="2017-08-04T17:28:14Z" level=debug msg="Sample discarded" error="sample with repeated timestamp but different value" sample=container_fs_writes_total{datacenter="dallas", device="/dev/mapper/vg_root-lv_var_tmp", env="dallas_non_prod", id="/", instance="192.168.23.1_42999", job="cadvisor_manager_dallas", role="swarm_manager"} => 0 @[1501867694.541] source="scrape.go:527"

I have this in the config file ... 

global:
  scrape_interval:     15s
  evaluation_interval: 15s


  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      datacenter: "dallas_non_prod_1"


# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  - "rules/*.rules"


scrape_configs:
  # Auto load *.yml files
  - job_name: 'all_configs'
    file_sd_configs:
    - files:
      - "/etc/prometheus/dallas/*.yml"
      refresh_interval: 3m
    relabel_configs:
    - source_labels: [__address__]
      regex: '([-_0-9A-Za-z.]+):([0-9]+)'
      target_label: instance
      replacement: '${1}'

My /etc/prometheus/dallas/cadvisor.yml file looks like so: 

- labels:
    datacenter: dallas
    env: dallas_non_prod
    job: cadvisor_manager_dallas
    role: swarm_manager
  targets:
  - 192.168.23.1:42999

What am I missing ??

[Björn Rabenstein]

Your relabel config labels away the port. (While it is in parentheses
in your regexp '([-_0-9A-Za-z.]+):([0-9]+)', it is not referred to in
the replacement string.)

My first guess would be that you have targets where the hostname is
the same and only the port differs. Your relabeling is then creating
identical metrics from different targets. However, that kind of
problem usually creates the "other" type of dropped samples, where the
time goes backwards.

In your case, the timestamp is identical, which usually means that
your monitored target is creating multiple copies of the same metric
in a single scrape. You can look directly at the `/metrics` endpoint
of the target is question and check if there are any lines in it that
only differ in the sample value. (Use your favorite pipe of shell
commands.)

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

[seth.r...@verizon.com]

The issue was CAdvisor 0.25 which had every id label with a value of "/"

I upgraded to CAdvisor 0.26 and the issue resolved.