Prometheus docker swarm container monitoring error <msg="append failed" err="no token found",>

[Raju Bokade]

Bug Report

What did you do?
installed prometheus , node_exporter , cadvisor and trying to monitor my containers

What did you expect to see?
Container is up and running in cluster , expecting Prometheus to report valid status of container

What did you see instead? Under which circumstances?
I see Prometheus target shows container "DOWN" status

Environment

• System information:
  Linux 3.10.0-693.el7.x86_64 x86_64

• Prometheus version:
  version=2.2.0

• Prometheus configuration file:

scrape_configs:
  - job_name: 'node-exporter'
    dns_sd_configs:
    - names: ['tasks.node-exporter']
      type: A
      port: 9100
  - job_name: 'cadvisor'
    dns_sd_configs:
    - names: ['tasks.cadvisor']
      type: A
      port: 8080
  - job_name: 'prometheus'
    dns_sd_configs:
    - names: ['tasks.prometheus']
      type: A
      port: 9090
  - job_name: 'cubing'
    dns_sd_configs:
    - names: ['tasks.cubing']
      type: A
      port: 8080

• Logs:
  level=info ts=2018-05-29T20:18:39.33689246Z caller=main.go:220 msg="Starting Prometheus" version="(version=2.2.0, branch=HEAD, revision=f63e7db4cbdb616337ca877b306b9b96f7f4e381)",
  level=info ts=2018-05-29T20:18:39.336973887Z caller=main.go:221 build_context="(go=go1.10, user=root@52af9f66ce71, date=20180308-16:40:42)",
  level=info ts=2018-05-29T20:18:39.336995927Z caller=main.go:222 host_details="(Linux 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 4893e2b826d4 (none))",
  level=info ts=2018-05-29T20:18:39.337011238Z caller=main.go:223 fd_limits="(soft=65536, hard=65536)",
  level=info ts=2018-05-29T20:18:39.341453015Z caller=main.go:504 msg="Starting TSDB ...",
  level=info ts=2018-05-29T20:18:39.341483656Z caller=web.go:382 component=web msg="Start listening for connections" address=0.0.0.0:9090,
  level=info ts=2018-05-29T20:18:39.571365171Z caller=main.go:514 msg="TSDB started",
  level=info ts=2018-05-29T20:18:39.571427853Z caller=main.go:588 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml,
  level=info ts=2018-05-29T20:18:39.572948911Z caller=main.go:491 msg="Server is ready to receive web requests.",
  level=error ts=2018-05-29T20:18:44.62617786Z caller=notifier.go:473 component=notifier alertmanager=http://alertmanager:9093/api/v1/alerts count=0 msg="Error sending alert" err="Post http://alertmanager:9093/api/v1/alerts: dial tcp: lookup alertmanager on 127.0.0.11:53: server misbehaving",
  level=warn ts=2018-05-29T20:18:47.027667717Z caller=scrape.go:697 component="scrape manager" scrape_pool=cubing target=http://10.0.1.220:8080/metrics msg="append failed" err="no token found",
  level=warn ts=2018-05-29T20:19:17.02425751Z caller=scrape.go:697 component="scrape manager" scrape_pool=cubing target=http://10.0.1.220:8080/metrics msg="append failed" err="no token found",
  level=warn ts=2018-05-29T20:19:47.022839055Z caller=scrape.go:697 component="scrape manager" scrape_pool=cubing target=http://10.0.1.220:8080/metrics msg="append failed" err="no token found",