empty external labels are not allowed for Thanos block

[Thomas Will]

Hello guys, while setting up sidecar, I am getting this error.

level=warn ts=2020-06-30T00:59:28.739042462Z caller=sidecar.go:131 msg="failed to fetch initial external labels. Is Prometheus running? Retrying" err="request config against http://ec2-54-166-121-208.compute-
level=info ts=2020-06-30T00:59:28.786279597Z caller=shipper.go:200 msg="upload new block" id=01EC194NMFNVXJV3P3ZJV711FX
level=error ts=2020-06-30T00:59:28.788272247Z caller=shipper.go:165 msg="shipping failed" block=01EC194NMFNVXJV3P3ZJV711FX err="empty external labels are not allowed for Thanos block."
level=info ts=2020-06-30T00:59:28.797210843Z caller=shipper.go:200 msg="upload new block" id=01EC19C00DSNBYDF0QY6SJ0RCJ
level=error ts=2020-06-30T00:59:28.798609749Z caller=shipper.go:165 msg="shipping failed" block=01EC19C00DSNBYDF0QY6SJ0RCJ err="empty external labels are not allowed for Thanos block."
level=info ts=2020-06-30T00:59:28.80781908Z caller=shipper.go:200 msg="upload new block" id=01EC19N4ZDHV3A9ZAV4CW0W50Z
level=error ts=2020-06-30T00:59:28.809984074Z caller=shipper.go:165 msg="shipping failed" block=01EC19N4ZDHV3A9ZAV4CW0W50Z err="empty external labels are not allowed for Thanos block."
level=info ts=2020-06-30T00:59:28.871277989Z caller=shipper.go:200 msg="upload new block" id=01EC19Y9XM4EVA6C5XQB6QHA1H
level=error ts=2020-06-30T00:59:28.873172562Z caller=shipper.go:165 msg="shipping failed" block=01EC19Y9XM4EVA6C5XQB6QHA1H err="empty external labels are not allowed for Thanos block."
level=info ts=2020-06-30T00:59:28.941585234Z caller=shipper.go:200 msg="upload new block" id=01EC1A7EXED6X31J8AVMME60N2
level=error ts=2020-06-30T00:59:28.943638781Z caller=shipper.go:165 msg="shipping failed" block=01EC1A7EXED6X31J8AVMME60N2 err="empty external labels are not allowed for Thanos block."
level=info ts=2020-06-30T00:59:28.959223519Z caller=shipper.go:200 msg="upload new block" id=01EC1AGKWE20FEFYPG2EVQZ973
level=error ts=2020-06-30T00:59:28.960929593Z caller=shipper.go:165 msg="shipping failed" block=01EC1AGKWE20FEFYPG2EVQZ973 err="empty external labels are not allowed for Thanos block."
level=info ts=2020-06-30T00:59:28.984357407Z caller=shipper.go:200 msg="upload new block" id=01EC1ASRVEPMF6YDVHWCZ3F9HG
level=error ts=2020-06-30T00:59:28.986786419Z caller=shipper.go:165 msg="shipping failed" block=01EC1ASRVEPMF6YDVHWCZ3F9HG err="empty external labels are not allowed for Thanos block."
level=info ts=2020-06-30T00:59:28.999918447Z caller=shipper.go:200 msg="upload new block" id=01EC1B2XTFF24K5QZZCW4FECRW
level=error ts=2020-06-30T00:59:29.00133826Z caller=shipper.go:165 msg="shipping failed" block=01EC1B2XTFF24K5QZZCW4FECRW err="empty external labels are not allowed for Thanos block."
level=info ts=2020-06-30T00:59:29.020650171Z caller=shipper.go:200 msg="upload new block" id=01EC1BC2SHK8Z9HX3DNAAAPCVM
level=error ts=2020-06-30T00:59:29.024475442Z caller=shipper.go:165 msg="shipping failed" block=01EC1BC2SHK8Z9HX3DNAAAPCVM err="empty external labels are not allowed for Thanos block."
level=info ts=2020-06-30T00:59:29.032014206Z caller=shipper.go:200 msg="upload new block" id=01EC1BN7RFWV6C93ZNG9BAYZ5M
level=error ts=2020-06-30T00:59:29.033591482Z caller=shipper.go:165 msg="shipping failed" block=01EC1BN7RFWV6C93ZNG9BAYZ5M err="empty external labels are not allowed for Thanos block."

sidecar config :-

# thanos sidecar
./thanos sidecar \
    --prometheus.url "http://ec2-42-685-242-86.compute-1.amazonaws.com:9090" \
    --tsdb.path "data" \
    --cluster.address  "localhost:20056" \
    --http-address     "localhost:20057" \
    --grpc-address     "localhost:20058" \
    --log.level=debug \
    >> log/sidecar.log 2>&1 &

prometheus config file :-

global:
  scrape_interval: 15s
  evaluation_interval: 1m
  external_labels:
    monitor: prometheus_monitor
    cluster: 102323

Can anyone please point out my mistake?

Thanks,

[Thomas Will]

        "ulid": "01EC1E4WSJVFQWCFH0J0CKRQTQ",
        "minTime": 1593480811780,
        "maxTime": 1593480900000,
        "stats": {
                "numSamples": 25450,
                "numSeries": 5046,
                "numChunks": 5046
        },
        "compaction": {
                "level": 1,
                "sources": [
                        "01EC1E4WSJVFQWCFH0J0CKRQTQ"
                ]
        },
        "version": 1

This is the Prometheus meta.json file. I think external_labels should be present in this?

[Thomas Will]

I can see external_labels on prometheus UI.

[Thomas Will]

Can anyone please help me out on this issue. It's a blocker for me.

Thanks,

[Bartłomiej Płotka]

Hej, For Thanos questions it's best to visit #thanos channel on CNCF slack. You can find a link to the slack here https://slack.cncf.io/

TL;DR It would be nice to know if sidecar connects to correct Promethus server, otherwise it takes labels driectly from config API, so we did not see any issue on that side. Best way is to also try newest version of Thanos (v0.13.0)

Bartek