Sidecar on other Kubernetes clusters is not reporting to Thanos Querier by DNS service discovery

[Zhang Zhao]

Thanos Sidecar running on other cluster was not discovered by dnssrv on Thanos Querier. In my Thanos Querier deployment yaml file, I used “—store=dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc.cluster.local” service discover the sidecar. But It ONLY discovered on the prometheus instances that run on the same cluster with Thanos Querier. For the other prometheus instances that run on other clusters, sidecar didn’t report to Thanos Querier so that I had to hard code in the IP address and port. Any idea why DNS Service Discovery did not work on other clusters? Thanks. I was using prometheus-operator helm chart to deploy prometheus and Thanos Sidecar. The service name is exactly identical as “prometheus-operated”.

alex@Zhangs-MacBook prometheus-operator % kubectl get service

NAME                                           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE

alertmanager-operated                          ClusterIP   None           <none>        9093/TCP,9094/TCP,9094/UDP   9d

prometheus-operated                            ClusterIP   None           <none>        9090/TCP,10901/TCP           9d

prometheus-operator-alertmanager               ClusterIP   10.0.55.23     <none>        9093/TCP                     9d

prometheus-operator-grafana                    ClusterIP   10.0.177.193   <none>        80/TCP                       9d

prometheus-operator-kube-state-metrics         ClusterIP   10.0.238.98    <none>        8080/TCP                     9d

prometheus-operator-operator                   ClusterIP   10.0.223.147   <none>        8080/TCP,443/TCP             9d

prometheus-operator-prometheus                 ClusterIP   10.0.127.8     <none>        9090/TCP                     9d

prometheus-operator-prometheus-node-exporter   ClusterIP   10.0.127.50    <none>        9100/TCP                     9d

prometheus-prometheus-operator-prometheus-0    NodePort    10.0.172.129   <none>        9090:30720/TCP               9d

thanos-bucket                                  ClusterIP   10.0.172.148   <none>        10902/TCP                    30h

thanos-compactor                               ClusterIP   10.0.243.77    <none>        10902/TCP                    6d

thanos-querier                                 ClusterIP   10.0.151.135   <none>        10901/TCP,9090/TCP           6d12h

thanos-store                                   ClusterIP   None           <none>        10901/TCP,10902/TCP          6d13h