Thanos Query is not able to discover sidecar running on remote clusters.

[Zhang Zhao]

I had 2 Kubernetes clusters on Azure. Thanos Query was installed on Cluster A along with Prometheus+sidecar via stable/prometheus-operator helm chart. I needed to query the remote cluster's sidecar. I had to discover the remote sidecar by ip (- --store=xx.xx.xx.xx:10901), which is a pain as the ip changes periodically and it breaks the connection to Thanos query. For solving this issue, I installed nginx controller. Below is my configuration. However after I re-deployed Thanos query (- --store=prometheus-ba1-nonprod.mydomain.com:80), the remote side was not discovered. Any advice?

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prometheus-ingress
  namespace: espr-prometheus-nonprod  
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/grpc-backend: "true"
spec:
  rules:
  - host: prometheus-ba1-nonprod.mydomain.com
    http:
      paths:
      - backend:
          serviceName: prometheus-operated 
          servicePort: 10901
        path: /

[zzhao05@esco-jumphost-prod-svc-vm-01-escojump000001 ~]$ kubectl  get po

NAME                                                     READY   STATUS    RESTARTS   AGE

alertmanager-prometheus-prometheus-oper-alertmanager-0   2/2     Running   0          14h

nginx-ingress-controller-54db9cb9b5-lk99l                1/1     Running   0          14h

nginx-ingress-controller-54db9cb9b5-swpjc                1/1     Running   0          13h

nginx-ingress-default-backend-7c6cd67dbb-lbmcp           1/1     Running   0          14h

prometheus-kube-state-metrics-7844d8fc49-5sfzw           1/1     Running   0          14h

prometheus-prometheus-node-exporter-bkr5m                1/1     Running   3          8d

prometheus-prometheus-node-exporter-cxqnb                1/1     Running   3          7d15h

prometheus-prometheus-node-exporter-dj86m                1/1     Running   2          26d

prometheus-prometheus-node-exporter-mznr4                1/1     Running   1          15h

prometheus-prometheus-node-exporter-ppvsc                1/1     Running   2          26d

prometheus-prometheus-node-exporter-pspvk                1/1     Running   1          38h

prometheus-prometheus-node-exporter-rspbz                1/1     Running   2          26d

prometheus-prometheus-oper-operator-67ccf47d85-5kbtk     2/2     Running   0          14h

prometheus-prometheus-prometheus-oper-prometheus-0       4/4     Running   1          14h

[zzhao05@esco-jumphost-prod-svc-vm-01-escojump000001 ~]$ kubectl get svc

NAME                                      TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE

alertmanager-operated                     ClusterIP      None           <none>           9093/TCP,9094/TCP,9094/UDP   26d

nginx-ingress-controller                  LoadBalancer   10.0.85.179    xx.xx.xx.xx   80:32615/TCP,443:30561/TCP   7d5h

nginx-ingress-default-backend             ClusterIP      10.0.26.35     <none>           80/TCP                       7d5h

prometheus-kube-state-metrics             ClusterIP      10.0.205.205   <none>           8080/TCP                     26d

prometheus-operated                       ClusterIP      None           <none>           9090/TCP,10901/TCP           26d

prometheus-prometheus-node-exporter       ClusterIP      10.0.1.99      <none>           9100/TCP                     26d

prometheus-prometheus-oper-alertmanager   ClusterIP      10.0.153.203   <none>           9093/TCP                     26d

prometheus-prometheus-oper-operator       ClusterIP      10.0.226.205   <none>           8080/TCP,443/TCP             26d

prometheus-prometheus-oper-prometheus     ClusterIP      10.0.119.255   <none>           9090/TCP                     26d