thanos+ingress-nginx+grpc

[Zhang Zhao]

I setup 2 Kubernetes clusters on Azure, Thanos Querier is in one cluster, and a local prometheus+sidecar. I need to query the remote cluster Thanos sidecar. I created ingress-nginx with support for grpc on remote cluster. But the sidecar grpc failed in Thanos Querier so that Querier was not able to discover the sidecar on remote cluster. I still had to use the ip and port to discover the sidecar on remote clusters. Everytime the pod is restarted, the ip changes and the connection is broken. Any advice?

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

  name: thanos-sidecar-ingress

  namespace: espr-prometheus-nonprod  

  annotations:

    kubernetes.io/ingress.class: “nginx”

    nginx.ingress.kubernetes.io/ssl-redirect: "false"

    nginx.ingress.kubernetes.io/backend-protocol: “GRPC”

spec:

  rules:

  - host: xxx.example.com 

     http:

      paths:

      - backend:

          serviceName: prometheus-operated 

          servicePort: 10901

        path: /(.*)

Zhang

[Tristan Colgate]

(before I answer, just a quick note that your mail client seems to use
HTML with a black background and white foreground for text, and it
looks super weird if the reader has a white background mail client as
it is all stripey).

Two things:

1. it sounds like maybe your nginx ingress isn't doing the right
thing. I've not used the nginx ingress, but you may need to indicate
to it that the backend requires http2.

2. Whether you point the querier at the nginx ingress, or the sidecar
pod, you probably want to do that via a kubernetes service rather than
direct to the pod's IP. If you use a service with a clusterIP, that IP
is fixed until such time as your delete that service. If you are in a
different cluster you may need to use a service with type
"loadbalancer", you may need to look at how to make that an internal
only load balancer in azure (normally done by adding an annotation),
which would restrict it to be internal to the network of your cloud
provider, and not public.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/97dd931f-4720-4572-9c89-cccb3d16ccf8o%40googlegroups.com.



--
Tristan Colgate-McFarlane
----
"You can get all your daily vitamins from 52 pints of guiness, and a
glass of milk"

[Zhang Zhao]

Hi Tristan,

Thank you for your reply and sorry for the weird background color. 

For the backend, it doesn’t actually use http protocol. It uses GRPC. For your 2nd point, actually that’s what I tried. I added the annotation below when I was configuring nginx ingress controller and an internal ip to my network was assigned to the ingress service and pointed it to the sidecar service. The problem was that on Thanos Query side, the sidecar was not able to be discovered by the ingress load balancer ip.

service.beta.kubernetes.io/azure-load-balancer-internal: "true"

[Tristan Colgate]

grpc uses http/2, some load balancers need to be explicitly configured, and most require http/2 over Https.

 You'll need to setup a DNS entry for the loadbalancer IP, or use the IP directly.  What options are you providing to query ?

[Zhang Zhao]

Yes. I setup a DNS entry. So where shall I add the “http/2” protocol?? Below is my nginx ingress config.

apiVersion: v1

kind: Service

metadata:

  annotations:

    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

  creationTimestamp: "2020-07-01T16:39:41Z"

  finalizers:

  - service.kubernetes.io/load-balancer-cleanup

  labels:

    app: nginx-ingress

    chart: nginx-ingress-1.40.2

    component: controller

    heritage: Helm

    release: nginx-ingress

  name: nginx-ingress-controller

  namespace: espr-prometheus-nonprod

 resourceVersion: "7098976"

  selfLink: /api/v1/namespaces/espr-prometheus-nonprod/services/nginx-ingress-controller

  uid: e1238468-5d80-446b-b2d0-364f068208b6

spec:

  clusterIP: xx.xx.xx.xx

  externalTrafficPolicy: Cluster

  ports:

  - name: http

    nodePort: 32615

    port: 80

    protocol: TCP

    targetPort: http

  - name: https

    nodePort: 30561

    port: 443

    protocol: TCP

    targetPort: https

  selector:

    app: nginx-ingress

    app.kubernetes.io/component: controller

    release: nginx-ingress

  sessionAffinity: None

  type: LoadBalancer

status:

  loadBalancer:

    ingress:

    - ip: yy.yy.yy.yy