Unable to communicate with Thanos sidecar behind the Nginx Ingress

[Volker Dormeyer]

Hi!

I have a problem within Thanos. I know, this is a Prometheus mailing list...

I experience following issue: The Query instance is not able to
communicate with the Thanos Sidecar. The error message says the following:

|level=info ts=2022-06-21T13:12:13.219335188Z caller=client.go:55
msg="enabling client to server TLS" level=info
ts=2022-06-21T13:12:13.219501389Z caller=options.go:115 msg="TLS client
using provided certificate pool" level=info
ts=2022-06-21T13:12:13.21951672Z caller=options.go:148 msg="TLS client
authentication enabled" level=info ts=2022-06-21T13:12:13.223788976Z
caller=options.go:31 protocol=gRPC msg="enabling server side TLS"
level=info ts=2022-06-21T13:12:13.22419521Z caller=options.go:61
protocol=gRPC msg="server TLS client verification enabled" level=info
ts=2022-06-21T13:12:13.224696576Z caller=query.go:705 msg="starting
query node" level=info ts=2022-06-21T13:12:13.224820551Z
caller=intrumentation.go:75 msg="changing probe status" status=healthy
level=info ts=2022-06-21T13:12:13.224871106Z caller=http.go:73
service=http/server component=query msg="listening for requests and
metrics" address=0.0.0.0:10902 level=info
ts=2022-06-21T13:12:13.225078636Z caller=intrumentation.go:56
msg="changing probe status" status=ready level=info
ts=2022-06-21T13:12:13.225222457Z caller=tls_config.go:195
service=http/server component=query msg="TLS is disabled." http2=false
level=info ts=2022-06-21T13:12:13.225290133Z caller=grpc.go:131
service=gRPC/server component=query msg="listening for serving gRPC"
address=0.0.0.0:10901 level=warn ts=2022-06-21T13:12:18.226170081Z
caller=endpointset.go:517 component=endpointset msg="update of node
failed" err="getting metadata: fallback fetching info from
thanos-sc.dev.example.org:443: rpc error: code = DeadlineExceeded desc =
context deadline exceeded" address=thanos-sc.dev.example.org:443 |

I already checked the certificates twice and more. What makes me
thinking that the grpcurl result also in a error:

|$ grpcurl -insecure thanos-sc.local:443 list Failed to dial target host
"thanos-sc.dev.example.org:443": remote error: tls: no application
protocol I am able to port-forward the 10901 an them access the port
successfully via grpcurl. |*My Ingress:*

|apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations:
meta.helm.sh/release-name: prom meta.helm.sh/release-namespace: mon
nginx.ingress.kubernetes.io/backend-protocol: GRPC
nginx.ingress.kubernetes.io/ssl-redirect: "true" labels: app:
kube-prometheus-stack-prometheus app.kubernetes.io/instance: prom
app.kubernetes.io/managed-by: Helm app.kubernetes.io/part-of:
kube-prometheus-stack app.kubernetes.io/version: 32.2.1 chart:
kube-prometheus-stack-32.2.1 heritage: Helm release: prom name:
prom-kube-prometheus-stack-thanos-gateway namespace: mon spec:
ingressClassName: nginx rules: - host: thanos-sc.dev.example.org http:
paths: - backend: service: name: prom-kube-prometheus-stack-prometheus
port: number: 10901 path: / pathType: ImplementationSpecific tls: -
secretName: new-tls-secret hosts: - thanos-sc.dev.example.org |

*Thanos Query:*

||

      containers:
      - args:
        - query
        - --log.level=info
        - --log.format=logfmt
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --query.replica-label=replica
        - --store=thanos-sc.dev.example.org:443
        - --grpc-server-tls-cert=/certs/server/tls.crt
        - --grpc-server-tls-key=/certs/server/tls.key
        - --grpc-server-tls-client-ca=/certs/server/ca.crt
        - --grpc-client-tls-secure
        - --grpc-client-tls-cert=/certs/client/tls-cert
        - --grpc-client-tls-key=/certs/client/tls-key
        - --grpc-client-tls-ca=/certs/client/ca-cert
        - --grpc-client-server-name=thanos-sc.dev.example.org


Do have some idea of it.

Best Regards,
Volker

||

[Volker Dormeyer]

my first mail was in a ugly format... this is a resent

My Ingress:

My Thanos Query:

      containers:
      - args:
        - query
        - --log.level=info
        - --log.format=logfmt
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --query.replica-label=replica
        - --store=thanos-sc.dev.example.org:443
        - --grpc-server-tls-cert=/certs/server/tls.crt
        - --grpc-server-tls-key=/certs/server/tls.key
        - --grpc-server-tls-client-ca=/certs/server/ca.crt
        - --grpc-client-tls-secure
        - --grpc-client-tls-cert=/certs/client/tls-cert
        - --grpc-client-tls-key=/certs/client/tls-key
        - --grpc-client-tls-ca=/certs/client/ca-cert
        - --grpc-client-server-name=thanos-sc.dev.example.org

Thanks,
Volker

[Volker Dormeyer]

Hello all,

my problem is solved.

Thanks,
Volker

[Brian Candler]

For the benefit of the list, what was the issue and the solution?

[Volker Dormeyer]

Unfortunatly the problem does still exist. :( When I wrote my last Mail, I was in opinion that the problem is solved.

Meanwhile I use an Envoy proxy to have a better TLS handling. Now the same problem exist between the envoy forward proxy and the Ingress in the other Cluster.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bedf5439-5117-4c30-9389-53e2357e42b5n%40googlegroups.com.

[Volker Dormeyer]

Hello list,

I was able to solve my problem after a long journey. I simply replaced
the Ingressinc (https://github.com/nginxinc/kubernetes-ingress) by
https://github.com/kubernetes/ingress-nginx and it started to work.

Best Regards,

Volker

>> > metrics" address=0.0.0.0:10902 <http://0.0.0.0:10902>

>> > level=info ts=2022-06-21T13:12:13.225078636Z
>> > caller=intrumentation.go:56 msg="changing probe status"
>> status=ready
>> > level=info ts=2022-06-21T13:12:13.225222457Z
>> caller=tls_config.go:195
>> > service=http/server component=query msg="TLS is disabled."
>> http2=false
>> > level=info ts=2022-06-21T13:12:13.225290133Z caller=grpc.go:131
>> > service=gRPC/server component=query msg="listening for serving
>> gRPC"

>> > address=0.0.0.0:10901 <http://0.0.0.0:10901>

>> > level=warn ts=2022-06-21T13:12:18.226170081Z
>> caller=endpointset.go:517
>> > component=endpointset msg="update of node failed" err="getting
>> > metadata: fallback fetching info from
>> thanos-sc.dev.example.org:443

>> <http://thanos-sc.dev.example.org:443>:

>> > rpc error: code = DeadlineExceeded desc = context deadline
>> exceeded"
>> > address=thanos-sc.dev.example.org:443

>> <http://thanos-sc.dev.example.org:443>

>> >
>> > I already checked the certificates twice and more. What makes me
>> > thinking that the grpcurl result also in a error:
>> >
>> > $ grpcurl -insecure thanos-sc.local:443 list
>> > Failed to dial target host "thanos-sc.dev.example.org:443

>> <http://thanos-sc.dev.example.org:443>": remote

>> > error: tls: no application protocol
>> >
>> > I am able to port-forward the 10901 an them access the port
>> > successfully via grpcurl.
>> >
>> > My Ingress:
>> >

>> > apiVersion: networking.k8s.io/v1 <http://networking.k8s.io/v1>
>> > kind: Ingress
>> > metadata:
>> >   annotations:
>> > meta.helm.sh/release-name <http://meta.helm.sh/release-name>: prom
>> > meta.helm.sh/release-namespace
>> <http://meta.helm.sh/release-namespace>: mon
>> > nginx.ingress.kubernetes.io/backend-protocol
>> <http://nginx.ingress.kubernetes.io/backend-protocol>: GRPC
>> > nginx.ingress.kubernetes.io/ssl-redirect
>> <http://nginx.ingress.kubernetes.io/ssl-redirect>: "true"
>> >   labels:
>> >     app: kube-prometheus-stack-prometheus
>> > app.kubernetes.io/instance <http://app.kubernetes.io/instance>:
>> prom
>> > app.kubernetes.io/managed-by
>> <http://app.kubernetes.io/managed-by>: Helm
>> > app.kubernetes.io/part-of <http://app.kubernetes.io/part-of>:
>> kube-prometheus-stack
>> > app.kubernetes.io/version <http://app.kubernetes.io/version>:

>> 32.2.1
>> >     chart: kube-prometheus-stack-32.2.1
>> >     heritage: Helm
>> >     release: prom
>> >   name: prom-kube-prometheus-stack-thanos-gateway
>> >   namespace: mon
>> > spec:
>> >   ingressClassName: nginx
>> >   rules:
>> >   - host: thanos-sc.dev.example.org

>> <http://thanos-sc.dev.example.org>

>> >     http:
>> >       paths:
>> >       - backend:
>> >           service:
>> >             name: prom-kube-prometheus-stack-prometheus
>> >             port:
>> >               number: 10901
>> >         path: /
>> >         pathType: ImplementationSpecific
>> >   tls:
>> >   - secretName: new-tls-secret
>> >     hosts:
>> >       - thanos-sc.dev.example.org

>> <http://thanos-sc.dev.example.org>

>> >
>> > My Thanos Query:
>> >
>> >       containers:
>> >       - args:
>> >         - query
>> >         - --log.level=info
>> >         - --log.format=logfmt

>> >         - --grpc-address=0.0.0.0:10901 <http://0.0.0.0:10901>
>> >         - --http-address=0.0.0.0:10902 <http://0.0.0.0:10902>

>> >         - --query.replica-label=replica
>> >         - --store=thanos-sc.dev.example.org:443

>> <http://thanos-sc.dev.example.org:443>

>> >         - --grpc-server-tls-cert=/certs/server/tls.crt
>> >         - --grpc-server-tls-key=/certs/server/tls.key
>> >         - --grpc-server-tls-client-ca=/certs/server/ca.crt
>> >         - --grpc-client-tls-secure
>> >         - --grpc-client-tls-cert=/certs/client/tls-cert
>> >         - --grpc-client-tls-key=/certs/client/tls-key
>> >         - --grpc-client-tls-ca=/certs/client/ca-cert
>> >         - --grpc-client-server-name=thanos-sc.dev.example.org

>> <http://thanos-sc.dev.example.org>

>> <http://0.0.0.0:10902> level=info

>> >> ts=2022-06-21T13:12:13.225078636Z caller=intrumentation.go:56
>> >> msg="changing probe status" status=ready level=info
>> >> ts=2022-06-21T13:12:13.225222457Z caller=tls_config.go:195
>> >> service=http/server component=query msg="TLS is disabled."
>> >> http2=false level=info ts=2022-06-21T13:12:13.225290133Z
>> >> caller=grpc.go:131 service=gRPC/server component=query
>> msg="listening

>> >> for serving gRPC" address=0.0.0.0:10901 <http://0.0.0.0:10901>

>> level=warn
>> >> ts=2022-06-21T13:12:18.226170081Z caller=endpointset.go:517
>> >> component=endpointset msg="update of node failed" err="getting
>> >> metadata: fallback fetching info from
>> thanos-sc.dev.example.org:443

>> <http://thanos-sc.dev.example.org:443>:

>> >> rpc error: code = DeadlineExceeded desc = context deadline
>> exceeded"
>> >> address=thanos-sc.dev.example.org:443

>> <http://thanos-sc.dev.example.org:443> |

>> >>
>> >> I already checked the certificates twice and more. What makes me
>> >> thinking that the grpcurl result also in a error:
>> >>
>> >> |$ grpcurl -insecure thanos-sc.local:443 list Failed to dial
>> target
>> >> host "thanos-sc.dev.example.org:443

>> <http://thanos-sc.dev.example.org:443>": remote error: tls: no

>> >> application protocol I am able to port-forward the 10901 an them
>> >> access the port successfully via grpcurl. |*My Ingress:*
>> >>
>> >> |apiVersion: networking.k8s.io/v1

>> <http://networking.k8s.io/v1> kind: Ingress metadata:
>> >> annotations: meta.helm.sh/release-name
>> <http://meta.helm.sh/release-name>: prom
>> >> meta.helm.sh/release-namespace
>> <http://meta.helm.sh/release-namespace>: mon
>> >> nginx.ingress.kubernetes.io/backend-protocol
>> <http://nginx.ingress.kubernetes.io/backend-protocol>: GRPC
>> >> nginx.ingress.kubernetes.io/ssl-redirect
>> <http://nginx.ingress.kubernetes.io/ssl-redirect>: "true" labels:
>> app:
>> >> kube-prometheus-stack-prometheus app.kubernetes.io/instance
>> <http://app.kubernetes.io/instance>: prom
>> >> app.kubernetes.io/managed-by
>> <http://app.kubernetes.io/managed-by>: Helm
>> app.kubernetes.io/part-of <http://app.kubernetes.io/part-of>:
>> >> kube-prometheus-stack app.kubernetes.io/version
>> <http://app.kubernetes.io/version>: 32.2.1 chart:

>> >> kube-prometheus-stack-32.2.1 heritage: Helm release: prom name:
>> >> prom-kube-prometheus-stack-thanos-gateway namespace: mon spec:
>> >> ingressClassName: nginx rules: - host:

>> thanos-sc.dev.example.org <http://thanos-sc.dev.example.org>

>> >> http: paths: - backend: service: name:
>> >> prom-kube-prometheus-stack-prometheus port: number: 10901 path: /
>> >> pathType: ImplementationSpecific tls: - secretName:
>> new-tls-secret
>> >> hosts: - thanos-sc.dev.example.org

>> <http://thanos-sc.dev.example.org> |

>> >>
>> >> *Thanos Query:*
>> >>
>> >> ||
>> >>
>> >>       containers:
>> >>       - args:
>> >>         - query
>> >>         - --log.level=info
>> >>         - --log.format=logfmt

>> >>         - --grpc-address=0.0.0.0:10901 <http://0.0.0.0:10901>
>> >>         - --http-address=0.0.0.0:10902 <http://0.0.0.0:10902>

>> >>         - --query.replica-label=replica
>> >>         - --store=thanos-sc.dev.example.org:443

>> <http://thanos-sc.dev.example.org:443>

>> >>         - --grpc-server-tls-cert=/certs/server/tls.crt
>> >>         - --grpc-server-tls-key=/certs/server/tls.key
>> >>         - --grpc-server-tls-client-ca=/certs/server/ca.crt
>> >>         - --grpc-client-tls-secure
>> >>         - --grpc-client-tls-cert=/certs/client/tls-cert
>> >>         - --grpc-client-tls-key=/certs/client/tls-key
>> >>         - --grpc-client-tls-ca=/certs/client/ca-cert
>> >>         - --grpc-client-server-name=thanos-sc.dev.example.org

>> <http://thanos-sc.dev.example.org>

>> >>
>> >>
>> >> Do have some idea of it.
>> >>
>> >> Best Regards,
>> >> Volker
>> >>
>> >> ||
>> >>
>> >
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/bedf5439-5117-4c30-9389-53e2357e42b5n%40googlegroups.com

>> <https://groups.google.com/d/msgid/prometheus-users/bedf5439-5117-4c30-9389-53e2357e42b5n%40googlegroups.com?utm_medium=email&utm_source=footer>.