Failed to fetch a list of nodes from Kubernetes API
1,640 views
Skip to first unread message
蓝之忘
Nov 8, 2022, 3:21:15 AM11/8/22
to rabbitmq-users
Hello everyone,
we are using the RabbitMQ Cluster Operator on Kubernetes v1.24.3 (with containerd://1.6.6).
The Cluster Operator by default uses the rabbitmq:3.8.12-management image for server pods.
I use a dual-stack tree network environment. When I change the ipFamilies of services to IPv6, the endpoints are as follows
apiVersion: v1
kind: Endpoints
metadata:
name: my-rabbitmq-nodes
namespace: rabbitmq
subsets:
- addresses:
- hostname: my-rabbitmq-server-0
ip: fd00:10:16::1923
nodeName: 192.168.132.208
targetRef:
kind: Pod
name: my-rabbitmq-server-0
namespace: hz-rabbitmq
resourceVersion: "9709482"
uid: 1d5e0378-7309-46f7-8c70-587d4b68f853
- hostname: my-rabbitmq-server-1
ip: fd00:10:16::1924
nodeName: 192.168.132.209
targetRef:
kind: Pod
name: my-rabbitmq-server-1
namespace: hz-rabbitmq
resourceVersion: "9709492"
uid: c0846f86-d4a1-4ef2-8424-3ddb43a4026b
ports:
- name: cluster-rpc
port: 25672
protocol: TCP
- name: epmd
port: 4369
protocol: TCP
rabbitmq log：
2022-11-08 14:03:43.243 [debug] <0.274.0> GET https://kubernetes.default:443/api/v1/namespaces/hz-rabbitmq/endpoints/my-rabbitmq-nodes
2022-11-08 14:03:43.292 [debug] <0.274.0> Response: {error,{failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}}
2022-11-08 14:03:43.292 [debug] <0.274.0> HTTP error {failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}
2022-11-08 14:03:43.293 [error] <0.274.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
{inet6,[inet6],nxdomain}]}
2022-11-08 14:03:43.296 [debug] <0.274.0> POST https://kubernetes.default:443/api/v1/namespaces/hz-rabbitmq/events/ ["{\"count\":1,\"involvedObject\":{\"apiVersion\":\"v1\",\"kind\":\"RabbitMQ\",\"name\":\"pod/my-rabbitmq-server-0\",\"namespace\":\"hz-rabbitmq\"},\"lastTimestamp\":\"2022-11-08T14:03:43+08:00\",\"message\":\"Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{\\\"kubernetes.default\\\",443}},\\n {inet6,[inet6],nxdomain}]}\",\"metadata\":{\"name\":\"my-rabbitmq-server-0.1667887423294\",\"namespace\":\"hz-rabbitmq\"},\"reason\":\"Failed\",\"source\":{\"component\":\"my-rabbitmq-server-0/rabbitmq_peer_discovery\",\"host\":\"my-rabbitmq-server-0\"},\"type\":\"Warning\"}"]
2022-11-08 14:03:43.302 [debug] <0.274.0> Response: [{error,{failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}}]
2022-11-08 14:03:43.303 [debug] <0.274.0> HTTP error {failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}
2022-11-08 14:03:43.303 [error] <0.274.0> Peer discovery returned an error: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n {inet6,[inet6],nxdomain}]}". Will retry after a delay of 500 ms, 9 retries left...
2022-11-08 14:03:43.804 [debug] <0.274.0> GET https://kubernetes.default:443/api/v1/namespaces/hz-rabbitmq/endpoints/my-rabbitmq-nodes
2022-11-08 14:03:43.810 [debug] <0.274.0> Response: {error,{failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}}
2022-11-08 14:03:43.811 [debug] <0.274.0> HTTP error {failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}
2022-11-08 14:03:43.811 [error] <0.274.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
{inet6,[inet6],nxdomain}]}
2022-11-08 14:03:43.812 [debug] <0.274.0> POST https://kubernetes.default:443/api/v1/namespaces/hz-rabbitmq/events/ ["{\"count\":1,\"involvedObject\":{\"apiVersion\":\"v1\",\"kind\":\"RabbitMQ\",\"name\":\"pod/my-rabbitmq-server-0\",\"namespace\":\"hz-rabbitmq\"},\"lastTimestamp\":\"2022-11-08T14:03:43+08:00\",\"message\":\"Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{\\\"kubernetes.default\\\",443}},\\n {inet6,[inet6],nxdomain}]}\",\"metadata\":{\"name\":\"my-rabbitmq-server-0.1667887423812\",\"namespace\":\"hz-rabbitmq\"},\"reason\":\"Failed\",\"source\":{\"component\":\"my-rabbitmq-server-0/rabbitmq_peer_discovery\",\"host\":\"my-rabbitmq-server-0\"},\"type\":\"Warning\"}"]
2022-11-08 14:03:43.818 [debug] <0.274.0> Response: [{error,{failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}}]
2022-11-08 14:03:43.818 [debug] <0.274.0> HTTP error {failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}
2022-11-08 14:03:43.818 [error] <0.274.0> Peer discovery returned an error: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n {inet6,[inet6],nxdomain}]}". Will retry after a delay of 500 ms, 8 retries left...
2022-11-08 14:03:44.320 [debug] <0.274.0> GET https://kubernetes.default:443/api/v1/namespaces/hz-rabbitmq/endpoints/my-rabbitmq-nodes
2022-11-08 14:03:44.358 [debug] <0.274.0> Response: {error,{failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}}
2022-11-08 14:03:44.358 [debug] <0.274.0> HTTP error {failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}
2022-11-08 14:03:44.358 [error] <0.274.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
{inet6,[inet6],nxdomain}]}
2022-11-08 14:03:44.360 [debug] <0.274.0> POST https://kubernetes.default:443/api/v1/namespaces/hz-rabbitmq/events/ ["{\"count\":1,\"involvedObject\":{\"apiVersion\":\"v1\",\"kind\":\"RabbitMQ\",\"name\":\"pod/my-rabbitmq-server-0\",\"namespace\":\"hz-rabbitmq\"},\"lastTimestamp\":\"2022-11-08T14:03:44+08:00\",\"message\":\"Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{\\\"kubernetes.default\\\",443}},\\n {inet6,[inet6],nxdomain}]}\",\"metadata\":{\"name\":\"my-rabbitmq-server-0.1667887424359\",\"namespace\":\"hz-rabbitmq\"},\"reason\":\"Failed\",\"source\":{\"component\":\"my-rabbitmq-server-0/rabbitmq_peer_discovery\",\"host\":\"my-rabbitmq-server-0\"},\"type\":\"Warning\"}"]
2022-11-08 14:03:44.367 [debug] <0.274.0> Response: [{error,{failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}}]
2022-11-08 14:03:44.367 [debug] <0.274.0> HTTP error {failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}
2022-11-08 14:03:44.367 [error] <0.274.0> Peer discovery returned an error: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n {inet6,[inet6],nxdomain}]}". Will retry after a delay of 500 ms, 7 retries left...
2022-11-08 14:03:44.869 [debug] <0.274.0> GET https://kubernetes.default:443/api/v1/namespaces/hz-rabbitmq/endpoints/my-rabbitmq-nodes
2022-11-08 14:03:44.873 [debug] <0.274.0> Response: {error,{failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}}
2022-11-08 14:03:44.873 [debug] <0.274.0> HTTP error {failed_connect,[{to_address,{"kubernetes.default",443}},{inet6,[inet6],nxdomain}]}
2022-11-08 14:03:44.873 [error] <0.274.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
{inet6,[inet6],nxdomain}]}
$ cat /etc/rabbitmq/rabbitmq-env.conf
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp"
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp"
$ cat /etc/rabbitmq/erl_inetrc
{inet6,true}.
OTP_VERSION 23.2.6
RABBITMQ_VERSION 3.8.12
rabbitmq_peer_discovery_aws-3.8.12.ez
rabbitmq_peer_discovery_common-3.8.12.ez
rabbitmq_peer_discovery_consul-3.8.12.ez
rabbitmq_peer_discovery_etcd-3.8.12.ez
rabbitmq_peer_discovery_k8s-3.8.12.ez
rabbitmq_peer_discovery does not work in ipv6 environment, does this need to be configured or fixed?
Michal Kuratczyk
Nov 8, 2022, 3:39:59 AM11/8/22
to rabbitm...@googlegroups.com
Hi,
I'm pretty sure we never tested RabbitMQ Kubernetes peer discovery with IPv6.
You are using an old Operator version and therefore an old RabbitMQ version (by default). RabbitMQ 3.8.x has been out of support for quite some time.
If the problem is still there with the latest Operator/RabbitMQ/Erlang, I'd suggest raising an issue on Github. Ideally with the details of how to get it set up on the Kubernetes side as well, if that's necessary.
Best,
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/d0e0e62b-05c4-4e6a-b583-771773b3ba7an%40googlegroups.com.
Michał
RabbitMQ team蓝之忘
Nov 8, 2022, 7:00:48 AM11/8/22
to rabbitmq-users
hi,
Using the latest version of operators and rabbitmq server, my validation steps are as follows:
1. Install cluster-operator 2.0.0 https://github.com/rabbitmq/cluster-operator/releases/tag/v2.0.0
2. Based on rabbitmq:3.11.2-management, add the /etc/rabbitmq/erl_inetrc file
FROM rabbitmq:3.11.2-management
RUN echo '{inet6,true}.' > /etc/rabbitmq/erl_inetrc; \
chown rabbitmq:rabbitmq /etc/rabbitmq/erl_inetrc
Suppose the image built according to the above dockerfile is rabbitmq:3.11.2-management-inetrc
3. Use the following yaml to deploy rabbitmq
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: sample
namespace: rabbitmq-system
spec:
image: rabbitmq:3.11.2-management-inetrc
replicas: 3
service:
type: ClusterIP
persistence:
storage: 2Gi
storageClassName: sc-topolvm
rabbitmq:
envConfig: |
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp"
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp"
- name: epmd
port: 4369
protocol: TCP
- name: cluster-rpc
port: 25672
protocol: TCP
- name: epmd
port: 4369
protocol: TCP
- name: cluster-rpc
port: 25672
protocol: TCP
apiVersion: v1
kind: Endpoints
metadata:
nodeName: 192.168.132.208
targetRef:
kind: Pod
nodeName: 192.168.132.209
targetRef:
kind: Pod
ports:
- name: cluster-rpc
port: 25672
protocol: TCP
- name: epmd
port: 4369
protocol: TCP
My goal is to communicate via Ipv6 between the client and rabbitmq, between the rabbitmq nodes, and between the cli tool and the rabbitmq nodes.
Using the latest version of operators and rabbitmq server, my validation steps are as follows:
1. Install cluster-operator 2.0.0 https://github.com/rabbitmq/cluster-operator/releases/tag/v2.0.0
2. Based on rabbitmq:3.11.2-management, add the /etc/rabbitmq/erl_inetrc file
FROM rabbitmq:3.11.2-management
RUN echo '{inet6,true}.' > /etc/rabbitmq/erl_inetrc; \
chown rabbitmq:rabbitmq /etc/rabbitmq/erl_inetrc
Suppose the image built according to the above dockerfile is rabbitmq:3.11.2-management-inetrc
3. Use the following yaml to deploy rabbitmq
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: sample
namespace: rabbitmq-system
spec:
image: rabbitmq:3.11.2-management-inetrc
replicas: 3
service:
type: ClusterIP
persistence:
storage: 2Gi
storageClassName: sc-topolvm
rabbitmq:
envConfig: |
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp"
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp"
4. After rabbitmq is successfully deployed, add labels to RabbitmqCluster
labels:
rabbitmq.com/pauseReconciliation: "true"
Make operators not in Reconcile RabbitmqCluster
5. Modify sample-nodes services ipFamilies to IPv6
Before modification:
spec:
clusterIP: None
clusterIPs:
- None
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
labels:
rabbitmq.com/pauseReconciliation: "true"
Make operators not in Reconcile RabbitmqCluster
5. Modify sample-nodes services ipFamilies to IPv6
Before modification:
spec:
clusterIP: None
clusterIPs:
- None
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: epmd
port: 4369
protocol: TCP
targetPort: 4369
- name: cluster-rpc
port: 25672
protocol: TCP
targetPort: 25672
publishNotReadyAddresses: true
selector:
app.kubernetes.io/name: sample
sessionAffinity: None
type: ClusterIP
after modification:
spec:
clusterIP: None
clusterIPs:
- None
internalTrafficPolicy: Cluster
ipFamilies:
- IPv6
ipFamilyPolicy: SingleStack
ports:
publishNotReadyAddresses: true
selector:
app.kubernetes.io/name: sample
sessionAffinity: None
type: ClusterIP
after modification:
spec:
clusterIP: None
clusterIPs:
- None
internalTrafficPolicy: Cluster
ipFamilies:
- IPv6
ipFamilyPolicy: SingleStack
ports:
- name: epmd
port: 4369
protocol: TCP
targetPort: 4369
- name: cluster-rpc
port: 25672
protocol: TCP
targetPort: 25672
publishNotReadyAddresses: true
selector:
app.kubernetes.io/name: sample
sessionAffinity: None
type: ClusterIP
6. Delete pods to recreate pods
We use kubeovn https://kubeovn.github.io/docs/v1.10.x/ network plugin And our k8s environment is ipv4 and ipv6 dual stack environment
So each pod is assigned two ips like this:
podIP: 10.3.26.155
podIPs:
- ip: 10.3.26.155
- ip: fd00:10:16::1a9b
When I modify the sample-nodes services ipFamilies to IPv6, the corresponding endpoints sample-nodes will use ipv6 as shown below
publishNotReadyAddresses: true
selector:
app.kubernetes.io/name: sample
sessionAffinity: None
type: ClusterIP
6. Delete pods to recreate pods
We use kubeovn https://kubeovn.github.io/docs/v1.10.x/ network plugin And our k8s environment is ipv4 and ipv6 dual stack environment
So each pod is assigned two ips like this:
podIP: 10.3.26.155
podIPs:
- ip: 10.3.26.155
- ip: fd00:10:16::1a9b
When I modify the sample-nodes services ipFamilies to IPv6, the corresponding endpoints sample-nodes will use ipv6 as shown below
apiVersion: v1
kind: Endpoints
metadata:
name: sample-nodes
namespace: rabbitmq-system
subsets:
- addresses:
- hostname: sample-server-0
ip: fd00:10:16::1a9a
nodeName: 192.168.132.210
targetRef:
kind: Pod
name: sample-server-0
namespace: rabbitmq-system
resourceVersion: "10213487"
uid: 46e21c2a-fa71-4147-b297-fbe34303e7f3
- hostname: sample-server-1
ip: fd00:10:16::1a9b
namespace: rabbitmq-system
subsets:
- addresses:
- hostname: sample-server-0
ip: fd00:10:16::1a9a
nodeName: 192.168.132.210
targetRef:
kind: Pod
name: sample-server-0
namespace: rabbitmq-system
resourceVersion: "10213487"
uid: 46e21c2a-fa71-4147-b297-fbe34303e7f3
- hostname: sample-server-1
ip: fd00:10:16::1a9b
nodeName: 192.168.132.208
targetRef:
kind: Pod
name: sample-server-1
namespace: rabbitmq-system
resourceVersion: "10213505"
uid: 631cb0e5-fa15-4375-80e0-2222202c7a42
- hostname: sample-server-2
ip: fd00:10:16::1a9c
namespace: rabbitmq-system
resourceVersion: "10213505"
uid: 631cb0e5-fa15-4375-80e0-2222202c7a42
- hostname: sample-server-2
ip: fd00:10:16::1a9c
nodeName: 192.168.132.209
targetRef:
kind: Pod
name: sample-server-2
namespace: rabbitmq-system
resourceVersion: "10213522"
uid: 8f0ac1ed-e02e-4e49-8305-a16f6b9007be
namespace: rabbitmq-system
resourceVersion: "10213522"
uid: 8f0ac1ed-e02e-4e49-8305-a16f6b9007be
ports:
- name: cluster-rpc
port: 25672
protocol: TCP
- name: epmd
port: 4369
protocol: TCP
This is the rabbit_peer_discovery_k8s plugin in the rabbitmq pod with the following error:
2022-11-08 11:53:01.929730+00:00 [info] <0.229.0> Configured peer discovery backend: rabbit_peer_discovery_k8s
2022-11-08 11:53:01.930173+00:00 [info] <0.229.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2022-11-08 11:53:01.933513+00:00 [warning] <0.323.0> Description: "Authenticity is not established by certificate path validation"
2022-11-08 11:53:01.933513+00:00 [warning] <0.323.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2022-11-08 11:53:01.933513+00:00 [warning] <0.323.0>
2022-11-08 11:53:01.940395+00:00 [error] <0.229.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2022-11-08 11:53:01.940395+00:00 [error] <0.229.0> {inet6,[inet6],nxdomain}]}
2022-11-08 11:53:01.944164+00:00 [warning] <0.324.0> Description: "Authenticity is not established by certificate path validation"
2022-11-08 11:53:01.944164+00:00 [warning] <0.324.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2022-11-08 11:53:01.944164+00:00 [warning] <0.324.0>
2022-11-08 11:53:01.951042+00:00 [error] <0.229.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n {inet6,[inet6],nxdomain}]}"
2022-11-08 11:53:31.952194+00:00 [info] <0.229.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2022-11-08 11:53:31.958161+00:00 [warning] <0.325.0> Description: "Authenticity is not established by certificate path validation"
2022-11-08 11:53:31.958161+00:00 [warning] <0.325.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2022-11-08 11:53:31.958161+00:00 [warning] <0.325.0>
2022-11-08 11:53:31.968657+00:00 [error] <0.229.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2022-11-08 11:53:31.968657+00:00 [error] <0.229.0> {inet6,[inet6],nxdomain}]}
2022-11-08 11:53:31.976923+00:00 [warning] <0.326.0> Description: "Authenticity is not established by certificate path validation"
2022-11-08 11:53:31.976923+00:00 [warning] <0.326.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2022-11-08 11:53:31.976923+00:00 [warning] <0.326.0>
The k8s version is v1.23.8
What I want to ask is if my verification steps are correct or if rabbitmq supports communication using pure ipv6
2022-11-08 11:53:01.929730+00:00 [info] <0.229.0> Configured peer discovery backend: rabbit_peer_discovery_k8s
2022-11-08 11:53:01.930173+00:00 [info] <0.229.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2022-11-08 11:53:01.933513+00:00 [warning] <0.323.0> Description: "Authenticity is not established by certificate path validation"
2022-11-08 11:53:01.933513+00:00 [warning] <0.323.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2022-11-08 11:53:01.933513+00:00 [warning] <0.323.0>
2022-11-08 11:53:01.940395+00:00 [error] <0.229.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2022-11-08 11:53:01.940395+00:00 [error] <0.229.0> {inet6,[inet6],nxdomain}]}
2022-11-08 11:53:01.944164+00:00 [warning] <0.324.0> Description: "Authenticity is not established by certificate path validation"
2022-11-08 11:53:01.944164+00:00 [warning] <0.324.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2022-11-08 11:53:01.944164+00:00 [warning] <0.324.0>
2022-11-08 11:53:01.951042+00:00 [error] <0.229.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n {inet6,[inet6],nxdomain}]}"
2022-11-08 11:53:31.952194+00:00 [info] <0.229.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2022-11-08 11:53:31.958161+00:00 [warning] <0.325.0> Description: "Authenticity is not established by certificate path validation"
2022-11-08 11:53:31.958161+00:00 [warning] <0.325.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2022-11-08 11:53:31.958161+00:00 [warning] <0.325.0>
2022-11-08 11:53:31.968657+00:00 [error] <0.229.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2022-11-08 11:53:31.968657+00:00 [error] <0.229.0> {inet6,[inet6],nxdomain}]}
2022-11-08 11:53:31.976923+00:00 [warning] <0.326.0> Description: "Authenticity is not established by certificate path validation"
2022-11-08 11:53:31.976923+00:00 [warning] <0.326.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2022-11-08 11:53:31.976923+00:00 [warning] <0.326.0>
The k8s version is v1.23.8
What I want to ask is if my verification steps are correct or if rabbitmq supports communication using pure ipv6
Luke Bakken
Nov 8, 2022, 10:39:51 AM11/8/22
to rabbitmq-users
Hello -
Yes, Erlang (and thus RabbitMQ) supports an IPv6-only environment.
Please attach all of your configuration files to your next message. Saving them all in an archive is fine.
I'm most interested in your RabbitMQ configuration and erl_inetrc file.
The {inet6,[inet6],nxdomain}]} in your output looks "fishy". It seems like DNS may not be working correctly using IPv6.
From within a running IPv6-only pod, if you can test a DNS lookup of kubernetes.default that would be great.
Thanks,
Luke
蓝之忘
Nov 9, 2022, 3:06:09 AM11/9/22
to rabbitmq-users
According to the previous verification steps, https://groups.google.com/g/rabbitmq-users/c/6RJ0XOq7Pug/m/OZyeVQKBAgAJ the following is the configuration and log information of rabbitmq:
and I can confirm that DNS is working
$ nslookup sample-nodes.rabbitmq-system
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: sample-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::2186
Name: sample-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::2185
Name: sample-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::2184
$ nslookup sample-server-0.sample-nodes.rabbitmq-system.svc.cluster.local
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: sample-server-0.sample-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::2184
$ nslookup sample-server-1.sample-nodes.rabbitmq-system.svc.cluster.local
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: sample-server-1.sample-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::2185
$ nslookup sample-server-2.sample-nodes.rabbitmq-system.svc.cluster.local
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: sample-server-2.sample-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::2186
and I can confirm that DNS is working
$ nslookup sample-nodes.rabbitmq-system
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: sample-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::2186
Name: sample-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::2185
Name: sample-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::2184
$ nslookup sample-server-0.sample-nodes.rabbitmq-system.svc.cluster.local
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: sample-server-0.sample-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::2184
$ nslookup sample-server-1.sample-nodes.rabbitmq-system.svc.cluster.local
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: sample-server-1.sample-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::2185
$ nslookup sample-server-2.sample-nodes.rabbitmq-system.svc.cluster.local
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: sample-server-2.sample-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::2186
Is there any other information needed, based on which the problem can be found
Thanks
Luke Bakken
Nov 9, 2022, 10:22:26 AM11/9/22
to rabbitmq-users
Hello -
My guess is that the issue is due to having Erlang/RabbitMQ configured to be IPv6 only, yet the DNS resolver is configured with an IPv4 address for your DNS server (10.4.0.10).
You need to configure the pod that runs Erlang/RabbitMQ so that it's DNS resolver uses an IPv6 address, or add the appropriate entries to your erl_inetrc file:
Thanks,
Luke
Luke
蓝之忘
Jan 7, 2023, 9:00:45 AM1/7/23
to rabbitmq-users
I want to verify that rabbimq works properly on ipv6 network protocol stack.
I'm k8s v1.23.8 deployed rabbimq cluster operator (https://github.com/rabbitmq/cluster-operator/releases/tag/v2.1.0),
Then deploy rabbitmq:3.10.2-management, which is the default for operator.
By default rabbitmq works properly
[middleware-develop-6852]$ kubectl -n rabbitmq-system get pods
NAME READY STATUS RESTARTS AGE
my-rabbitmq-server-0 1/1 Running 0 37m
my-rabbitmq-server-1 1/1 Running 0 37m
my-rabbitmq-server-2 1/1 Running 0 37m
I checked the rabbitmq configuration and the DNS related configuration as follows
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- ls -al /etc/rabbitmq
total 4
drwxrwxrwx 1 rabbitmq rabbitmq 20 May 20 2022 .
drwxr-xr-x 1 root root 38 Jan 7 12:13 ..
drwxrwxrwx 1 rabbitmq rabbitmq 106 Jan 7 12:13 conf.d
-rw-r--r-- 1 root root 43 May 20 2022 enabled_plugins
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- ls -al /etc/rabbitmq/conf.d
total 16
drwxrwxrwx 1 rabbitmq rabbitmq 106 Jan 7 12:13 .
drwxrwxrwx 1 rabbitmq rabbitmq 20 May 20 2022 ..
-rw-rw-r-- 1 rabbitmq rabbitmq 491 May 20 2022 10-defaults.conf
-rw-r--r-- 1 root root 459 Jan 7 12:12 10-operatorDefaults.conf
-rw-r--r-- 1 root root 96 Jan 7 12:12 11-default_user.conf
-rw-r--r-- 1 root root 51 Jan 7 12:12 90-userDefinedConfiguration.conf
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/enabled_plugins
[rabbitmq_management,rabbitmq_prometheus].
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/conf.d/10-defaults.conf
## DEFAULT SETTINGS ARE NOT MEANT TO BE TAKEN STRAIGHT INTO PRODUCTION
## see https://www.rabbitmq.com/configure.html for further information
## on configuring RabbitMQ
## allow access to the guest user from anywhere on the network
## https://www.rabbitmq.com/access-control.html#loopback-users
## https://www.rabbitmq.com/production-checklist.html#users
loopback_users.guest = false
## Send all logs to stdout/TTY. Necessary to see logs when running via
## a container
log.console = true
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/conf.d/10-operatorDefaults.conf
queue_master_locator = min-masters
disk_free_limit.absolute = 2GB
cluster_partition_handling = pause_minority
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default
cluster_formation.k8s.address_type = hostname
cluster_formation.target_cluster_size_hint = 3
cluster_name = my-rabbitmq
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/conf.d/11-default_user.conf
default_user = default_user_HHzAT0LOPmQrBZDDbtK
default_pass = rtfO1IwUgiaNHG-LPe0r1inbNlZ9WJu9
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf
total_memory_available_override_value = 1717986919
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.3.203.129 my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local my-rabbitmq-server-0
fd00:10:16::cb81 my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local my-rabbitmq-server-0
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/resolv.conf
nameserver 10.4.0.10
search rabbitmq-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- hostname
my-rabbitmq-server-0
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: 10.3.203.129
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: 10.3.203.129
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0
Server: 10.4.0.10
Address: 10.4.0.10#53
** server can't find my-rabbitmq-server-0: NXDOMAIN
command terminated with exit code 1
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
["my-rabbitmq-server-0"],
inet,4,
[{10,3,203,129}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet,4,
[{10,3,203,129}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
["my-rabbitmq-server-0"],
inet,4,
[{10,3,203,129}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet6,16,
[{64768,16,22,0,0,0,0,52097}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet6).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet6,16,
[{64768,16,22,0,0,0,0,52097}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, a).'
{ok,{dns_rec,{dns_header,2,true,query,true,false,true,false,false,0},
[{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
a,in,false}],
[{dns_rr,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
a,in,0,21,
{10,3,203,129},
undefined,[],false}],
[],[]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, a).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, a).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, aaaa).'
{ok,{dns_rec,{dns_header,5,true,query,true,false,true,false,false,0},
[{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
aaaa,in,false}],
[],
[{dns_rr,"cluster.local",soa,in,0,30,
{"ns.dns.cluster.local","hostmaster.cluster.local",
1673094992,7200,1800,86400,30},
undefined,[],false}],
[]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, aaaa).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, aaaa).'
{error,nxdomain}
##############################################################################################################################
The second verification is that the /etc/rabbitmq-erl_inetrc file is added to the image,
but is not configured to enable ipv6
[middleware-develop-6852]$ cat /etc/rabbitmq/erl_inetrc
{inet6, true}.
{resolv_conf, "/etc/resolv.conf"}.
{lookup, [dns]}.
rabbitmq will also work, and the rest of the configuration is the same as before
[middleware-develop-6852]$ kubectl -n rabbitmq-system get pods
NAME READY STATUS RESTARTS AGE
my-rabbitmq-server-0 1/1 Running 0 104s
my-rabbitmq-server-1 1/1 Running 0 104s
my-rabbitmq-server-2 1/1 Running 0 104s
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.3.203.202 my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local my-rabbitmq-server-0
fd00:10:16::cbca my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local my-rabbitmq-server-0
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/resolv.conf
nameserver 10.4.0.10
search rabbitmq-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- hostname
my-rabbitmq-server-0
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: 10.3.203.202
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: 10.3.203.202
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0
Server: 10.4.0.10
Address: 10.4.0.10#53
** server can't find my-rabbitmq-server-0: NXDOMAIN
command terminated with exit code 1
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
["my-rabbitmq-server-0"],
inet,4,
[{10,3,203,202}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet,4,
[{10,3,203,202}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
["my-rabbitmq-server-0"],
inet,4,
[{10,3,203,202}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet6,16,
[{64768,16,22,0,0,0,0,52170}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet6).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet6,16,
[{64768,16,22,0,0,0,0,52170}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, a).'
{ok,{dns_rec,{dns_header,1,true,query,true,false,true,false,false,0},
[{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
a,in,false}],
[{dns_rr,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
a,in,0,30,
{10,3,203,202},
undefined,[],false}],
[],[]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, a).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, a).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, aaaa).'
{ok,{dns_rec,{dns_header,4,true,query,true,false,true,false,false,0},
[{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
aaaa,in,false}],
[],
[{dns_rr,"cluster.local",soa,in,0,30,
{"ns.dns.cluster.local","hostmaster.cluster.local",
1673097281,7200,1800,86400,30},
undefined,[],false}],
[]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, aaaa).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, aaaa).'
{error,nxdomain}
##############################################################################################################################
The third verification is that you are adding the /etc/rabbitmq-erl_inetrc file to the image and enabling ipv6
In this case, rabbitmq does not work properly. The rabbitmq logs are as follows：
2023-01-07 13:30:08.123689+00:00 [info] <0.219.0> Configured peer discovery backend: rabbit_peer_discovery_k8s
2023-01-07 13:30:08.124245+00:00 [info] <0.219.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2023-01-07 13:30:08.127846+00:00 [warning] <0.315.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:30:08.127846+00:00 [warning] <0.315.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:30:08.127846+00:00 [warning] <0.315.0>
2023-01-07 13:30:08.185568+00:00 [error] <0.219.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2023-01-07 13:30:08.185568+00:00 [error] <0.219.0> {inet6,[inet6],nxdomain}]}
2023-01-07 13:30:08.188142+00:00 [warning] <0.316.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:30:08.188142+00:00 [warning] <0.316.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:30:08.188142+00:00 [warning] <0.316.0>
2023-01-07 13:30:08.219100+00:00 [error] <0.219.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n {inet6,[inet6],nxdomain}]}"
2023-01-07 13:30:38.220353+00:00 [info] <0.219.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2023-01-07 13:30:38.223474+00:00 [warning] <0.365.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:30:38.223474+00:00 [warning] <0.365.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:30:38.223474+00:00 [warning] <0.365.0>
2023-01-07 13:30:38.259894+00:00 [error] <0.219.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2023-01-07 13:30:38.259894+00:00 [error] <0.219.0> {inet6,[inet6],nxdomain}]}
2023-01-07 13:30:38.262612+00:00 [warning] <0.366.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:30:38.262612+00:00 [warning] <0.366.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:30:38.262612+00:00 [warning] <0.366.0>
2023-01-07 13:30:38.301716+00:00 [error] <0.219.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n {inet6,[inet6],nxdomain}]}"
2023-01-07 13:31:08.302540+00:00 [info] <0.219.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2023-01-07 13:31:08.308443+00:00 [warning] <0.415.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:31:08.308443+00:00 [warning] <0.415.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:31:08.308443+00:00 [warning] <0.415.0>
2023-01-07 13:31:08.345506+00:00 [error] <0.219.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2023-01-07 13:31:08.345506+00:00 [error] <0.219.0> {inet6,[inet6],nxdomain}]}
2023-01-07 13:31:08.350786+00:00 [warning] <0.416.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:31:08.350786+00:00 [warning] <0.416.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:31:08.350786+00:00 [warning] <0.416.0>
2023-01-07 13:31:08.387389+00:00 [error] <0.219.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n {inet6,[inet6],nxdomain}]}"
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- ls -al /etc/rabbitmq
total 12
drwxrwxrwx 1 rabbitmq rabbitmq 45 Jan 7 13:29 .
drwxr-xr-x 1 root root 38 Jan 7 13:29 ..
drwxrwxrwx 1 rabbitmq rabbitmq 106 Jan 7 13:29 conf.d
-rw-r--r-- 1 root root 43 May 20 2022 enabled_plugins
-rw-r--r-- 1 rabbitmq rabbitmq 67 Jan 7 11:05 erl_inetrc
-rw-r--r-- 1 root root 148 Jan 7 13:29 rabbitmq-env.conf
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/erl_inetrc
{inet6, true}.
{resolv_conf, "/etc/resolv.conf"}.
{lookup, [dns]}.
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/rabbitmq-env.conf
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp"
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp"
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.3.203.238 my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local my-rabbitmq-server-0
fd00:10:16::cbee my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local my-rabbitmq-server-0
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/resolv.conf
nameserver 10.4.0.10
search rabbitmq-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- hostname
my-rabbitmq-server-0
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::cbee
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::cbee
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0
Server: 10.4.0.10
Address: 10.4.0.10#53
** server can't find my-rabbitmq-server-0: NXDOMAIN
command terminated with exit code 1
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet,4,
[{127,0,0,1}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet).'
{ok,{hostent,"my-rabbitmq-server-0",[],inet,4,[{127,0,0,1}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet6,16,
[{64768,16,22,0,0,0,0,52206}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet6,16,
[{64768,16,22,0,0,0,0,52206}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet6).'
{ok,{hostent,"my-rabbitmq-server-0",[],inet6,16,[{0,0,0,0,0,0,0,1}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, a).'
{ok,{dns_rec,{dns_header,5,true,query,true,false,true,false,false,0},
[{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
a,in,false}],
[],
[{dns_rr,"cluster.local",soa,in,0,30,
{"ns.dns.cluster.local","hostmaster.cluster.local",
1673098842,7200,1800,86400,30},
undefined,[],false}],
[]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, a).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, a).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, aaaa).'
{ok,{dns_rec,{dns_header,24,true,query,true,false,true,false,false,0},
[{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
aaaa,in,false}],
[{dns_rr,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
aaaa,in,0,30,
{64768,16,22,0,0,0,0,52206},
undefined,[],false}],
[],[]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, aaaa).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, aaaa).'
{error,nxdomain}
What is the problem and how should I solve it? Or what verification do I need to do
I'm k8s v1.23.8 deployed rabbimq cluster operator (https://github.com/rabbitmq/cluster-operator/releases/tag/v2.1.0),
Then deploy rabbitmq:3.10.2-management, which is the default for operator.
By default rabbitmq works properly
[middleware-develop-6852]$ kubectl -n rabbitmq-system get pods
NAME READY STATUS RESTARTS AGE
my-rabbitmq-server-0 1/1 Running 0 37m
my-rabbitmq-server-1 1/1 Running 0 37m
my-rabbitmq-server-2 1/1 Running 0 37m
I checked the rabbitmq configuration and the DNS related configuration as follows
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- ls -al /etc/rabbitmq
total 4
drwxrwxrwx 1 rabbitmq rabbitmq 20 May 20 2022 .
drwxr-xr-x 1 root root 38 Jan 7 12:13 ..
drwxrwxrwx 1 rabbitmq rabbitmq 106 Jan 7 12:13 conf.d
-rw-r--r-- 1 root root 43 May 20 2022 enabled_plugins
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- ls -al /etc/rabbitmq/conf.d
total 16
drwxrwxrwx 1 rabbitmq rabbitmq 106 Jan 7 12:13 .
drwxrwxrwx 1 rabbitmq rabbitmq 20 May 20 2022 ..
-rw-rw-r-- 1 rabbitmq rabbitmq 491 May 20 2022 10-defaults.conf
-rw-r--r-- 1 root root 459 Jan 7 12:12 10-operatorDefaults.conf
-rw-r--r-- 1 root root 96 Jan 7 12:12 11-default_user.conf
-rw-r--r-- 1 root root 51 Jan 7 12:12 90-userDefinedConfiguration.conf
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/enabled_plugins
[rabbitmq_management,rabbitmq_prometheus].
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/conf.d/10-defaults.conf
## DEFAULT SETTINGS ARE NOT MEANT TO BE TAKEN STRAIGHT INTO PRODUCTION
## see https://www.rabbitmq.com/configure.html for further information
## on configuring RabbitMQ
## allow access to the guest user from anywhere on the network
## https://www.rabbitmq.com/access-control.html#loopback-users
## https://www.rabbitmq.com/production-checklist.html#users
loopback_users.guest = false
## Send all logs to stdout/TTY. Necessary to see logs when running via
## a container
log.console = true
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/conf.d/10-operatorDefaults.conf
queue_master_locator = min-masters
disk_free_limit.absolute = 2GB
cluster_partition_handling = pause_minority
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default
cluster_formation.k8s.address_type = hostname
cluster_formation.target_cluster_size_hint = 3
cluster_name = my-rabbitmq
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/conf.d/11-default_user.conf
default_user = default_user_HHzAT0LOPmQrBZDDbtK
default_pass = rtfO1IwUgiaNHG-LPe0r1inbNlZ9WJu9
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf
total_memory_available_override_value = 1717986919
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.3.203.129 my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local my-rabbitmq-server-0
fd00:10:16::cb81 my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local my-rabbitmq-server-0
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/resolv.conf
nameserver 10.4.0.10
search rabbitmq-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- hostname
my-rabbitmq-server-0
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: 10.3.203.129
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: 10.3.203.129
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0
Server: 10.4.0.10
Address: 10.4.0.10#53
** server can't find my-rabbitmq-server-0: NXDOMAIN
command terminated with exit code 1
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
["my-rabbitmq-server-0"],
inet,4,
[{10,3,203,129}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet,4,
[{10,3,203,129}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
["my-rabbitmq-server-0"],
inet,4,
[{10,3,203,129}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet6,16,
[{64768,16,22,0,0,0,0,52097}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet6).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet6,16,
[{64768,16,22,0,0,0,0,52097}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, a).'
{ok,{dns_rec,{dns_header,2,true,query,true,false,true,false,false,0},
[{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
a,in,false}],
[{dns_rr,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
a,in,0,21,
{10,3,203,129},
undefined,[],false}],
[],[]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, a).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, a).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, aaaa).'
{ok,{dns_rec,{dns_header,5,true,query,true,false,true,false,false,0},
[{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
aaaa,in,false}],
[],
[{dns_rr,"cluster.local",soa,in,0,30,
{"ns.dns.cluster.local","hostmaster.cluster.local",
1673094992,7200,1800,86400,30},
undefined,[],false}],
[]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, aaaa).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, aaaa).'
{error,nxdomain}
##############################################################################################################################
The second verification is that the /etc/rabbitmq-erl_inetrc file is added to the image,
but is not configured to enable ipv6
[middleware-develop-6852]$ cat /etc/rabbitmq/erl_inetrc
{inet6, true}.
{resolv_conf, "/etc/resolv.conf"}.
{lookup, [dns]}.
rabbitmq will also work, and the rest of the configuration is the same as before
[middleware-develop-6852]$ kubectl -n rabbitmq-system get pods
NAME READY STATUS RESTARTS AGE
my-rabbitmq-server-0 1/1 Running 0 104s
my-rabbitmq-server-1 1/1 Running 0 104s
my-rabbitmq-server-2 1/1 Running 0 104s
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.3.203.202 my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local my-rabbitmq-server-0
fd00:10:16::cbca my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local my-rabbitmq-server-0
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/resolv.conf
nameserver 10.4.0.10
search rabbitmq-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- hostname
my-rabbitmq-server-0
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: 10.3.203.202
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: 10.3.203.202
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0
Server: 10.4.0.10
Address: 10.4.0.10#53
** server can't find my-rabbitmq-server-0: NXDOMAIN
command terminated with exit code 1
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
["my-rabbitmq-server-0"],
inet,4,
[{10,3,203,202}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet,4,
[{10,3,203,202}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
["my-rabbitmq-server-0"],
inet,4,
[{10,3,203,202}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet6,16,
[{64768,16,22,0,0,0,0,52170}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet6).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet6,16,
[{64768,16,22,0,0,0,0,52170}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, a).'
{ok,{dns_rec,{dns_header,1,true,query,true,false,true,false,false,0},
[{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
a,in,false}],
[{dns_rr,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
a,in,0,30,
{10,3,203,202},
undefined,[],false}],
[],[]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, a).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, a).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, aaaa).'
{ok,{dns_rec,{dns_header,4,true,query,true,false,true,false,false,0},
[{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
aaaa,in,false}],
[],
[{dns_rr,"cluster.local",soa,in,0,30,
{"ns.dns.cluster.local","hostmaster.cluster.local",
1673097281,7200,1800,86400,30},
undefined,[],false}],
[]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, aaaa).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, aaaa).'
{error,nxdomain}
##############################################################################################################################
The third verification is that you are adding the /etc/rabbitmq-erl_inetrc file to the image and enabling ipv6
In this case, rabbitmq does not work properly. The rabbitmq logs are as follows：
2023-01-07 13:30:08.123689+00:00 [info] <0.219.0> Configured peer discovery backend: rabbit_peer_discovery_k8s
2023-01-07 13:30:08.124245+00:00 [info] <0.219.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2023-01-07 13:30:08.127846+00:00 [warning] <0.315.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:30:08.127846+00:00 [warning] <0.315.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:30:08.127846+00:00 [warning] <0.315.0>
2023-01-07 13:30:08.185568+00:00 [error] <0.219.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2023-01-07 13:30:08.185568+00:00 [error] <0.219.0> {inet6,[inet6],nxdomain}]}
2023-01-07 13:30:08.188142+00:00 [warning] <0.316.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:30:08.188142+00:00 [warning] <0.316.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:30:08.188142+00:00 [warning] <0.316.0>
2023-01-07 13:30:08.219100+00:00 [error] <0.219.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n {inet6,[inet6],nxdomain}]}"
2023-01-07 13:30:38.220353+00:00 [info] <0.219.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2023-01-07 13:30:38.223474+00:00 [warning] <0.365.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:30:38.223474+00:00 [warning] <0.365.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:30:38.223474+00:00 [warning] <0.365.0>
2023-01-07 13:30:38.259894+00:00 [error] <0.219.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2023-01-07 13:30:38.259894+00:00 [error] <0.219.0> {inet6,[inet6],nxdomain}]}
2023-01-07 13:30:38.262612+00:00 [warning] <0.366.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:30:38.262612+00:00 [warning] <0.366.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:30:38.262612+00:00 [warning] <0.366.0>
2023-01-07 13:30:38.301716+00:00 [error] <0.219.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n {inet6,[inet6],nxdomain}]}"
2023-01-07 13:31:08.302540+00:00 [info] <0.219.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2023-01-07 13:31:08.308443+00:00 [warning] <0.415.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:31:08.308443+00:00 [warning] <0.415.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:31:08.308443+00:00 [warning] <0.415.0>
2023-01-07 13:31:08.345506+00:00 [error] <0.219.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
2023-01-07 13:31:08.345506+00:00 [error] <0.219.0> {inet6,[inet6],nxdomain}]}
2023-01-07 13:31:08.350786+00:00 [warning] <0.416.0> Description: "Authenticity is not established by certificate path validation"
2023-01-07 13:31:08.350786+00:00 [warning] <0.416.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-01-07 13:31:08.350786+00:00 [warning] <0.416.0>
2023-01-07 13:31:08.387389+00:00 [error] <0.219.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n {inet6,[inet6],nxdomain}]}"
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- ls -al /etc/rabbitmq
total 12
drwxrwxrwx 1 rabbitmq rabbitmq 45 Jan 7 13:29 .
drwxr-xr-x 1 root root 38 Jan 7 13:29 ..
drwxrwxrwx 1 rabbitmq rabbitmq 106 Jan 7 13:29 conf.d
-rw-r--r-- 1 root root 43 May 20 2022 enabled_plugins
-rw-r--r-- 1 rabbitmq rabbitmq 67 Jan 7 11:05 erl_inetrc
-rw-r--r-- 1 root root 148 Jan 7 13:29 rabbitmq-env.conf
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/erl_inetrc
{inet6, true}.
{resolv_conf, "/etc/resolv.conf"}.
{lookup, [dns]}.
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/rabbitmq/rabbitmq-env.conf
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp"
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp"
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.3.203.238 my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local my-rabbitmq-server-0
fd00:10:16::cbee my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local my-rabbitmq-server-0
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- cat /etc/resolv.conf
nameserver 10.4.0.10
search rabbitmq-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- hostname
my-rabbitmq-server-0
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::cbee
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0.my-rabbitmq-nodes
Server: 10.4.0.10
Address: 10.4.0.10#53
Name: my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local
Address: fd00:10:16::cbee
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- nslookup my-rabbitmq-server-0
Server: 10.4.0.10
Address: 10.4.0.10#53
** server can't find my-rabbitmq-server-0: NXDOMAIN
command terminated with exit code 1
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet,4,
[{127,0,0,1}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet).'
{ok,{hostent,"my-rabbitmq-server-0",[],inet,4,[{127,0,0,1}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet6,16,
[{64768,16,22,0,0,0,0,52206}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0.my-rabbitmq-nodes", inet6).'
{ok,{hostent,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
[],inet6,16,
[{64768,16,22,0,0,0,0,52206}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet:gethostbyname("my-rabbitmq-server-0", inet6).'
{ok,{hostent,"my-rabbitmq-server-0",[],inet6,16,[{0,0,0,0,0,0,0,1}]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, a).'
{ok,{dns_rec,{dns_header,5,true,query,true,false,true,false,false,0},
[{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
a,in,false}],
[],
[{dns_rr,"cluster.local",soa,in,0,30,
{"ns.dns.cluster.local","hostmaster.cluster.local",
1673098842,7200,1800,86400,30},
undefined,[],false}],
[]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, a).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, a).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local", in, aaaa).'
{ok,{dns_rec,{dns_header,24,true,query,true,false,true,false,false,0},
[{dns_query,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
aaaa,in,false}],
[{dns_rr,"my-rabbitmq-server-0.my-rabbitmq-nodes.rabbitmq-system.svc.cluster.local",
aaaa,in,0,30,
{64768,16,22,0,0,0,0,52206},
undefined,[],false}],
[],[]}}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0.my-rabbitmq-nodes", in, aaaa).'
{error,nxdomain}
[middleware-develop-6852]$ kubectl -n rabbitmq-system exec -ti my-rabbitmq-server-0 -- rabbitmqctl eval 'inet_res:resolve("my-rabbitmq-server-0", in, aaaa).'
{error,nxdomain}
What is the problem and how should I solve it? Or what verification do I need to do
Luke Bakken
Jan 7, 2023, 9:46:52 AM1/7/23
to rabbitmq-users
Please continue the discussion here: https://github.com/rabbitmq/rabbitmq-server/discussions/6820
Reply all
Reply to author
Forward
0 new messages