rabbitmq 3.7.18 cluster failed to startup in the pure ipv6 environment

[zou tao]

Hi,

I am currently trying rabbitmq ( rabbitmq_3.7.18-1.el7_erlang_22.0.7-1.el7) in k8s.

the question is similiar as https://groups.google.com/forum/#!msg/rabbitmq-users/LAJASCIMQbU/RB-I1duqCAAJ.

I followed the https://www.rabbitmq.com/networking.html#distribution-ipv6

My parameter(in helm chart):

   environment: |-
      RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+A 128 -kernel inetrc '/etc/rabbitmq/erl_inetrc'  -proto_dist inet6_tcp"
      RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp "

  erl_inetrc: |-
    {inet6, true}.

and I found error in log:

2019-10-15 07:33:55.000 [info] <0.238.0> Peer discovery backend does not support locking, falling back to randomized delay
2019-10-15 07:33:55.000 [info] <0.238.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized start
up delay.
2019-10-15 07:33:55.000 [debug] <0.238.0> GET https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/tazou/endpoints/zt4-crmq
2019-10-15 07:33:55.015 [debug] <0.238.0> Response: {error,{failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},{inet,[inet]
,nxdomain}]}}
2019-10-15 07:33:55.015 [debug] <0.238.0> HTTP Error {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},{inet,[inet],nxdom
ain}]}
2019-10-15 07:33:55.015 [info] <0.238.0> Failed to get nodes from k8s - {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}}
,
                 {inet,[inet],nxdomain}]}
2019-10-15 07:33:55.016 [error] <0.237.0> CRASH REPORT Process <0.237.0> with 0 neighbours exited with reason: no case clause matching {error,"{fa
iled_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n                 {inet,[inet],nxdomain}]}"} in rabbit_mnesia:init_from
_config/0 line 167 in application_master:init/4 line 138
2019-10-15 07:33:55.016 [info] <0.43.0> Application rabbit exited with reason: no case clause matching {error,"{failed_connect,[{to_address,{\"kub
ernetes.default.svc.cluster.local\",443}},\n                 {inet,[inet],nxdomain}]}"} in rabbit_mnesia:init_from_config/0 line 167

when i query in k8s

# nslookup -type=AAAA kubernetes.default.svc.cluster.local
Server:         2019:282:4000:2001::6
Address:        2019:282:4000:2001::6#53

kubernetes.default.svc.cluster.local    has AAAA address fd01:abcd::1

any suggestion? is it erlang setting or something else?

B.R,

Tao

[Wesley Peng]

on 2019/10/15 16:16, zou tao wrote:
> when i query in k8s
> # nslookup -type=AAAA kubernetes.default.svc.cluster.local
> Server:         2019:282:4000:2001::6
> Address:        2019:282:4000:2001::6#53
>
> kubernetes.default.svc.cluster.local    has AAAA address fd01:abcd::1

Does it have an IPv4 addr as well?

regards

[zou tao]

No. I can't resolve its ipv4 address.

[root@bcmt-sandbox2-3c3w2e-1909-1-cs-01 cloud-user]# nslookup  kubernetes.default.svc.cluster.local

Server:         2019:282:4000:2001::6
Address:        2019:282:4000:2001::6#53

*** Can't find kubernetes.default.svc.cluster.local: No answer

在 2019年10月15日星期二 UTC+8下午4:30:01，Wesley Peng写道：

[Wesley Peng]

on 2019/10/16 8:41, zou tao wrote:
> No. I can't resolve its ipv4 address.
> [root@bcmt-sandbox2-3c3w2e-1909-1-cs-01 cloud-user]# nslookup
> kubernetes.default.svc.cluster.local
> Server:         2019:282:4000:2001::6
> Address:        2019:282:4000:2001::6#53
>
> *** Can't find kubernetes.default.svc.cluster.local: No answer

This is maybe the system bug for now. but can you setup IPV4 resolving
as well?

regards.

[zou tao]

No, it needs to test it under pure ipv6 stack.

there is no ipv4 address available.

what's do you mean system bug here now?

for erlang or for k8s?

B.R,

Tao

在 2019年10月16日星期三 UTC+8上午9:13:20，Wesley Peng写道：

[zou tao]

did you mean rabbit_peer_discovery_k8s only support dual ipv6/ipv4 stack?

did it support ipv6 only ?

在 2019年10月16日星期三 UTC+8上午9:13:20，Wesley Peng写道：

[Wesley Peng]

on 2019/10/16 13:26, zou tao wrote:
> what's do you mean system bug here now?
> for erlang or for k8s?

What's the erlang version you are using with?

[zou tao]

rabbitmq_3.7.18-1.el7_erlang_22.0.7-1.el7

在 2019年10月16日星期三 UTC+8下午2:09:56，Wesley Peng写道：

[Wesley Peng]

Hi

on 2019/10/16 14:15, zou tao wrote:
> rabbitmq_3.7.18-1.el7_erlang_22.0.7-1.el7

I am not sure, but you could read out the full thread below:
http://erlang.org/pipermail/erlang-questions/2010-August/052985.html

regards.

[Tao]

[root@sandbox]# kubectl exec -ti zt4-crmq-0 rabbitmqctl eval 'inet:gethostbyname("kubernetes.default.svc.cluster.local").'

{error,nxdomain}

[root@sandbox]#  kubectl exec -ti zt4-crmq-0 rabbitmqctl eval 'inet_res:resolve("kubernetes.default.svc.cluster.local", in, aaaa).'
{ok,{dns_rec,{dns_header,1,true,query,true,false,true,true,false,0},
             [{dns_query,"kubernetes.default.svc.cluster.local",aaaa,in}],
             [{dns_rr,"kubernetes.default.svc.cluster.local",aaaa,in,0,5,
                      {64769,43981,0,0,0,0,0,1},
                      undefined,[],false}],
             [],[]}}

I don't know when rabbit_peer_discovery_k8s call httpc:request() to send the request, which function will be invoked to query ip address of kubernetes.default.svc.cluster.local.

the second call worked but the first one failed.

[Wesley Peng]

Hi

on 2019/10/16 15:13, Tao wrote:
> I don't know when rabbit_peer_discovery_k8s call httpc:request() to send
> the request, which function will be invoked to query ip address of
> kubernetes.default.svc.cluster.local.

gethostbyname works for ipv4 only.
https://stackoverflow.com/questions/2598965/gethostbyname-replacement-for-ipv6-addresses

regards.

[Tao]

Backend functions used by inet:gethostbyname/1,2.

This function uses resolver option search just like getbyname/2,3.

If resolver option inet6 is true, an IPv6 address is looked up.

http://erlang.org/doc/man/inet_res.html#gethostbyname-1

for inet package, it seems that gethostbyname support ipv6

[Wesley Peng]

on 2019/10/16 15:43, Tao wrote:
> Backend functions used by inet:gethostbyname/1,2

> <http://erlang.org/doc/man/inet.html#gethostbyname-1>.

>
> This function uses resolver option search just like getbyname/2,3

> <http://erlang.org/doc/man/inet_res.html#getbyname-2>.

>
> If resolver option inet6 is true, an IPv6 address is looked up.
>

Thanks for pointing out that.
So, for your question, what's the conslusion?

regards.

[Tao]

[root]# kubectl exec -ti zt5-crmq-0 rabbitmqctl eval 'inet:gethostbyname("kubernetes.default.svc.cluster.local", inet6).'
{ok,{hostent,"kubernetes.default.svc.cluster.local",[],inet6,16,
             [{64769,43981,0,0,0,0,0,1}]}}

I don't know why httpc:request get response nxdomain. the low layer inet could work if parameter is right.

I don't the detail of httpc lib.

is there other parameter I could set in erl_inetrc?

在 2019年10月16日星期三 UTC+8下午3:47:50，Wesley Peng写道：

[Luke Bakken]

Hi,

Please see this discussion: https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/55#issuecomment-543228426

Thanks,

Luke