did management plugin support pure ipv6 network?

[Tao]

Hi,

After updating rabbitmq_peer_discovery_k8s plugin, now I could deploy rabbitmq cluster on  k8s which ONLY have ipv6 network.

but I failed to access the management api.

[root]# curl -v -6  -i -u admin:admin http://zt2-crmq-ext.default.svc.cluster.local:15672/api/vhosts
* About to connect() to zt2-crmq-ext.default.svc.cluster.local port 15672 (#0)
*   Trying fd01:abcd::2b81...
* Connection refused
* Failed connect to zt2-crmq-ext.default.svc.cluster.local:15672; Connection refused
* Closing connection 0
curl: (7) Failed connect to zt2-crmq-ext.default.svc.cluster.local:15672; Connection refused

inside docker:

bash-4.2$ more /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
2002:db8:1235:0:6d44:2cb8:2c7b:6348     zt2-crmq-0.zt2-crmq.default.svc.cluster.local   zt2-crmq-0

bash-4.2$ curl -6 -i -u admin:admin http://ip6-localhost:15672/api/vhosts
curl: (7) Failed connect to ip6-localhost:15672; Connection refused

bash-4.2$ curl -4 -i -u admin:admin http://localhost:15672/api/vhosts
HTTP/1.1 200 OK
cache-control: no-cache
content-length: 78
content-security-policy: default-src 'self'
content-type: application/json
date: Mon, 28 Oct 2019 02:46:31 GMT
server: Cowboy
vary: accept, accept-encoding, origin

[{"cluster_state":{"rabbit@zt2-crmq-0":"running"},"name":"/","tracing":false}]bash-4.2$

bash-4.2$ curl -v -6 -i -u admin:admin http://localhost:15672/api/vhosts
* About to connect() to localhost port 15672 (#0)
*   Trying ::1...
* Connection refused
* Failed connect to localhost:15672; Connection refused

B.R,

Tao

[Wesley Peng]

Hi

on 2019/10/28 11:05, Tao wrote:
> After updating rabbitmq_peer_discovery_k8s plugin, now I could deploy
> rabbitmq cluster on  k8s which ONLY have ipv6 network.
> but I failed to access the management api.

RabbitMQ development just uses the standard system level libraries for
TCP/IP stack. you may dig into the system output and error to figure out
more details for this kind of issue.

regards.

[Gabriele Santomaggio]

If you are using this version: https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/55#issuecomment-543240281

yes, it works only in ipv6, but it was just a test.

you should use the last rabbitmq with the fixes [1], [2]

1-https://github.com/rabbitmq/rabbitmq-peer-discovery-common/pull/11

2-https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/pull/56

-

Gabriele Santomaggio 

[Luke Bakken]

Hi Tao,

"Connection refused" means that the management plugin isn't enabled or it's not configured to listen on an IPv6 address.

Could you please attach (in a file, not pasted in the message) the output of rabbitmqctl environment to your next response?

Thanks,

Luke

[Tao]

Hi,

API could be visited from docker shell, management have enabled and listened on ipv4 port.

I don't know any option management plugin support to listen on only ipv6 interface.

I'm afraid simply bind to a interface only listening on ipv4.

B.R,

Tao

[Luke Bakken]

Hi Tao,

Thanks for that information. From within docker, could you please run netstat -an ?

[Tao]

Hi,

There is no such tools within docker. I'll create a new image to test that.

[Tao]

rabbitmq didn't listen on tcp6 15672 port.
bash-4.2$ netstat -nat
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:15672           0.0.0.0:*               LISTEN
tcp6       0      0 :::5672                 :::*                    LISTEN
tcp6       0      0 :::25672                :::*                    LISTEN
tcp6       0      0 :::4369                 :::*                    LISTEN
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:33007 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:53483 TIME_WAIT
tcp6       0      0 ::1:50938               ::1:4369                TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:55029 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:60273 TIME_WAIT
tcp6       0      0 ::1:38037               ::1:4369                TIME_WAIT
tcp6       0      0 ::1:59000               ::1:4369                TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:42959 TIME_WAIT
tcp6       0      0 ::1:38901               ::1:4369                TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:57057 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:44306 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:46156 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:33986 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:32985 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:49887 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:54600 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:50246 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:41940 TIME_WAIT
tcp6       0      0 ::1:40390               ::1:4369                TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:50119 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 ::1:48527               ::1:4369                TIME_WAIT
tcp6       0      0 ::1:51131               ::1:4369                ESTABLISHED
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:48338 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:56367 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 ::1:50335               ::1:4369                TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:60530 TIME_WAIT
tcp6       0      0 ::1:48643               ::1:4369                TIME_WAIT
tcp6       0      0 ::1:35276               ::1:4369                TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:41906 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 ::1:44615               ::1:4369                TIME_WAIT
tcp6       0      0 ::1:53120               ::1:4369                TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:59770 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:33994 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:47889 TIME_WAIT
tcp6       0      0 ::1:4369                ::1:51131               ESTABLISHED
tcp6       0      0 2002:db8:1235:0:1:33720 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 ::1:44066               ::1:4369                TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:50178 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:34020 TIME_WAIT
tcp6       0      0 ::1:38073               ::1:4369                TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:39419 fd01:abcd::1:443        ESTABLISHED
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:36959 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:36372 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:57408 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:60637 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 ::1:36682               ::1:4369                TIME_WAIT
tcp6       0      0 ::1:43658               ::1:4369                TIME_WAIT
tcp6       0      0 2002:db8:1235:0:1:33460 2002:db8:1235:0:1:25672 TIME_WAIT
tcp6       0      0 2002:db8:1235:0:14:4369 2002:db8:1235:0:1:52971 TIME_WAIT

[Luke Bakken]

Hi Tao,

Yes, that is the problem. Please add these lines to your rabbitmq.conf file:

management.tcp.port = 15672

management.tcp.ip   = ::

If you are using the rabbitmq.config (note the .config extension) file, this is what you need to have in your rabbitmq_management section:

    {rabbitmq_management, [

        {listener, [

            {port, 15672},

            {ip, {0, 0, 0, 0, 0, 0, 0, 0}}

        ]}

    ]}

[Tao]

the management plugin failed to start.

You could find the detail in the attachment.

Error description:
    init:do_boot/3
    init:start_em/1
    rabbit:start_it/1 line 491
    rabbit:broker_start/1 line 367
    rabbit:start_loaded_apps/2 line 592
    app_utils:manage_applications/6 line 126
    lists:foldl/3 line 1263
    rabbit:'-handle_app_error/1-fun-0-'/3 line 713

[Luke Bakken]

Tao -

Please attach your complete RabbitMQ configuration file to your next response.

I tested the settings I provided using RabbitMQ 3.8.0 and they worked fine. Please let us know what version of RabbitMQ and Erlang you are using.

Thanks,

Luke

[Tao]

 Hi Luke,

Thanks for your reply.

I found that port are declared in both rabbitmq.conf and advanced.config.

after I removed the port configuration in advanced.config.  it worked.

so are configuration NOT allowed to declare in different configuration file? even the value is the same?

  rabbitmq.conf: |-
    log.default.level = debug
    listeners.tcp.default=5672

    management.tcp.port = 15672
    management.tcp.ip   = ::

    ## Clustering
    cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.k8s.address_type = hostname
    cluster_partition_handling = autoheal
    ## queue master locator
    queue_master_locator=min-masters
    ## enable guest user
    loopback_users.guest = false

  advanced.config: |-
    [
      {rabbitmq_management,
        [{listener, [{port,     15672}
                    ]}
        ]}
    ].

B.R,

Tao

[Luke Bakken]

Hi Tao,

If you use both rabbitmq.conf and advanced.config and configure the same thing, advanced.config will take precedence.

I've opened the following issue to address this behavior -

https://github.com/rabbitmq/rabbitmq-server/issues/2157

Thanks,

Luke