RabbitMQ in Kubernetes

[Andrew Smith]

Hi

I'm trying to set up a rabbitmq cluster inside my kubernetes cluster and running in to some difficulties. I was wondering if anyone had any experience setting one up.

So far my problem is nodenames / hostnames. When I do "join_cluster" I need to give a parameter that resolves with DNS. 

To achieve this in Kubernetes I am using the DNS addon, this allows services to resolve to IPs. I then define a service that points to my RabbitMQ pod.

However - with this setup I have a DNS name "rabbitmq.default.cluster.local" that resolves correctly but the actual container's hostname is something else (rabbitmq-dfjs1).

So when I try and add it to the cluster I get this error:

attempted to contact: ['rab...@rabbitmqmaster.default.cluster.local']

rab...@rabbitmqmaster.default.cluster.local:

  * connected to epmd (port 4369) on rabbitmqmaster.default.cluster.local

  * epmd reports node 'rabbit' running on port 25672

  * TCP connection succeeded but Erlang distribution failed

  * suggestion: hostname mismatch?

  * suggestion: is the cookie set correctly?

  * suggestion: is the Erlang distribution using TLS?

(The cookies are identical, TLS is disabled)

Which I suspect is because my container running rabbitmq has the hostname "rabbitmq-dfjs1", not "rabbitmq.default.cluster.local".

Have any kubernetes users come across this issue and found a decent solution?

Thanks

Andy

[Michael Klishin]

 On 31 October 2015 at 01:38:24, Andrew Smith (m...@andrewmichaelsmith.com) wrote:
> Which I suspect is because my container running rabbitmq has
> the hostname "rabbitmq-dfjs1", not "rabbitmq.default.cluster.local".

You can specify any node name to rabbitmqctl using the -n flag.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Andrew Smith]

Aha, I had totally missed that flag.

However, when specifying the nodename I get this error:

Clustering node 'rabbit@rabbitmqmaster-cgimp' with 'rabbitmqmaster.default.cluster.local' ...

Error: unable to connect to node 'rabbit@rabbitmqmaster-cgimp': nodedown

DIAGNOSTICS

===========

attempted to contact: ['rabbit@rabbitmqmaster-cgimp']

rabbit@rabbitmqmaster-cgimp:

  * unable to connect to epmd (port 4369) on rabbitmqmaster-cgimp: nxdomain (non-existing domain)

So it looks like it's trying to resolve the nodename locally, which fails because only the host "rabbitmqmaster-cgimp" is aware of that hostname.

Am I missing something here?

[Andrew Smith]

Re-reading this I think I misunderstood. I was giving the remote node name to -n but I think that -n is for the local node name. I don't think my problem comes from talking to the local node.

I see 2 possible solutions and neither work.

1. Leave the $HOSTNAME/$RABBITMQ_NODENAME alone and try and connect based on a DNS host that points at the node but the node does not know about.

As described above, this gives me this error:

  * epmd reports node 'rabbit' running on port 25672

  * TCP connection succeeded but Erlang distribution failed

  * suggestion: hostname mismatch?

I suspect this is because we're saying to the node "Hey, you're hostname is a.b.c" but that is not the node's hostname.

2. Set $HOSTNAME/rabbit@$RABBITMQ_NODENAME to host that resolves to that node.

I also have to set RABBITMQ_USE_LONGNAME

Rabbit fails to start, I suspect (but have not confirmed) that this is because $HOSTNAME does not match up with /etc/hosts. This is because in Kubernetes I cannot set the hostname of the running container.

Error description:

   {could_not_start,rabbit,

       {error,

           {{shutdown,

                {failed_to_start_child,rabbit_epmd_monitor,

                    {{badmatch,closed},

                     [{rabbit_epmd_monitor,init,1,[]},

                      {gen_server,init_it,6,

                          [{file,"gen_server.erl"},{line,304}]},

                      {proc_lib,init_p_do_apply,3,

                          [{file,"proc_lib.erl"},{line,239}]}]}}},

            {child,undefined,rabbit_epmd_monitor_sup,

                {rabbit_restartable_sup,start_link,

                    [rabbit_epmd_monitor_sup,

                     {rabbit_epmd_monitor,start_link,[]},

                     false]},

                transient,infinity,supervisor,

                [rabbit_restartable_sup]}}}}

I'm a bit stumped here, it seems quite tricky in Kubernetes to get a container with a DNS resolvable hostname that matches what it believes it's hostname to be. And I think that is a requirement for rabbit to start/cluster successfully.

[Michael Klishin]

On 2 November 2015 at 14:15:21, Andrew Smith (m...@andrewmichaelsmith.com) wrote:
> I'm a bit stumped here, it seems quite tricky in Kubernetes to
> get a container with a DNS resolvable hostname that matches what
> it believes it's hostname to be. And I think that is a requirement
> for rabbit to start/cluster successfully.

Resolvable hostnames are, it doesn't have to be via DNS.

May I ask why you want to run RabbitMQ under Docker and Kubernetes in the first place?
I personally see no real benefit in running data services inside containers.

[Michael Klishin]

On 2 November 2015 at 14:15:21, Andrew Smith (m...@andrewmichaelsmith.com) wrote:

> in Kubernetes I cannot set the hostname of the running container.

then you can try retrieving effective hostname with $(hostname -f) or similar
and using those values in RABBITMQ_NODENAME, rabbitmqctl -n, and so on. 

[Andrew Smith]

But if I take $(hostname -f) this hostname is only resolvable locally - so I still can't give it as a parameter to join_cluster?

When you say it doesn't have to be DNS, what else is there? The parameter I give to join_cluster can be resolved in another way?

As for why - there's a few advantages for us (easy to have identical version+config in dev as in production) but the main reason is just: this is how we deploy applications, we'd rather not have to special case for rabbit.

[Michael Klishin]

On 2 November 2015 at 14:45:33, Andrew Smith (m...@andrewmichaelsmith.com) wrote:
> But if I take $(hostname -f) this hostname is only resolvable
> locally - so I still can't give it as a parameter to join_cluster?

So you want hostnames to be resolvable across nodes. Which is exactly why DNS exists.

> When you say it doesn't have to be DNS, what else is there? The parameter
> I give to join_cluster can be resolved in another way?

/etc/hosts. ERL_INETRC: http://www.erlang.org/doc/apps/erts/inet_cfg.html. 

[Andrew Smith]

The problem I'm facing is this: In kubernetes it is not possible for me to set the hostname of the container. 

I can create a DNS hostname that will always resolve to the I.P. address of that container but the container will not know what that hostname is.

I did not know about ERL_INETRC, from the looks of it I can set the domain, which I think will let me achieve what I want to (option 2 + erl_inetrc). Thanks I'll give that a shot.

[Michael Klishin]

On 2 November 2015 at 15:05:55, Andrew Smith (m...@andrewmichaelsmith.com) wrote:
> The problem I'm facing is this: In kubernetes it is not possible
> for me to set the hostname of the container.
>
> I can create a DNS hostname that will always resolve to the I.P.
> address of that container but the container will not know what
> that hostname is.
>
> I did not know about ERL_INETRC, from the looks of it I can set the
> domain, which I think will let me achieve what I want to (option
> 2 + erl_inetrc). Thanks I'll give that a shot.

Take a look at https://github.com/aweber/rabbitmq-autocluster.

[Andrew Smith]

Do you know if this solves my problem - the hostname pointing at the node not being the hostname of the node? 

I'm not clear but this just looks like a way to automatically run "join_cluster host".

[Michael Klishin]

On 2 November 2015 at 17:42:47, Andrew Smith (m...@andrewmichaelsmith.com) wrote:
> Do you know if this solves my problem - the hostname pointing
> at the node not being the hostname of the node?
>
> I'm not clear but this just looks like a way to automatically run
> "join_cluster host".

It is a way to discover existing cluster members before joining them,
using a variety of mechanisms.

It cannot magically make hostnames match what they are but hostnames
reported by nodes themselves without overridden NODENAME should be correct. 

[Andrew Smith]

I think that's exactly my problem, in Kubernetes the nodename reported by a node will not resolve by DNS. 

So I think I need to fix this elsewhere.

[Martin Bergman]

Hello Andrew,

I am currently looking into setting this up myself. Have you had any luck in resolving these issues?

Regards,
Martin Bergman

[nenggang pan]

hi Andrew 

have you solved this problem? I met exactly the same issue. if you setup the cluster successfully, can you share some information with me?

[Ifebunmi Ogidan]

Hi Andrew, I'm stuck with a similar issue setting up rabbit clusters on Kubernetes. Please, have you had any luck on this?

[Pon Umapathy Kailash S]

Hi,

 You can use the annotation "pod.beta.kubernetes.io/hostname" to set the hostname in kubernetes to a fixed value. This works for me, btw.

Replace {{itemid}} with the node number, for example. 

Deployment spec:

--------------------------

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

  name: mq{{ itemid }}

spec:

  replicas: 1

  template:

    metadata:

      annotations:

        "pod.beta.kubernetes.io/hostname": mq{{ itemid }}

      labels:

        app: mq{{ itemid }}

        microservice: mq{{ itemid }}

        tier: infra

    spec:

      containers:

      - name: mq{{ itemid }}

        image: xxxx

..

..

Service Spec

-----------------

apiVersion: v1

kind: Service

metadata:

  name: mq{{ itemid }}

  annotations:

    "pod.beta.kubernetes.io/hostname": mq{{ itemid }}

  labels:

    app: mq{{ itemid }}

..

..

Regards,

Umapathy

[Ifebunmi Ogidan]

Thanks. I actually found a resource online which is pretty straightforward and worked for me.

https://hub.docker.com/r/kuberstack/kubernetes-rabbitmq-autocluster/

but you need to have some mechanisms (aws, k8s, dns, etc.) as backend. I used dns.

[Michael Klishin]

The "mechanism" is a pretty well known plugin, https://github.com/rabbitmq/rabbitmq-autocluster.

3.7.0 will include most of its ideas in the core, with

https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s to target Kubernetes specifically.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.