OCF resource agent

[ni...@nine.ch]

Heya,
we have a 2 node rabbitmq cluster with mirrored queues in use. Yesterday I wanted to configure pacemaker to switch a virtual IP between the 2 nodes. I used the following pacemaker config:

"""
primitive res_rabbitmq_ip_internal ocf:heartbeat:IPaddr2 \
    params ip="192.168.1.3" nic="bond1" cidr_netmask="32" \
    op monitor interval="30s"
primitive res_rabbitmq_server ocf:rabbitmq:rabbitmq-server \
    op monitor interval="20s" timeout="10s"
clone clone_rabbitmq res_rabbitmq_server \
    meta clone-node-max="1" target-role="Started"
"""

Sadly pacemaker was not able to start the resources on both hosts. After a bit of investigation I saw that the node name in the OCF resource agent is set to "rabbit@localhost" by default. This prevents the "rabbitmqctl wait" command to finish successfully. After I changed the default to "rabbit@$(hostname -s)" it was working. Shouldn't that be the default in the resource agent? When using a rabbitmq/pacemaker/drbd setup the nodename can be given as a param in pacemaker to overwrite the default.

Maybe there are other ways to configure this without touching the resource agent?


Cheers
Nick

[Michael Klishin]

  On 6 February 2015 at 10:19:53, ni...@nine.ch (ni...@nine.ch) wrote:
> Sadly pacemaker was not able to start the resources on both hosts.
> After a bit of investigation I saw that the node name in the OCF
> resource agent is set to "rabbit@localhost" by default. This
> prevents the "rabbitmqctl wait" command to finish successfully.
> After I changed the default to "rabbit@$(hostname -s)" it was
> working. Shouldn't that be the default in the resource agent?
> When using a rabbitmq/pacemaker/drbd setup the nodename can
> be given as a param in pacemaker to overwrite the default.
>
> Maybe there are other ways to configure this without touching
> the resource agent?

Nick,

I'm not particularly familiar with OCF but you can override node name using an environment
variable: http://rabbitmq.com/configure.html.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[ni...@nine.ch]

Hey Michael,
thank you for your response. The default setting for a nodename in rabbitmq seems to be "rabbit@<hostname>" (according to the documentation). In the OCF resource agent the default for the nodename is:

"""
OCF_RESKEY_nodename_default="rabbit@localhost"
"""

You can change the nodename in pacemaker, but this configuration would be the same on both nodes then (as pacemaker syncs it's config for all the nodes). So I was just wondering if it wouldn't be better have this in the OCF resource agent:

"""
OCF_RESKEY_nodename_default="rabbit@$(hostname -s)"
"""

as it would match the default setting a bit better.


Cheers
Nick

[Michael Klishin]

Yes, this makes perfect sense. How can this be updated? (in what project/repository)?

MK

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ni...@nine.ch]

I took the debian/ubuntu package (rabbitmq-server_3.4.3-1_all.deb package) from:

http://www.rabbitmq.com/debian/ testing main

In there the resource agent is in "usr/lib/ocf/resource.d/rabbitmq/rabbitmq-server".


Cheers
Nick

[Michael Klishin]

  On 10 February 2015 at 09:52:55, ni...@nine.ch (ni...@nine.ch) wrote:
> I took the debian/ubuntu package (rabbitmq-server_3.4.3-1_all.deb
> package) from:
>
> http://www.rabbitmq.com/debian/ testing main
>
> In there the resource agent is in "usr/lib/ocf/resource.d/rabbitmq/rabbitmq-server".

Indeed:
http://hg.rabbitmq.com/rabbitmq-server/file/316a6bd2585f/packaging/common/rabbitmq-server.ocf

However, I see that it's nodename_default that is set to `rabbit@localhost`. It still can be overriden
(with `nodename`):
http://hg.rabbitmq.com/rabbitmq-server/file/316a6bd2585f/packaging/common/rabbitmq-server.ocf#l171

I wonder if this was the idea when Pacemaker resource file was first developed (which was years ago).
Are you sure it's safe to make the suggested change?

I also can't find any docs on the use of variables in resource agent files, and testing this would take more
effort than we currently can put into.

[ni...@nine.ch]

In my understanding the setting rabbit@localhost might work with a rabbitmq/drbd/pacemaker (active/passive) setup as you switch the mnesa file store between 2 nodes (via drbd). The node name of the rabbitmq installation will be always the same (as the mnesa directory content is always the same). I hope so far I am right :-)

We are using a setup with mirrored queues in rabbitmq (having 2 nodes). So we have one rabbitmq installation on each node. They are named differently (as the hostnames are different). With pacemaker we make sure that both instances are running (using a 'clone' resource) and additionally we switch a virtual IP between the 2 nodes to make rabbitmq high available. The problem is that pacemaker will sync the configuration (aka our "nodename" parameter) for one resource (our rabbitmq clone resource) between the 2 nodes. So giving a nodename here, will be wrong on one side of the cluster. Using "rabbit@$(hostname -s)" as default in the OCF resource agent solves this problem. However it might be possible that people using a rabbitmq/drbd/pacemaker setup will now have to set the nodename explicitly in pacemaker with a param (if their installation is named "rabbit@localhost").

What do you think?


Cheers
Nick

[Michael Klishin]

 

On 10 February 2015 at 11:19:40, ni...@nine.ch (ni...@nine.ch) wrote:
> > In my understanding the setting rabbit@localhost might work
> with a rabbitmq/drbd/pacemaker (active/passive) setup as
> you switch the mnesa file store between 2 nodes (via drbd). The
> node name of the rabbitmq installation will be always the same
> (as the mnesa directory content is always the same). I hope so
> far I am right :-)
>
> We are using a setup with mirrored queues in rabbitmq (having
> 2 nodes). So we have one rabbitmq installation on each node. They
> are named differently (as the hostnames are different). With
> pacemaker we make sure that both instances are running (using
> a 'clone' resource) and additionally we switch a virtual IP between
> the 2 nodes to make rabbitmq high available. The problem is that
> pacemaker will sync the configuration (aka our "nodename" parameter)
> for one resource (our rabbitmq clone resource) between the 2
> nodes. So giving a nodename here, will be wrong on one side of the
> cluster.

Yes, that's precisely the concern I had.

> Using "rabbit@$(hostname -s)" as default in the OCF
> resource agent solves this problem. However it might be possible
> that people using a rabbitmq/drbd/pacemaker setup will now
> have to set the nodename explicitly in pacemaker with a param
> (if their installation is named "rabbit@localhost").

Also good point.

> What do you think?

Is there any documentation on the use of variables (or shell outs) such as $(hostname -f)?
I wonder if the same can be accomplished by using the nodename parameter instead of changing the default.

Thank you for the clarification, by the way.

[ni...@nine.ch]

The only resource I found for OCF variables was: http://www.linux-ha.org/doc/dev-guides/_api_definitions.html#_environment_variables. So far as I know you can't have node specific parameters for resource agents. Pacemaker will always try to sync them between the nodes.

Maybe we can try to use the settings from rabbitmq-env.conf in the resource agent (as we specify the nodename per installation there as well)?


Cheers
Nick

[Michael Klishin]

 On 10 February 2015 at 11:49:05, ni...@nine.ch (ni...@nine.ch) wrote:
> Maybe we can try to use the settings from rabbitmq-env.conf
> in the resource agent (as we specify the nodename per installation
> there as well)?

I like this idea. This is more in line how RabbitMQ is typically configured.

Do you want to work on a patch for this? You can use `hg export` or `git format-patch` (if github mirror is more convenient for you),
then post it to this list.

Thank you!