Slave host not changing to new Master host

[Carmen Sarlo]

I am currently testing a 3 node setup. 

db1 - master/slave

db2 - master/slave

db3 - slave only

writer vip (master) was on db2 and I wanted to see what would happen id db2 was stopped. I killed the PID. The writer vip moved to db1, but the master host did not change on db3.

I am running: Pacemaker 1.1.6, Corosync 1.4.2

My config:

node db1 \

  attributes standby="off"

node db2

node db3 \

  attributes standby="off"

primitive db_reader_vip1 ocf:heartbeat:IPaddr2 \

  params ip="192.168.33.101" nic="eth1" cidr_netmask="24" \

  op monitor interval="10s" \

  meta target-role="Started"

primitive db_reader_vip2 ocf:heartbeat:IPaddr2 \

  params ip="192.168.33.102" nic="eth1" cidr_netmask="24" \

  op monitor interval="10s" \

  meta target-role="Started"

primitive db_writer_vip ocf:heartbeat:IPaddr2 \

  params ip="192.168.33.100" nic="eth1" cidr_netmask="24" \

  op monitor interval="10s" \

  meta target-role="Started"

primitive p_mysql ocf:percona:mysql \

  params config="/etc/mysql/my.cnf" pid="/var/run/mysqld/mysqld.pid" socket="/var/run/mysqld/mysqld.sock" replication_user="repluser" replication_passwd="replpass" max_slave_lag="60" evict_outdated_slaves="false" binary="/usr/sbin/mysqld" test_user="testuser" test_passwd="testpass" reader_attribute="prod_readable" \

  op monitor interval="5s" role="Master" ocf_check_level="1" \

  op monitor interval="3s" role="Slave" ocf_check_level="1" \

  op start interval="0" timeout="180s" \

  op stop interval="0" timeout="180s" \

  meta target-role="Started"

ms ms_MySQL p_mysql \

  meta master-max="1" master-node-max="1" clone-max="3" clone-node-max="1" notify="true" globally-unique="false" target-role="Slave" is-managed="true"

location cli-prefer-db_reader_vip2 db_reader_vip2 \

  rule $id="cli-prefer-rule-db_reader_vip2" inf: #uname eq db1

location db3_never_master ms_MySQL \

  rule $id="db3_never_master-rule" $role="Master" -inf: #uname eq db3

location loc-no-db-reader-vip-1 db_reader_vip1 \

  rule $id="loc-no-db-reader-vip-1-rule" -inf: reader_attribute lt 1

location loc-no-db-reader-vip2 db_reader_vip2 \

  rule $id="loc-no-db-reader-vip2-rule" -inf: readable lt 1

colocation col_vip_dislike_each_other -200: db_reader_vip1 db_writer_vip

colocation db_writer_vip_on_master inf: db_writer_vip ms_MySQL:Master

order ms_MySQL_promote_before_vip inf: ms_MySQL:promote db_writer_vip:start

property $id="cib-bootstrap-options" \

  stonith-enabled="false" \

  expected-quorum-votes="3" \

  dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c" \

  cluster-infrastructure="openais" \

  last-lrm-refresh="1440097869"

property $id="mysql_replication" \

  p_mysql_REPL_INFO="db1|master-bin.000006|107"

[Yves Trudeau]

Hi Carmen,

   any errors?  Also, based on the status, you seem to be using an old agent have you downloaded it from the Percona github?

https://github.com/percona/percona-pacemaker-agents/raw/master/agents/mysql_prm

Regards,

Yves

--
You received this message because you are subscribed to the Google Groups "PRM-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prm-discuss...@googlegroups.com.
To post to this group, send email to prm-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/prm-discuss.
For more options, visit https://groups.google.com/d/optout.

[Yves Trudeau]

ms ms_MySQL p_mysql \

  meta master-max="1" master-node-max="1" clone-max="3" clone-node-max="1" notify="true" globally-unique="false" target-role="Slave" is-managed="true"

target-role="Slave" is wrong, it should be Master.

Regards,

Yves

[Yves Trudeau]

Here, the output of "crm_mon -A1" would be useful too.

Regards,

Yves

[Carmen Sarlo]

I am not sure why target-role="Slave" happened. In my original config it was set to Master. 

The mysql_prm agent is the same.

[Carmen Sarlo]

Here's that crm_mon output:

============

Last updated: Thu Aug 20 21:00:39 2015

Last change: Thu Aug 20 20:56:12 2015 via cibadmin on db1

Stack: openais

Current DC: db2 - partition with quorum

Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c

3 Nodes configured, 3 expected votes

6 Resources configured.

============

Online: [ db2 db1 db3 ]

 db_writer_vip  (ocf::heartbeat:IPaddr2): Started db1

 db_reader_vip1 (ocf::heartbeat:IPaddr2): Started db2

 db_reader_vip2 (ocf::heartbeat:IPaddr2): Started db3

 Master/Slave Set: ms_MySQL [p_mysql]

     Masters: [ db1 ]

     Slaves: [ db2 db3 ]

Node Attributes:

* Node db2:

    + master-p_mysql:1                  : 60        

    + p_mysql_last_trx_md5              : mysqld-relay-bin.000002@

    + prod_readable                     : 1         

* Node db1:

    + master-p_mysql:0                  : 2147483647

    + p_mysql_last_trx_md5              : mysqld-relay-bin.000006@

    + prod_readable                     : 1         

* Node db3:

    + master-p_mysql:2                  : 60        

    + prod_readable                     : 1         

[Carmen Sarlo]

I ran another test today. 

db1 was master

db1 placed in standby, db2 became master and db3 changed master to db2.

db1 back online, db2 still master that's fine

db2 in standby, db1 becomes master db3 changes master to db1

I edited the config so db3 is never master

location db3_never_master ms_MySQL \

        rule $id="db3_never_master-rule" $role="Master" -inf: #uname eq db3

db2 online, db1 still master

db1 in standby, db2 becomes master. db3 does not change master

[Carmen Sarlo]

interesting in the trace log

++ timeout 5 cibadmin --query --xpath '/cib/configuration/constraints/rsc_location[@rsc='\''ms_MySQL'\'']/rule[@score='\''-INFINITY'\'']/expression[@attribute='\''#uname'\'' and @value='\''db3'\'']'

I do not see  $role=Master in there.

[Yves Trudeau]

Hi Carmen,

    this is very interesting, I'll find some time sort that out but I am booked solid for a few days.

Regards,

Yves

[Carmen Sarlo]

Hi Yves,

I was unsuccessful using a negative infinity score, but was successful using a negative score of -1500.

Carmen

[Yves Trudeau]

Hi Carmen,

   We had to add a location rule check to avoid issues with arbitrator nodes,  it seems to cause issue when a node needs to be excluded from the master role.  I suggest you set a high negative score instead of -INF like:

location db3_never_master ms_MySQL \

  rule $id="db3_never_master-rule" $role="Master" -99999: #uname eq db3

This has the side effect of allowing db3 to be the master if the other two nodes are down.  Is this a bad thing?

Regards,

Yves

[Yves Trudeau]

Hi,

     yes, that was the topic of my previous message.  I'll modify the documentation or try to find a workaround that satisfies both requirements for the standby arbitrator and the non-master-able slave.

Regards,

Yves

[Carmen Sarlo]

In my production environment, I will have 2 hosts be available as master, 2 hosts available as slaves, and 1 as host that performs backups. For the slaves I was thinking of a negative value like -1500 for role="Master", the 2 hosts I want to designate as the masters they'll have higher score like 1 at 1500 and the other at 1000. The host for backups I will do -9999.