trying to understand why a reader vip is not moving to a new host and how to resolve it

[Carmen Sarlo]

Hi, 

I am trying to move a reader vip resource to another host, but can not. I saw using crm_mon -A1 that the host (db3) that I would like to move the vip to is readable = 0. I checked slave status seconds behind master and its 0. Why would the node be readable 0 and how can I make it not be? Is my vip condition wrong?

crm_mon -A1 

============

Last updated: Thu Jan  7 17:38:43 2016

Last change: Thu Jan  7 17:21:34 2016 via crmd on db2

Stack: openais

Current DC: db1 - partition with quorum

Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c

4 Nodes configured, 4 expected votes

7 Resources configured.

============

Online: [ db2 db1 db3 db4 ]

 db_writer_vip  (ocf::heartbeat:IPaddr2): Started db1

 db_reader_vip1 (ocf::heartbeat:IPaddr2): Started db4

 db_reader_vip2 (ocf::heartbeat:IPaddr2): Started db4

 Master/Slave Set: ms_MySQL [p_mysql]

     Masters: [ db1 ]

     Slaves: [ db2 db3 db4 ]

Node Attributes:

* Node db2:

    + master-p_mysql:2                  : 0         

    + p_mysql_mysql_master_IP           : x.x.x.22

    + readable                          : 0         

* Node db1:

    + master-p_mysql:0                  : 2147483647

    + master-p_mysql:3                  : 0         

    + p_mysql_last_trx_md5              : 6cbab0d2a1f57579a5c4ce17dd364b86

    + p_mysql_mysql_master_IP           : x.x.x.21

    + readable                          : 1         

* Node db3:

    + master-p_mysql:2                  : 0         

    + readable                          : 0         

* Node db4:

    + master-p_mysql:2                  : 12        

    + p_mysql_last_trx_md5              : 6cbab0d2a1f57579a5c4ce17dd364b86

    + readable                          : 1      

Corosync version:1.4.2

Pacemaker version:1.1.6

percona mysql version: 5.5

config:

node db1 \

  attributes p_mysql_mysql_master_IP="x.x.x.21" standby="off"

node db2 \

  attributes p_mysql_mysql_master_IP="x.x.x.22" standby="off"

node db3 \

  attributes standby="off"

node db4 \

  attributes standby="off"

primitive db_reader_vip1 ocf:heartbeat:IPaddr2 \

  params ip="x.x.x.101" nic="bond0" cidr_netmask="24" \

  op monitor interval="10s" \

  meta target-role="Started"

primitive db_reader_vip2 ocf:heartbeat:IPaddr2 \

  params ip="x.x.x.102" nic="bond0" cidr_netmask="24" \

  op monitor interval="10s" \

  meta target-role="Started"

primitive db_writer_vip ocf:heartbeat:IPaddr2 \

  params ip="x.x.x.100" nic="bond0" cidr_netmask="24" \

  op monitor interval="10s" \

  meta target-role="Started"

primitive p_mysql ocf:percona:mysql \

  params config="/etc/mysql/my.cnf" datadir="/data/var/mysql" pid="/var/run/mysqld/mysqld.pid" socket="/var/run/mysqld/mysqld.sock" replication_user="someuser_for_replication" replication_passwd="****************" max_slave_lag="60" evict_outdated_slaves="false" binary="/usr/sbin/mysqld" test_user="sometest_user_prm" test_passwd="**************" async_stop="1" additional_parameters="--log-error=/data/var/mysql/mysql_err.log" \

  op monitor interval="5s" role="Master" timeout="60s" ocf_check_level="1" \

  op monitor interval="2s" role="Slave" timeout="60s" ocf_check_level="1" \

  op start interval="0" timeout="180s" \

  op stop interval="0" timeout="180s" \

  meta target-role="Started"

ms ms_MySQL p_mysql \

  meta master-max="1" master-node-max="1" clone-max="4" clone-node-max="1" notify="true" globally-unique="false" target-role="Master" is-managed="true"

location db1-master ms_MySQL \

  rule $id="db1-master-rule" $role="Master" 9000: #uname eq db1

location db2-master ms_MySQL \

  rule $id="db2-master-rule" $role="Master" 5500: #uname eq db2

location db3-no-master ms_MySQL \

  rule $id="db3-no-master-rule" $role="Master" -9900: #uname eq db3

location db3-prefer-reader-vip1 db_reader_vip1 \

  rule $id="db3-prefer-reader-vip1-rule" 1000: #uname eq db3

location db3-prefer-reader-vip2 db_reader_vip2 \

  rule $id="db3-prefer-reader-vip2-rule" 2500: #uname eq db3

location db4-no-master ms_MySQL \

  rule $id="db4-no-master-rule" $role="Master" -9900: #uname eq db4

location db4-prefer-reader-vip1 db_reader_vip1 \

  rule $id="db4-prefer-reader-vip1-rule" 2500: #uname eq db4

location db4-prefer-reader-vip2 db_reader_vip2 \

  rule $id="db4-prefer-reader-vip2-rule" 1000: #uname eq db4

location cli-prefer-db_reader_vip1 db_reader_vip1 \

  rule $id="cli-prefer-rule-db_reader_vip1" inf: #uname eq db4

location cli-prefer-db_reader_vip2 db_reader_vip2 \

  rule $id="cli-prefer-rule-db_reader_vip2" inf: #uname eq db3

location loc-no-db-reader-vip-1 db_reader_vip1 \

  rule $id="loc-no-db-reader-vip-1-rule" -inf: readable lt 1

location loc-no-db-reader-vip2 db_reader_vip2 \

  rule $id="loc-no-db-reader-vip2-rule" -inf: readable lt 1

colocation col_reader1_dislike_reader2 -200: db_reader_vip1 db_reader_vip2

colocation col_reader1_dislike_writer -200: db_reader_vip1 db_writer_vip

colocation col_reader2_dislike_writer -200: db_reader_vip2 db_writer_vip

colocation db_writer_vip_on_master inf: db_writer_vip ms_MySQL:Master

order ms_MySQL_promote_before_vip inf: ms_MySQL:promote db_writer_vip:start

property $id="cib-bootstrap-options" \

  dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c" \

  cluster-infrastructure="openais" \

  expected-quorum-votes="4" \

  stonith-enabled="false" \

  no-quorum-policy="ignore" \

  last-lrm-refresh="1452185040" \

  maintenance-mode="false"

property $id="mysql_replication" \

  p_mysql_REPL_INFO="x.x.x.21|master-bin.006884|16357940"

 

[Yves Trudeau]

Hi Carmen,

   The problem is that the readable attribute is updated every 2s by the agent.  I suggest you do something a bit like:

https://github.com/percona/percona-pacemaker-agents/blob/master/doc/PRM-setup-guide.rst#configuring-a-report-slave-with-a-dedicated-vip

but instead of setting the rule on the hostname, you can use a custom node attribute like "db_reader_vip1_allowed".    First setup the attribute for all nodes:

crm_attribute -l forever -G --node NodeName --name db_reader_vip1_allowed -v "1"

and repeat for each NodeName

Then add the ruled like:

location loc-db-reader-vip-1-allowed db_reader_vip1 \

  rule $id="loc-no-db-reader-vip-1-allowed-rule" -inf: db_reader_vip1_allowed lt 1

and then set the attribute to 0 with the above crm_attribute command where you don't want it.  Once moved, it should stay there because of the default stickiness.

Regards,

Yves

--
You received this message because you are subscribed to the Google Groups "PRM-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prm-discuss...@googlegroups.com.
To post to this group, send email to prm-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/prm-discuss.
For more options, visit https://groups.google.com/d/optout.

[Yves Trudeau]

Hi, 

  ok, I see.  So readable is 0.  The best would be to enable the trace for maybe 15s and then see why it is not setting the attribute to 1 if replication is ok.

on db3:

mkdir -p /tmp/mysql.ocf.ra.debug

touch /tmp/mysql.ocf.ra.debug/log

sleep 15

mv  /tmp/mysql.ocf.ra.debug/log  /tmp/mysql.ocf.ra.debug/log.1

then check in the file.  If you want, I also can have a look.

Regards,

Yves

2016-01-08 17:30 GMT-05:00 Carmen Sarlo <carmen...@voxmedia.com>:

Thats more of a workaround. Why can I not move a reader vip to a another the db3 host? I can make a reader vip to be a preferred host, but I want to move it. I did not see anything in the trace log to think something was wrong. 

I did put the cluster in maintenance mode a while back. that reader vip was on db3. I had to do some hw maintenance, so i moved the vip to db4 and then put the cluster in maintenance mode. After the i was done with the hardware work and server rebooted, i took the cluster out of maintenance mode. I have not moved a vip since then. Could that be a problem?

--

Carmen Sarlo | Principal Operations Engineer | (856) 630-3146

Vox Media is The Verge, Vox.com, SB Nation, 
Polygon, Eater, Racked, Curbed, and Re/code

New at Vox Media: The agency view on Vox Media

[Carmen Sarlo]

Hi,

I ran a trace log on where the vip is now(db4) and where I want to move it (db3) and did a diff on them. I only see differences in places I'd expect like host names, 

OCF_RESOURCE_INSTANCE, pids, OCF_RESKEY_CRM_meta_clone, and mktemp file names.