make reader vips prefer to be on the same host before moving to master

[Justin Burnham]

I'm looking to make the reader vips move to any remaining slaves before moving to the master if no other host is left. 

Example. 

db01 - writer_vip

db02 - reader_vip_1

db03 - reader_vip_2

crm node standby db02

Currently reader_vip_1 will move to db01. I would rather it move to db03. If db03 was also removed from the cluster, both reader vips should move to db01.

I can't seem to get the colocations working to get the right effect. My idea is to make having the writer_vip and reader_vip_* have a lower score than the dislike colocation for one reader vip to the other. For some reason though, when I tweak the scores, it doesn't have the right affect and it baffles me. Can anyone guide me on getting the right config to make this work?

As a reference here is the current set of rules for the vips:

colocation reader_vip_1_dislike_reader_vip_2 -200: reader_vip_1 reader_vip_2

colocation reader_vip_1_dislike_writer_vip -200: reader_vip_1 writer_vip

colocation reader_vip_2_dislike_writer_vip -200: reader_vip_2 writer_vip

colocation writer_vip_on_master inf: writer_vip ms_mysql:Master

There is a resource stickiness of 150 and the current scores from crm_simulate -Ls are:

native_color: reader_vip_2 allocation score on db01: -200

native_color: reader_vip_2 allocation score on db02: 150

native_color: reader_vip_2 allocation score on db03: 0

native_color: reader_vip_1 allocation score on db01: -200

native_color: reader_vip_1 allocation score on db02: -200

native_color: reader_vip_1 allocation score on db03: 150

Thanks

[Yves Trudeau]

Hi,

   what have you tried in term of scores?  Personally, I would have first tried a lower negative value like -150 for reader_vip_1_dislike_reader_vip_2.  With the same score, it is up to pacemaker to decide where to send reader_vip_2.

Regards,

Yves

--
You received this message because you are subscribed to the Google Groups "PRM-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prm-discuss...@googlegroups.com.
To post to this group, send email to prm-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/prm-discuss.
For more options, visit https://groups.google.com/groups/opt_out.

[Justin Burnham]

The problem is the scores are different when I simulate a node going down. For example with the dislike being -150:

db02 going down:

crm_simulate -Ls --node-down db02
Current cluster status:
Online: [ db01 db02 db03 ]
 reader_vip_2   (ocf::heartbeat:IPaddr2):       Started db03
 writer_vip     (ocf::heartbeat:IPaddr2):       Started db01
 reader_vip_1   (ocf::heartbeat:IPaddr2):       Started db02
 Master/Slave Set: ms_mysql [mysql_prm]
     Masters: [ db01 ]
     Slaves: [ db02 db03 ]
Performing requested modifications
 + Taking node db02 offline
Allocation scores:
...
native_color: writer_vip allocation score on db01: 1360
native_color: writer_vip allocation score on db02: -INFINITY
native_color: writer_vip allocation score on db03: -INFINITY

native_color: reader_vip_2 allocation score on db01: -200

native_color: reader_vip_2 allocation score on db02: 0
native_color: reader_vip_2 allocation score on db03: 150
native_color: reader_vip_1 allocation score on db01: 0
native_color: reader_vip_1 allocation score on db02: 0
native_color: reader_vip_1 allocation score on db03: -150
Transition Summary:
 * Start   reader_vip_1 (db01)

What happened to the -200 for the host with the writer_vip on the allocation score for reader_vip_1??

db03 going down happens to do what I want

crm_simulate -Ls --node-down db03
Current cluster status:
Online: [ db01 db02 db03 ]
 reader_vip_2   (ocf::heartbeat:IPaddr2):       Started db03
 writer_vip     (ocf::heartbeat:IPaddr2):       Started db01
 reader_vip_1   (ocf::heartbeat:IPaddr2):       Started db02
 Master/Slave Set: ms_mysql [mysql_prm]
     Masters: [ db01 ]
     Slaves: [ db02 db03 ]
Performing requested modifications
 + Taking node db03 offline
Allocation scores:
...
native_color: writer_vip allocation score on db01: 1360
native_color: writer_vip allocation score on db02: -INFINITY
native_color: writer_vip allocation score on db03: -INFINITY

native_color: reader_vip_2 allocation score on db01: -200

native_color: reader_vip_2 allocation score on db02: 0

native_color: reader_vip_2 allocation score on db03: 0
native_color: reader_vip_1 allocation score on db01: -200

native_color: reader_vip_1 allocation score on db02: 0
native_color: reader_vip_1 allocation score on db03: 0
Transition Summary:
 * Start   reader_vip_2 (db02)

The scores without a node going down look sane and would suggest that the other slave is the second best option:

native_color: reader_vip_2 allocation score on db01: -200

native_color: reader_vip_2 allocation score on db02: 0
native_color: reader_vip_2 allocation score on db03: 150

native_color: reader_vip_1 allocation score on db01: -200

native_color: reader_vip_1 allocation score on db02: 150
native_color: reader_vip_1 allocation score on db03: -150

 

[Yves Trudeau]

Hi Justin,

  I can easily reproduce your problem,  the ordering of the evaluation is the problem.  I tried something to force the order and it seems to work:

order ms_MySQL_promote_before_vip inf: ms_MySQL:promote writer_vip:start reader_vip_1:start reader_vip_2:start

so that forces to reevaluate the scores of reader_vip_1 and _2.  With this I now have:

root@pacemaker-1:~# crm status
============
Last updated: Tue Feb 25 16:05:23 2014
Last change: Tue Feb 25 16:05:09 2014 via cibadmin on pacemaker-1
Stack: openais
Current DC: pacemaker-1 - partition with quorum
Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff
3 Nodes configured, 3 expected votes
6 Resources configured.
============

Online: [ pacemaker-1 pacemaker-2 pacemaker-3 ]

 reader_vip_1   (ocf::heartbeat:IPaddr2):       Started pacemaker-1
 reader_vip_2   (ocf::heartbeat:IPaddr2):       Started pacemaker-3
 writer_vip     (ocf::heartbeat:IPaddr2):       Started pacemaker-2
 Master/Slave Set: ms_MySQL [p_mysql]
     Masters: [ pacemaker-2 ]
     Slaves: [ pacemaker-1 pacemaker-3 ]
root@pacemaker-1:~# crm_simulate -Ls --node-down pacemaker-1 | grep vip
 reader_vip_1   (ocf::heartbeat:IPaddr2):       Started pacemaker-1
 reader_vip_2   (ocf::heartbeat:IPaddr2):       Started pacemaker-3
 writer_vip     (ocf::heartbeat:IPaddr2):       Started pacemaker-2
native_color: reader_vip_2 allocation score on pacemaker-1: 0
native_color: reader_vip_2 allocation score on pacemaker-2: 0
native_color: reader_vip_2 allocation score on pacemaker-3: 0
native_color: reader_vip_1 allocation score on pacemaker-1: 0
native_color: reader_vip_1 allocation score on pacemaker-2: 0
native_color: reader_vip_1 allocation score on pacemaker-3: -200
native_color: writer_vip allocation score on pacemaker-1: -INFINITY
native_color: writer_vip allocation score on pacemaker-2: 816
native_color: writer_vip allocation score on pacemaker-3: -INFINITY
 * Start   reader_vip_1 (pacemaker-2)
 * Restart reader_vip_2 (Started pacemaker-3)
root@pacemaker-1:~# crm_simulate -Ls --node-down pacemaker-3 | grep vip
 reader_vip_1   (ocf::heartbeat:IPaddr2):       Started pacemaker-1
 reader_vip_2   (ocf::heartbeat:IPaddr2):       Started pacemaker-3
 writer_vip     (ocf::heartbeat:IPaddr2):       Started pacemaker-2
native_color: reader_vip_2 allocation score on pacemaker-1: 0
native_color: reader_vip_2 allocation score on pacemaker-2: 0
native_color: reader_vip_2 allocation score on pacemaker-3: 0
native_color: reader_vip_1 allocation score on pacemaker-1: -200
native_color: reader_vip_1 allocation score on pacemaker-2: 0
native_color: reader_vip_1 allocation score on pacemaker-3: 0
native_color: writer_vip allocation score on pacemaker-1: -INFINITY
native_color: writer_vip allocation score on pacemaker-2: 816
native_color: writer_vip allocation score on pacemaker-3: -INFINITY
 * Move    reader_vip_1 (Started pacemaker-1 -> pacemaker-2)
 * Start   reader_vip_2 (pacemaker-1)

which is the expected behavior. 

Regards,

Yves

[Justin Burnham]

Hi Yves, 

  See my reply inline.

Thanks

Since pacemaker-2 is the master, in this simulation, this is still doing what it did before and moving one of the reader_vips to the master instead of what I was looking to do which is have both reader_vips co-exist on the same box where there are only two boxes that are readable.

 

root@pacemaker-1:~# crm_simulate -Ls --node-down pacemaker-3 | grep vip
 reader_vip_1   (ocf::heartbeat:IPaddr2):       Started pacemaker-1
 reader_vip_2   (ocf::heartbeat:IPaddr2):       Started pacemaker-3
 writer_vip     (ocf::heartbeat:IPaddr2):       Started pacemaker-2
native_color: reader_vip_2 allocation score on pacemaker-1: 0
native_color: reader_vip_2 allocation score on pacemaker-2: 0
native_color: reader_vip_2 allocation score on pacemaker-3: 0
native_color: reader_vip_1 allocation score on pacemaker-1: -200
native_color: reader_vip_1 allocation score on pacemaker-2: 0
native_color: reader_vip_1 allocation score on pacemaker-3: 0
native_color: writer_vip allocation score on pacemaker-1: -INFINITY
native_color: writer_vip allocation score on pacemaker-2: 816
native_color: writer_vip allocation score on pacemaker-3: -INFINITY
 * Move    reader_vip_1 (Started pacemaker-1 -> pacemaker-2)
 * Start   reader_vip_2 (pacemaker-1)

Same issue here as well. I am looking to have reader_vip_1 stay on pacemaker-1 and move reader_vip_2 to pacemaker-1.

which is the expected behavior. 

This is the existing behavior and I'm looking to change it. 

[Yves Trudeau]

Nope, read carefully:

 * Move    reader_vip_1 (Started pacemaker-1 -> pacemaker-2)
 * Start   reader_vip_2 (pacemaker-1)

It ends on pacemaker-1, not -2 which is the master.  I also did the real thing and the surviving slave has the 2 read vips.

Regards,

Yves

[Justin Burnham]

The line you're showing says it's moving reader_vip_1 from pacemaker-1 (slave) TO pacemaker-2 (master). reader_vip_2 which was on pacemaker-3 is starting on pacemaker-1.

My config has the following orders and colocations:

ms ms_mysql mysql_prm \

        meta master-max="1" master-node-max="1" clone-max="3" clone-node-max="1" notify="true" globally-unique="false" target-role="Master"

location loc-no-reader-vip-1 reader_vip_1 \

        rule $id="loc-no-reader-vip-1-rule" -inf: readable eq 0

location loc-no-reader-vip-2 reader_vip_2 \

        rule $id="loc-no-reader-vip-2-rule" -inf: readable eq 0

location loc-not-defined-reader-vip-1 reader_vip_1 \

        rule $id="loc-not-defined-reader-vip-1-rule" -inf: not_defined readable

location loc-not-defined-reader-vip-2 reader_vip_2 \

        rule $id="loc-not-defined-reader-vip-2-rule" -inf: not_defined readable

colocation reader_vip_1_dislike_reader_vip_2 -150: reader_vip_1 reader_vip_2

colocation reader_vip_1_dislike_writer_vip -200: reader_vip_1 writer_vip

colocation reader_vip_2_dislike_writer_vip -200: reader_vip_2 writer_vip

colocation writer_vip_on_master inf: writer_vip ms_mysql:Master

order ms_MySQL_promote_before_vip inf: ms_mysql:promote writer_vip:start reader_vip_1:start reader_vip_2:start

A simulation of mine as well as doing crm node standby shows the following effect:

[root@db01 ~]# crm_simulate -Ls --node-down db03 | grep vip

 reader_vip_2   (ocf::heartbeat:IPaddr2):       Started db01

 writer_vip     (ocf::heartbeat:IPaddr2):       Started db02

 reader_vip_1   (ocf::heartbeat:IPaddr2):       Started db03

native_color: writer_vip allocation score on db01: -INFINITY

native_color: writer_vip allocation score on db02: 1360

native_color: writer_vip allocation score on db03: -INFINITY

native_color: reader_vip_2 allocation score on db01: 150

native_color: reader_vip_2 allocation score on db02: -200

native_color: reader_vip_2 allocation score on db03: 0

native_color: reader_vip_1 allocation score on db01: -150

native_color: reader_vip_1 allocation score on db02: 0

native_color: reader_vip_1 allocation score on db03: 0

 * Restart reader_vip_2 (Started db01)

 * Start   reader_vip_1 (db02) <--- this is on the master now

The other node going down does seem to work properly, this is probably due to the colocation of vip_1 not liking vip_2

[root@db01 ~]# crm_simulate -Ls --node-down db01 | grep vip

 reader_vip_2   (ocf::heartbeat:IPaddr2):       Started db01

 writer_vip     (ocf::heartbeat:IPaddr2):       Started db02

 reader_vip_1   (ocf::heartbeat:IPaddr2):       Started db03

native_color: writer_vip allocation score on db01: -INFINITY

native_color: writer_vip allocation score on db02: 1360

native_color: writer_vip allocation score on db03: -INFINITY

native_color: reader_vip_2 allocation score on db01: 0

native_color: reader_vip_2 allocation score on db02: -200

native_color: reader_vip_2 allocation score on db03: 0

native_color: reader_vip_1 allocation score on db01: 0

native_color: reader_vip_1 allocation score on db02: -200

native_color: reader_vip_1 allocation score on db03: 0

 * Start   reader_vip_2 (db03)

[Yves Trudeau]

I see what you mean, I'll test something and come back to you.

Regards,

Yves

[Yves Trudeau]

That smells like a bug, I'll test with a recent version and submit a bug.

Regards,

Yves