Ronald Bradford
unread,Dec 17, 2008, 3:13:11 PM12/17/08Sign in to reply to author
Sign in to forward
You do not have permission to delete messages in this group
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to flippe...@googlegroups.com
I have two new installations of 5.1.30 (codename: alpha 192.168.2.187 & beta 192.168.2.181), and flipper will not assign the Write IP address due to the error "Replication could not be synced"
A closer investigation shows slave replication on beta is up to date.
On alpha however, the slave is not.
We have a Master position of 4350 (the master pos on beta)
On the Slave we have a Read_Master_log_pos of 4350, but an Exec_master_log_pos of 2747.
Slave threads are running and there are no errors.
Restarting slave has no impact.
Analysis of the "commands between 2747 & 4350" shows all commands that were initiated on the alpha server, so would be automatically excluded when it comes back to alpha.
Yes server-id's are different.
I don't really know how to address this, other then resetting the master pos with CHANGE_MASTER
Any input appreciated.
$ ./flipper --verbose prod set write alpha
Connection to 192.168.2.187 closed.
INFO: Moving write from beta to alpha
Connection to 192.168.2.181 closed.
WARNING: write IP is not up on beta node.
WARNING: Won't attempt to take down write IP on beta node.
ERROR: Replication could not be synced. Write IP won't be moved.
rbradfor@ronald:/tmp/local/scripts$ ./flipper
The following masterpairs exist:
prod
rbradfor@ronald:/tmp/local/scripts$ ./flipper --debug prod set write alpha
DEBUG: Executing ssh -l flipper 192.168.2.187 "uname -s;/sbin/ifconfig -a"
Connection to 192.168.2.187 closed.
About to get IP info from beta
DEBUG: Executing ssh -l flipper 192.168.2.181 "uname -s;/sbin/ifconfig -a"
Connection to 192.168.2.181 closed.
WARNING: write IP is not up on beta node.
WARNING: Won't attempt to take down write IP on beta node.
Master: File master-log.000001 Position 4350
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
ERROR: Replication could not be synced. Write IP won't be moved.
rbradfor@ronald:/tmp/local/scripts$ mysql -uupcodba -p -h192.168.2.187
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 43
Server version: 5.1.30-log MySQL Community Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> show master status;
+-------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| master-log.000001 | 4350 | | |
+-------------------+----------+--------------+------------------+
1 row in set (0.00 sec)
mysql> exit
Bye
rbradfor@ronald:/tmp/local/scripts$ mysql -uupcodba -p -h192.168.2.181
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 23
Server version: 5.1.30-log MySQL Community Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.2.187
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-log.000001
Read_Master_Log_Pos: 4350
Relay_Log_File: slave-log.000002
Relay_Log_Pos: 3395
Relay_Master_Log_File: master-log.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 4350
Relay_Log_Space: 3544
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
1 row in set (0.01 sec)
mysql> show master status;
+-------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| master-log.000001 | 4350 | | |
+-------------------+----------+--------------+------------------+
1 row in set (0.00 sec)
mysql> exit
Bye
rbradfor@ronald:/tmp/local/scripts$ mysql -uupcodba -p -h192.168.2.187
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 44
Server version: 5.1.30-log MySQL Community Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.2.181
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-log.000001
Read_Master_Log_Pos: 4350
Relay_Log_File: slave-log.000002
Relay_Log_Pos: 252
Relay_Master_Log_File: master-log.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 2747
Relay_Log_Space: 1001
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
1 row in set (0.00 sec)
mysql> slave stop;
Query OK, 0 rows affected (0.00 sec)
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: 192.168.2.181
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-log.000001
Read_Master_Log_Pos: 4350
Relay_Log_File: slave-log.000002
Relay_Log_Pos: 252
Relay_Master_Log_File: master-log.000001
Slave_IO_Running: No
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 2747
Relay_Log_Space: 1001
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
1 row in set (0.00 sec)
mysql> slave start;
Query OK, 0 rows affected (0.00 sec)
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.2.181
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-log.000001
Read_Master_Log_Pos: 4350
Relay_Log_File: slave-log.000003
Relay_Log_Pos: 252
Relay_Master_Log_File: master-log.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 2747
Relay_Log_Space: 1147
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
1 row in set (0.00 sec)
# at 2747
#081217 19:34:50 server id 181 end_log_pos 2793 User_var
SET @`pair_name`:=_latin1 0x70726F64 COLLATE `latin1_swedish_ci`/*!*/;
# at 2793
#081217 19:34:50 server id 187 end_log_pos 2903 Query thread_id=21 exec_time=0 error_code=0
SET TIMESTAMP=1229542490/*!*/;
DELETE FROM meta.node WHERE masterpair = @pair_name
/*!*/;
# at 2903
#081217 19:34:50 server id 181 end_log_pos 2949 User_var
SET @`pair_name`:=_latin1 0x70726F64 COLLATE `latin1_swedish_ci`/*!*/;
# at 2949
#081217 19:34:50 server id 181 end_log_pos 2997 User_var
SET @`alpha_name`:=_latin1 0x616C706861 COLLATE `latin1_swedish_ci`/*!*/;
# at 2997
#081217 19:34:50 server id 181 end_log_pos 3051 User_var
SET @`alpha_ip`:=_latin1 0x3139322E3136382E322E313837 COLLATE `latin1_swedish_ci`/*!*/;
# at 3051
#081217 19:34:50 server id 181 end_log_pos 3097 User_var
SET @`beta_name`:=_latin1 0x62657461 COLLATE `latin1_swedish_ci`/*!*/;
# at 3097
#081217 19:34:50 server id 181 end_log_pos 3150 User_var
SET @`beta_ip`:=_latin1 0x3139322E3136382E322E313831 COLLATE `latin1_swedish_ci`/*!*/;
# at 3150
#081217 19:34:50 server id 187 end_log_pos 3358 Query thread_id=21 exec_time=0 error_code=0
SET TIMESTAMP=1229542490/*!*/;
INSERT INTO meta.node (masterpair, node, name, value) VALUES
(@pair_name, @alpha_name, 'ip', @alpha_ip),
(@pair_name, @beta_name, 'ip', @beta_ip)
/*!*/;
# at 3358
#081217 19:34:50 server id 181 end_log_pos 3404 User_var
SET @`pair_name`:=_latin1 0x70726F64 COLLATE `latin1_swedish_ci`/*!*/;
# at 3404
#081217 19:34:50 server id 187 end_log_pos 3520 Query thread_id=21 exec_time=0 error_code=0
SET TIMESTAMP=1229542490/*!*/;
DELETE FROM meta.masterpair WHERE masterpair = @pair_name
/*!*/;
# at 3520
#081217 19:34:50 server id 181 end_log_pos 3566 User_var
SET @`pair_name`:=_latin1 0x70726F64 COLLATE `latin1_swedish_ci`/*!*/;
# at 3566
#081217 19:34:50 server id 181 end_log_pos 3620 User_var
SET @`master_ip`:=_latin1 0x3139322E3136382E322E3731 COLLATE `latin1_swedish_ci`/*!*/;
# at 3620
#081217 19:34:50 server id 181 end_log_pos 3673 User_var
SET @`slave_ip`:=_latin1 0x3139322E3136382E322E3831 COLLATE `latin1_swedish_ci`/*!*/;
# at 3673
#081217 19:34:50 server id 181 end_log_pos 3726 User_var
SET @`netmask`:=_latin1 0x3235352E3235352E3235352E30 COLLATE `latin1_swedish_ci`/*!*/;
# at 3726
#081217 19:34:50 server id 181 end_log_pos 3781 User_var
SET @`broadcast`:=_latin1 0x3139322E3136382E322E323535 COLLATE `latin1_swedish_ci`/*!*/;
# at 3781
#081217 19:34:50 server id 187 end_log_pos 4350 Query thread_id=21 exec_time=0 error_code=0
SET TIMESTAMP=1229542490/*!*/;
INSERT INTO meta.masterpair(masterpair, name, value) VALUES
(@pair_name, 'read_ip', @master_ip),
(@pair_name, 'write_ip', @slave_ip),
(@pair_name, 'netmask', @netmask),
(@pair_name, 'broadcast', @broadcast),
(@pair_name, 'mysql_user', 'flipper'),
(@pair_name, 'mysql_password', 'letmein'),
(@pair_name, 'ssh_user', 'flipper'),
(@pair_name, 'send_arp_command','/usr/lib/heartbeat/send_arp -p /tmp/send_arp.pid -i 100 -r 5 $sendarp_interface $sendarp_ip auto $sendarp_broadcast $sendarp_netmask')
/*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;