Interesting problem with "Replication could not be synced" error

20 views
Skip to first unread message

Ronald Bradford

unread,
Dec 17, 2008, 3:13:11 PM12/17/08
to flippe...@googlegroups.com
I have two new installations of 5.1.30 (codename:  alpha 192.168.2.187 & beta 192.168.2.181), and flipper will not assign the Write IP address due to the error "Replication could not be synced"

A closer investigation shows slave replication on beta is up to date.
On alpha however, the slave is not. 

We have a Master position of 4350 (the master pos on beta)
On the Slave we have a Read_Master_log_pos of 4350, but an Exec_master_log_pos of 2747. 
Slave threads are running and there are no errors.
Restarting slave has no impact.
Analysis of the "commands between 2747 & 4350" shows all commands that were initiated on the alpha server, so would be automatically excluded when it comes back to alpha.

Yes server-id's are different.

I don't really know how to address this, other then resetting the master pos with CHANGE_MASTER

Any input appreciated.


$ ./flipper --verbose prod set write alpha
Connection to 192.168.2.187 closed.
INFO: Moving write from beta to alpha
Connection to 192.168.2.181 closed.
WARNING: write IP is not up on beta node.
WARNING: Won't attempt to take down write IP on beta node.
ERROR: Replication could not be synced.  Write IP won't be moved.
rbradfor@ronald:/tmp/local/scripts$ ./flipper
The following masterpairs exist:
    prod

rbradfor@ronald:/tmp/local/scripts$ ./flipper --debug prod set write alpha
DEBUG:  Executing ssh -l flipper 192.168.2.187 "uname -s;/sbin/ifconfig -a"
Connection to 192.168.2.187 closed.
About to get IP info from beta
DEBUG:  Executing ssh -l flipper 192.168.2.181 "uname -s;/sbin/ifconfig -a"
Connection to 192.168.2.181 closed.
WARNING: write IP is not up on beta node.
WARNING: Won't attempt to take down write IP on beta node.
Master:  File master-log.000001 Position 4350
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
Slave: File master-log.000001 Position 2747
ERROR: Replication could not be synced.  Write IP won't be moved.
rbradfor@ronald:/tmp/local/scripts$ mysql -uupcodba -p -h192.168.2.187
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 43
Server version: 5.1.30-log MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> show master status;
+-------------------+----------+--------------+------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| master-log.000001 |     4350 |              |                  |
+-------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

mysql> exit
Bye
rbradfor@ronald:/tmp/local/scripts$ mysql -uupcodba -p -h192.168.2.181
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 23
Server version: 5.1.30-log MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.2.187
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-log.000001
          Read_Master_Log_Pos: 4350
               Relay_Log_File: slave-log.000002
                Relay_Log_Pos: 3395
        Relay_Master_Log_File: master-log.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 4350
              Relay_Log_Space: 3544
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
1 row in set (0.01 sec)

mysql> show master status;
+-------------------+----------+--------------+------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| master-log.000001 |     4350 |              |                  |
+-------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

mysql> exit
Bye
rbradfor@ronald:/tmp/local/scripts$ mysql -uupcodba -p -h192.168.2.187
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 44
Server version: 5.1.30-log MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.2.181
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-log.000001
          Read_Master_Log_Pos: 4350
               Relay_Log_File: slave-log.000002
                Relay_Log_Pos: 252
        Relay_Master_Log_File: master-log.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 2747
              Relay_Log_Space: 1001
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
1 row in set (0.00 sec)

mysql> slave stop;
Query OK, 0 rows affected (0.00 sec)

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State:
                  Master_Host: 192.168.2.181
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-log.000001
          Read_Master_Log_Pos: 4350
               Relay_Log_File: slave-log.000002
                Relay_Log_Pos: 252
        Relay_Master_Log_File: master-log.000001
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 2747
              Relay_Log_Space: 1001
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
1 row in set (0.00 sec)

mysql> slave start;
Query OK, 0 rows affected (0.00 sec)

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.2.181
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-log.000001
          Read_Master_Log_Pos: 4350
               Relay_Log_File: slave-log.000003
                Relay_Log_Pos: 252
        Relay_Master_Log_File: master-log.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 2747
              Relay_Log_Space: 1147
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
1 row in set (0.00 sec)





# at 2747
#081217 19:34:50 server id 181  end_log_pos 2793     User_var
SET @`pair_name`:=_latin1 0x70726F64 COLLATE `latin1_swedish_ci`/*!*/;
# at 2793
#081217 19:34:50 server id 187  end_log_pos 2903     Query    thread_id=21    exec_time=0    error_code=0
SET TIMESTAMP=1229542490/*!*/;
DELETE FROM meta.node WHERE masterpair = @pair_name
/*!*/;
# at 2903
#081217 19:34:50 server id 181  end_log_pos 2949     User_var
SET @`pair_name`:=_latin1 0x70726F64 COLLATE `latin1_swedish_ci`/*!*/;
# at 2949
#081217 19:34:50 server id 181  end_log_pos 2997     User_var
SET @`alpha_name`:=_latin1 0x616C706861 COLLATE `latin1_swedish_ci`/*!*/;
# at 2997
#081217 19:34:50 server id 181  end_log_pos 3051     User_var
SET @`alpha_ip`:=_latin1 0x3139322E3136382E322E313837 COLLATE `latin1_swedish_ci`/*!*/;
# at 3051
#081217 19:34:50 server id 181  end_log_pos 3097     User_var
SET @`beta_name`:=_latin1 0x62657461 COLLATE `latin1_swedish_ci`/*!*/;
# at 3097
#081217 19:34:50 server id 181  end_log_pos 3150     User_var
SET @`beta_ip`:=_latin1 0x3139322E3136382E322E313831 COLLATE `latin1_swedish_ci`/*!*/;
# at 3150
#081217 19:34:50 server id 187  end_log_pos 3358     Query    thread_id=21    exec_time=0    error_code=0
SET TIMESTAMP=1229542490/*!*/;
INSERT INTO meta.node (masterpair, node, name, value) VALUES
  (@pair_name, @alpha_name, 'ip', @alpha_ip),
  (@pair_name, @beta_name, 'ip', @beta_ip)
/*!*/;
# at 3358
#081217 19:34:50 server id 181  end_log_pos 3404     User_var
SET @`pair_name`:=_latin1 0x70726F64 COLLATE `latin1_swedish_ci`/*!*/;
# at 3404
#081217 19:34:50 server id 187  end_log_pos 3520     Query    thread_id=21    exec_time=0    error_code=0
SET TIMESTAMP=1229542490/*!*/;
DELETE FROM meta.masterpair WHERE masterpair = @pair_name
/*!*/;
# at 3520
#081217 19:34:50 server id 181  end_log_pos 3566     User_var
SET @`pair_name`:=_latin1 0x70726F64 COLLATE `latin1_swedish_ci`/*!*/;
# at 3566
#081217 19:34:50 server id 181  end_log_pos 3620     User_var
SET @`master_ip`:=_latin1 0x3139322E3136382E322E3731 COLLATE `latin1_swedish_ci`/*!*/;
# at 3620
#081217 19:34:50 server id 181  end_log_pos 3673     User_var
SET @`slave_ip`:=_latin1 0x3139322E3136382E322E3831 COLLATE `latin1_swedish_ci`/*!*/;
# at 3673
#081217 19:34:50 server id 181  end_log_pos 3726     User_var
SET @`netmask`:=_latin1 0x3235352E3235352E3235352E30 COLLATE `latin1_swedish_ci`/*!*/;
# at 3726
#081217 19:34:50 server id 181  end_log_pos 3781     User_var
SET @`broadcast`:=_latin1 0x3139322E3136382E322E323535 COLLATE `latin1_swedish_ci`/*!*/;
# at 3781
#081217 19:34:50 server id 187  end_log_pos 4350     Query    thread_id=21    exec_time=0    error_code=0
SET TIMESTAMP=1229542490/*!*/;
INSERT INTO meta.masterpair(masterpair, name, value) VALUES
  (@pair_name, 'read_ip', @master_ip),
  (@pair_name, 'write_ip', @slave_ip),
  (@pair_name, 'netmask', @netmask),
  (@pair_name, 'broadcast', @broadcast),
  (@pair_name, 'mysql_user', 'flipper'),
  (@pair_name, 'mysql_password', 'letmein'),
  (@pair_name, 'ssh_user', 'flipper'),
  (@pair_name, 'send_arp_command','/usr/lib/heartbeat/send_arp -p /tmp/send_arp.pid -i 100 -r 5 $sendarp_interface $sendarp_ip auto $sendarp_broadcast $sendarp_netmask')
/*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;


Reply all
Reply to author
Forward
0 new messages