mysql 5.7 replication failures after upgrading rundeck from 4.10 to 5.3

[nos...@gmail.com]

Hey Team, thanks for reading.

For the rundeck database we use a master/replica pair of mysql 5.7 servers (running the percona binary).  We recently upgraded rundeck from 4.10.1 to 5.3.0.  Stuff mostly went smoothly, however now we are getting replication errors:

Worker 1 failed executing transaction 'ANONYMOUS' at source log mysql-bin.163526, end_log_pos 235490727; Could not execute Delete_rows event on table rundeck3.workflow_step; Cannot delete or update a parent row: a foreign key constraint fails (`rundeck3`.`workflow_workflow_step`, CONSTRAINT `FKrln65du8n7hwu3qbinnea748g` FOREIGN KEY (`workflow_step_id`) REFERENCES `workflow_step` (`id`)), Error_code: 1451; handler error HA_ERR_ROW_IS_REFERENCED; the event's source log mysql-bin.163526, end_log_pos 235490727
Worker 1 failed executing transaction 'ANONYMOUS' at source log mysql-bin.163526, end_log_pos 805148250; Could not execute Delete_rows event on table rundeck3.workflow_workflow_step; Can't find record in 'workflow_workflow_step', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's source log mysql-bin.163526, end_log_pos 805148250

I am guessing Delete_rows is related to the execution cleanup routine we have enabled for our projects, we do need to run this routine, but for now we disabled them.

bin_log format is ROW, its always possible weirdness can cause data inconsistency on the replica, but we never saw this before the upgrade.

I would appreciate any input on this.

[rac...@rundeck.com]

Hi!

Rundeck isn’t designed to work natively with database replication environments. But after checking this: Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND probably you should take a look at this.

Greetings!

​