kyoto tycoon replication problems

[Dave Cottlehuber]

hi all

I've migrated our 2 node pair kyoto tycoon install to newer boxes, using
the following steps:

1. create new servers #3 and #4 and assign new sids
2. stop #1 to make a snapshot on disk
3. copy #1 snapshot to new server #3
4. set up replication from primary #1 -> new #3
5. delete old #2
6. set up replication from new #3 -> #4
7. switch db clients to use #3 from now on
8. delete #1

So far, this is all perfect.

The issue happens when enabling replication back from #4 -> #3 for the
usual master/master setup that I would put behind haproxy.

Old keys start appearing "back" in #3 again, as if it is replaying a
transaction log that somehow is present in the DB but isn't actively
used by Kyoto. What's really frustrating is that even keys actively used
on #3 then go "back in time" from data somehow found on node 3.

Any ideas what is wrong?

Should I re-use the sids from the old server pair to somehow fix this?

Config is following:

/usr/local/bin/ktserver \
-sid 1 \
-log /var/log/kyototycoon/kyoto.log \
-bgs /var/db/kyototycoon/snapshots \
-ulog /var/db/kyototycoon/updates \
-th 16 \
-ls \
-ulim 104857600 \
-rts /var/db/kyototycoon/timestamp.rts \
-bgsi 60 \
-host 127.0.0.1 -port 1978 \
-mhost 127.0.0.1 -mport 11979 \
/var/db/kyototycoon/iwmn.kch#opts=l#bnum=20000000#msiz=2g#dfunit=8#capsiz=1g

Thanks!
Dave