Help needed with aculturation tutorial

[Kevin Toppenberg]

I have been trying to follow exactly the steps given in the acculturation guide.  It directs me to set up three cities (Paris, Melbourne, Santiago).   I have been able to successfully have posts from Paris propagate out  to Melbourne and Santiago, with Paris being primary. 

Then, the instructions simulated a crash of Paris, by powering off the machine, and we set Melbourne as primary.  That was successful, and posts to Melbourne propigate to Santiago.  

I then brought Paris back up and did the rollback command as directed, that seemed to succeed as below. 

ydbuser@paris:~/jnlex$ mupip journal -rollback -backward -fetchresync=3000 -losttrans=/home/ydbuser/jnlex/Unreplic_Trans_Report_`date +%Y%m%d%H%M%S`.txt "*"
%YDB-I-MUJNLSTAT, Initial processing started at Tue Feb  3 17:23:16 2026
%YDB-I-RLBKJNLNOBIMG, Journal file /home/ydbuser/jnlex/%ydbjnlf.mjl has NOBEFORE_IMAGE journaling
%YDB-I-RLBKLOSTTNONLY, ROLLBACK will only create a lost transaction file (database and journal files will not be modified)
%YDB-I-MUJNLSTAT, FETCHRESYNC processing started at Tue Feb  3 17:23:16 2026
Tue Feb  3 17:23:16 2026 : Assuming primary supports multisite functionality. Connecting using multisite communication protocol.
Tue Feb  3 17:23:16 2026 : Waiting for a connection...
Tue Feb  3 17:23:17 2026 : Connection established, using TCP send buffer size 87040 receive buffer size 131072
Tue Feb  3 17:23:17 2026 : Connection information:: Local: ::ffff:10.0.2.15:3000 Remote: ::ffff:10.0.2.2:55150
Tue Feb  3 17:23:17 2026 : Sending REPL_FETCH_RESYNC message with seqno 4 [0x4]
Tue Feb  3 17:23:17 2026 : Source and Receiver sides have same endianness
Tue Feb  3 17:23:17 2026 : Remote side source log file path is /home/ydbuser/jnlex/source_paris_20260202:12:13:45.log; Source Server PID = 1995
Tue Feb  3 17:23:17 2026 : Received REPL_NEED_INSTINFO message from primary instance [melbourne]
Tue Feb  3 17:23:17 2026 : Sending REPL_INSTINFO message
Tue Feb  3 17:23:17 2026 : Received REPL_NEED_HISTINFO message for Seqno 4 [0x4]
Tue Feb  3 17:23:17 2026 : Sending REPL_HISTINFO message with seqno 1 [0x1]
Tue Feb  3 17:23:17 2026 : History sent : Start Seqno = 1 [0x1] : Stream Seqno = 0 [0x0] : Root Primary = [paris] : Cycle = [1] : Creator pid = 649 : Created time = 1769806178 [0x697d1962] : History number = 0 : Prev History number = -1 : Stream # = 0 : History type = 1
Tue Feb  3 17:23:17 2026 : Received REPL_RESYNC_SEQNO message
Tue Feb  3 17:23:17 2026 : Received RESYNC SEQNO is 4 [0x4]
%YDB-I-MUJNLSTAT, Backward processing started at Tue Feb  3 17:23:17 2026
%YDB-I-RESOLVESEQNO, Resolving until sequence number 4 [0x0000000000000004]
%YDB-I-MUJNLSTAT, Forward processing started at Tue Feb  3 17:23:17 2026
%YDB-I-FILENOTCREATE, Lost transactions extract file /home/ydbuser/jnlex/Unreplic_Trans_Report_20260203172316.txt not created
%YDB-S-JNLSUCCESS, Show successful
%YDB-S-JNLSUCCESS, Verify successful
%YDB-S-JNLSUCCESS, Rollback successful
%YDB-I-MUJNLSTAT, End processing at Tue Feb  3 17:23:17 2026

But when I follow the next step, it is not working, and the changes made to Melbourne are not found in Paris.  

ydbuser@paris:~/jnlex$ ./replicating_start
Tue Feb  3 17:26:24 2026 : Initiating START of source server for secondary instance [dummy]
Tue Feb  3 17:26:24 2026 : Source server startup failed. See source server log file
Tue Feb  3 17:26:24 2026 : Journal pool shared memory removed
Tue Feb  3 17:26:24 2026 : Journal pool semaphore removed
%YDB-E-NOJNLPOOL, No journal pool info found in the replication instance of /home/ydbuser/jnlex/paris.repl
ydbuser@paris:~/jnlex$ ydb

YDB>zwr ^Weather
^Weather("Paris",67600,79196)="Rainy"
^Weather("Paris",67600,79293)="Snowy"
^Weather("Paris",67600,79362)="Sunny"
<--------------  NOTE: missing entries here.  

YDB>h
ydbuser@paris:~/jnlex$ 

ydbuser@paris:~/jnlex$ cat ./replicating_start
#!/bin/sh
$ydb_dist/mupip replicate -source -start -passive -instsecondary=dummy -buffsize=1048576 -log=$ydb_dir/source_dummy.log
$ydb_dist/mupip replicate -receive -start -listenport=3000 -buffsize=1048576 -log=$ydb_dir/receive_`date +%Y%m%d:%H:%M:%S`.log


I don't know how to go forward.  I asked ChatGPT but it was confused and giving bad recommendations such as manually copying the database file over and recreating the .repl file.  

Can anyone help?

Thanks in advance,

Kevin

[Kevin Toppenberg]

Additional info

The error message says, "Tue Feb  3 17:26:24 2026 : Source server startup failed. See source server log file"  On the new primary (Melbourne) there is a file named source_paris_20260202:12:13:45.log.   I looked at this file and it is filled with entries like this. 

Wed Feb  4 03:50:01 2026 : Waiting for REPL_START_JNL_SEQNO or REPL_FETCH_RESYNC message
Wed Feb  4 03:50:01 2026 : Connection reset while receiving restart SEQNO. Status = 104 ; Connection reset by peer
Wed Feb  4 03:50:02 2026 : Connect hard tries count = 5, Connect hard tries period = 500 milliseconds
Wed Feb  4 03:50:02 2026 : Connected to secondary, using TCP send buffer size 425984 receive buffer size 131072
Wed Feb  4 03:50:02 2026 : Connection information:: Local: 10.0.2.15:39120 Remote: 10.0.2.2:5000

When I due a tail -f on this file, it is adding an entry about every 1 second, and they all look just like this.  There was no change to this behavior (the constant growth) when I tried the ./replication_start on the paris server.  So apparently the signal is not getting through somehow(?)

Again, can anyone help?  Thanks in advance. 

Kevin T

[K.S. Bhaskar]

Kevin, I'd really like to help, but I have some urgent matters to attend to before I leave for a trip. So, yes, I am ignoring you.

Apologetically

- Bhaskar

[Kevin Toppenberg]

Bhaskar,  Thanks for letting me know what is going on.  I'll keep plugging away and perhaps sometime in the future, after your trip, you can help.  Thank you!

Kevin