Storage Targets BuddyMirror - both targets "Needs-resync"

[Pinkesh Valdria]

I am trying Buddy mirror for the first time.  After install and configuration.  I rebooted all nodes for SE linux disable to be effective.  Both Storage targets have entered this "Needs-resync" state.  Looking for guidance on how can get them back up 

correctly?  

[root@client-1 ~]# beegfs-ctl --listtargets --nodetype=storage --state

TargetID     Reachability  Consistency   NodeID

========     ============  ===========   ======

     111           Online Needs-resync        1

     211           Online Needs-resync        2

[root@client-1 ~]#

111 is primary and 211 is secondary. 

I have 2 storage nodes with 1 storage target on each.  

[root@client-1 ~]# beegfs-ctl --startresync --nodetype=storage --targetid=111

Resync request sent.

[root@client-1 ~]#

I tried the following:  

[root@client-1 ~]# beegfs-ctl --startresync --nodetype=storage --targetid=111

Resync request sent.

[root@client-1 ~]# beegfs-ctl --resyncstats --nodetype=storage  --targetid=111

Couldn't find buddy of target ID: 111. Is it the target ID of the secondary?

[root@client-1 ~]# beegfs-ctl --resyncstats --nodetype=storage  --targetid=211

Job state: Not started

# of discovered dirs: 0

# of discovered files: 0

# of dir sync candidates: 0

# of file sync candidates: 0

# of synced dirs: 0

# of synced files: 0

# of dir sync errors: 0

# of file sync errors: 0

[root@client-1 ~]#

i see this in the Storage logs on primary target: 

(3) Apr27 22:12:46 XNodeSync [Decide resync] >> Target 111: Setting new target state: Needs-resync (old state: Needs-resync)

(2) Apr27 22:12:46 XNodeSync [InternodeSyncer.cpp:415] >> Target was a primary target and needs a resync. Waiting until it is marked offline on all clients. targetID: 111; remainingMS: 37996

(3) Apr27 22:13:16 XNodeSync [Decide resync] >> Target 111: Setting new target state: Needs-resync (old state: Needs-resync)

(2) Apr27 22:13:16 XNodeSync [InternodeSyncer.cpp:415] >> Target was a primary target and needs a resync. Waiting until it is marked offline on all clients. targetID: 111; remainingMS: 7995

(3) Apr27 22:13:46 XNodeSync [Decide resync] >> Target 111: Setting new target state: Needs-resync (old state: Needs-resync)

[Honghui Ding]

You can set the status of one target as good, then the sync will restart.

[Pinkesh Valdria]

Thanks Honghui, let me try that.   

--
You received this message because you are subscribed to the Google Groups "beegfs-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/cab0eb09-c854-424b-951d-4a7ef8514fd1%40googlegroups.com.

--

Thanks,

Pinkesh Valdria

Singapore: +65 8932-3639

USA: +1 425-205-7834

USA: +1 206-234-4314 (cancelled)

[Digital Wave Finance]

Stupid question: How can the target state be changed to "Good"?

[Sven Breuner]

Hi,

see "beegfs-ctl --setstate --help" to manually change the state.

All the best
Sven

--
Sven Breuner
Field CTO
Excelero

--
You received this message because you are subscribed to the Google Groups "beegfs-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/e063fba4-df31-4994-8c7b-7f6cb6d8eb5an%40googlegroups.com.

[John Desantis]

Hello,

A word of caution so that you do not lose any data!

Make sure that you peruse the logs to see which target was the primary before it ended up requiring a resync.  If you inadvertently set the state to "good" on the secondary, you run the risk of data loss. 

We lost a large swath of metadata during a similar procedure because the secondary was set to "good", which overwrote ~12 hours worth of data on the primary. 

John DeSantis