BeeGFS Client fail to mount after moving management service

[Moh Zidni Mubarok]

Hi Expert,

Setup:

Node 1 :  meta (buddy mirror), storage (buddy mirror) ,client

Node 2 : Mgmt, meta (buddy mirror),storage (buddy mirror),client

I followed this guide https://doc.beegfs.io/latest/advanced_topics/node_mamagement.html

then I delete node 2 after copying over mgmt files, now the setup is like this:

Node 1 :  mgmt, meta (buddy mirror), storage (buddy mirror) ,client

I managed to start mgmt, meta, storage but not the client.

I can run beegfs-ctl list nodes etc

root@node01 management]# beegfs-check-servers 

Management

==========

node01 [ID: 1]: reachable at 10.10.1.201:8008 (protocol: TCP)

Metadata

==========

node02 [ID: 1]: UNREACHABLE

node01 [ID: 2]: reachable at 10.10.1.201:8005 (protocol: TCP)

Storage

==========

node02 [ID: 1]: UNREACHABLE

node01 [ID: 2]: reachable at 10.10.1.201:8003 (protocol: TCP)

[root@node01 management]# beegfs-ctl --listmirrorgroups --nodetype=storage

     BuddyGroupID   PrimaryTargetID SecondaryTargetID

     ============   =============== =================

              100                 2                 1

Log for beegfs-client:

Sep  6 18:04:57 node01 kernel: beegfs: mount(7369): NodeConn (acquire stream): Connected: beegf...@10.10.1.201:8005 (protocol: TCP)

Sep  6 18:04:57 node01 kernel: beegfs: beegfs_XNodeSyn(7371): Init: Init complete.

Sep  6 18:04:57 node01 kernel: beegfs: mount(7369): NodeConn (acquire stream): Connected: beegfs-...@10.10.1.201:8003 (protocol: TCP)

Sep  6 18:04:57 node01 kernel: beegfs: mount(7369): Remoting (stat storage targets): Error target (storage): 1; Msg: Communication error

Sep  6 18:04:57 node01 kernel: beegfs: mount(7369): Mount sanity check: Retrieval of storage server free space info failed. Are the storage servers running and registered at the management daemon? Did you remove a storage target directory on a server? (Error: Communication error)

Sep  6 18:04:57 node01 kernel: beegfs: mount(7369): Mount sanity check failed. Canceling mount. (Log file may provide additional information. Check can be disabled with sysMountSanityCheckMS=0 in the config file.)

Sep  6 18:04:57 node01 kernel: beegfs: mount(7369): App (stop components): Stopping components...

Sep  6 18:04:57 node01 beegfs-mgmtd: (2) Sep06 18:04:57 Worker3 [RemoveNodeMsgEx.cpp:66] >> Node removed. node: beegfs-client 1CC9-6135E7C9-node01 [ID: 70]

Sep  6 18:04:57 node01 kernel: beegfs: beegfs_XNodeSyn(7371): Deregistration: Node deregistration successful.

Sep  6 18:04:58 node01 beegfs-mgmtd: (2) Sep06 18:04:58 Worker3 [Node registration] >> New node: beegfs-client F65-6135E7CA-hbcn01 [ID: 71]; Source: 10.10.1.101:41654

Sep  6 18:04:58 node01 beegfs-mgmtd: (2) Sep06 18:04:58 Worker2 [RemoveNodeMsgEx.cpp:66] >> Node removed. node: beegfs-client F65-6135E7CA-hbcn01 [ID: 71]

Sep  6 18:04:59 node01 kernel: beegfs: mount(7369): App (wait for component termination): Still waiting for this component to stop: beegfs_AckMgr

Sep  6 18:05:00 node01 kernel: beegfs: mount(7369): App (wait for component termination): Component stopped: beegfs_AckMgr

Sep  6 18:05:00 node01 kernel: beegfs: mount(7369): App (stop): All components stopped.

Sep  6 18:05:00 node01 beegfs-client: mount: mount beegfs_nodev on /mnt/beegfs failed: Operation canceled

Sep  6 18:05:00 node01 systemd: beegfs-client.service: main process exited, code=exited, status=32/n/a

Sep  6 18:05:00 node01 systemd: Failed to start Start BeeGFS Client.

Sep  6 18:05:00 node01 systemd: Unit beegfs-client.service entered failed state.

Sep  6 18:05:00 node01 systemd: beegfs-client.service failed.

Is there anything i need to check?