After power cut, metadata servers immediately segfault

[NCTGS]

We had a power issue that caused a RAID issue on one storage node.. after getting it back to health we found that beegfs-meta will immediately segfault when started. We can't seem to find why. There's some mention of quotas, but we don't have quotas enabled at all.

Verbose logs from beefs-meta:

(4) Jun24 08:29:16 Main [App] >> Allowed interfaces: eth2
(3) Jun24 08:29:16 Main [App.cpp:641] >> Limiting number of xattrs per inode.
(3) Jun24 08:29:16 Main [App] >> Root directory loaded.
(1) Jun24 08:29:16 Main [App] >> Root metadata server (by possession of root directory): 1
(4) Jun24 08:29:16 Main [App] >> Disposal directory loaded.
(4) Jun24 08:29:16 Main [App] >> Mirrored disposal directory loaded.
(3) Jun24 08:29:16 Main [RegDGramLis] >> Listening for UDP datagrams: Port 8005
(1) Jun24 08:29:16 Main [App] >> Waiting for beegfs-mgmtd@hts-dmstore70:8008...
(2) Jun24 08:29:32 RegDGramLis [Heartbeat incoming] >> New node: beegfs-mgmtd hts-dmstore70 [ID: 1]
(4) Jun24 08:29:32 RegDGramLis [Heartbeat incoming] >> Number of nodes: Meta: 0; Storage: 0
(4) Jun24 08:29:32 RegDGramLis [RegDGramLis] >> Component stopped.
(3) Jun24 08:29:32 Main [RegDGramLis] >> Listening for UDP datagrams: Port 8005
(2) Jun24 08:29:32 Main [Register node] >> Node registration successful.
(4) Jun24 08:29:32 Main [InternodeSyncer.cpp:623] >> Starting node list sync.
(4) Jun24 08:29:32 Main [NodeConn (acquire stream)] >> Establishing new TCP connection to: beegfs...@10.11.15.80:8008
(3) Jun24 08:29:32 Main [NodeConn (acquire stream)] >> Connected: beegfs...@10.11.15.80:8008 (protocol: TCP)
(2) Jun24 08:29:32 Main [printSyncResults] >> Nodes added (sync results): 1 (Type: beegfs-meta)
(2) Jun24 08:29:32 Main [printSyncResults] >> Nodes added (sync results): 10 (Type: beegfs-storage)
(4) Jun24 08:29:32 Main [InternodeSyncer.cpp:704] >> Syncing target mappings.
(4) Jun24 08:29:32 Main [InternodeSyncer.cpp:723] >> Syncing storage pools.
(4) Jun24 08:29:32 Main [InternodeSyncer.cpp:756] >> Downloading target states and buddy groups
(4) Jun24 08:29:32 Main [InternodeSyncer.cpp:296] >> Downloading capacity pools. Pool type: Meta
(4) Jun24 08:29:32 Main [InternodeSyncer.cpp:296] >> Downloading capacity pools. Pool type: Meta buddies
(4) Jun24 08:29:32 Main [InternodeSyncer.cpp:376] >> Starting state update.
(4) Jun24 08:29:32 Main [InternodeSyncer.cpp:401] >> Beginning target state update...
(0) Jun24 08:29:32 Main [PThread.cpp:99] >> Received a SIGSEGV. Trying to shut down...
(1) Jun24 08:29:32 Main [PThread::signalHandler] >> Backtrace:
1: /opt/beegfs/sbin/beegfs-meta(_ZN7PThread13signalHandlerEi+0x47) [0x6fd6d7]
2: /lib64/libc.so.6(+0x362f0) [0x7f42440082f0]
3: /opt/beegfs/sbin/beegfs-meta(_ZN18ExceededQuotaStore19updateExceededQuotaEPSt4listIjSaIjEE13QuotaDataType14QuotaLimitType+0x1e) [0x6f249e]
4: /opt/beegfs/sbin/beegfs-meta(_ZN15InternodeSyncer29downloadAllExceededQuotaListsESt10shared_ptrI11StoragePoolE+0x169) [0x59aaa9]
5: /opt/beegfs/sbin/beegfs-meta(_ZN15InternodeSyncer29downloadAllExceededQuotaListsERKSt6vectorISt10shared_ptrI11StoragePoolESaIS3_EE+0xb2) [0x59b482]
6: /opt/beegfs/sbin/beegfs-meta(_ZN3App16downloadMgmtInfoER22TargetConsistencyState+0x1fa) [0x487faa]
7: /opt/beegfs/sbin/beegfs-meta(_ZN3App9runNormalEv+0x137) [0x48dec7]
8: /opt/beegfs/sbin/beegfs-meta(_ZN3App3runEv+0x52) [0x48e442]
9: /opt/beegfs/sbin/beegfs-meta(_ZN7PThread9runStaticEPv+0xfe) [0x48279e]
10: /opt/beegfs/sbin/beegfs-meta(_ZN7Program4mainEiPPc+0x49) [0x47f8e9]
11: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f4243ff4445]
12: /opt/beegfs/sbin/beegfs-meta() [0x482095]
(0) Jun24 08:29:32 Main [PThread.cpp:135] >> Received a SIGABRT. Trying to shut down...
(1) Jun24 08:29:32 Main [PThread::signalHandler] >> Backtrace:
1: /opt/beegfs/sbin/beegfs-meta(_ZN7PThread13signalHandlerEi+0x47) [0x6fd6d7]
2: /lib64/libc.so.6(+0x362f0) [0x7f42440082f0]
3: /lib64/libc.so.6(gsignal+0x37) [0x7f4244008277]
4: /lib64/libc.so.6(abort+0x148) [0x7f4244009968]
5: /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165) [0x7f4244b33ac5]
6: /lib64/libstdc++.so.6(+0x5ea36) [0x7f4244b31a36]
7: /lib64/libstdc++.so.6(+0x5d9e9) [0x7f4244b309e9]
8: /lib64/libstdc++.so.6(__gxx_personality_v0+0x564) [0x7f4244b31654]
9: /lib64/libgcc_s.so.1(+0xf903) [0x7f42445ca903]
10: /lib64/libgcc_s.so.1(_Unwind_RaiseException+0xfb) [0x7f42445cac9b]
11: /lib64/libstdc++.so.6(__cxa_throw+0x66) [0x7f4244b31c76]
12: /opt/beegfs/sbin/beegfs-meta(_ZN7PThread13signalHandlerEi+0x296) [0x6fd926]
13: /lib64/libc.so.6(+0x362f0) [0x7f42440082f0]
14: /opt/beegfs/sbin/beegfs-meta(_ZN18ExceededQuotaStore19updateExceededQuotaEPSt4listIjSaIjEE13QuotaDataType14QuotaLimitType+0x1e) [0x6f249e]
15: /opt/beegfs/sbin/beegfs-meta(_ZN15InternodeSyncer29downloadAllExceededQuotaListsESt10shared_ptrI11StoragePoolE+0x169) [0x59aaa9]
16: /opt/beegfs/sbin/beegfs-meta(_ZN15InternodeSyncer29downloadAllExceededQuotaListsERKSt6vectorISt10shared_ptrI11StoragePoolESaIS3_EE+0xb2) [0x59b482]
17: /opt/beegfs/sbin/beegfs-meta(_ZN3App16downloadMgmtInfoER22TargetConsistencyState+0x1fa) [0x487faa]
18: /opt/beegfs/sbin/beegfs-meta(_ZN3App9runNormalEv+0x137) [0x48dec7]
19: /opt/beegfs/sbin/beegfs-meta(_ZN3App3runEv+0x52) [0x48e442]
20: /opt/beegfs/sbin/beegfs-meta(_ZN7PThread9runStaticEPv+0xfe) [0x48279e]
21: /opt/beegfs/sbin/beegfs-meta(_ZN7Program4mainEiPPc+0x49) [0x47f8e9]
22: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f4243ff4445]
23: /opt/beegfs/sbin/beegfs-meta() [0x482095]
(5) Jun24 08:29:38 RegDGramLis [RegDGramLis] >> Received a message that is invalid within the current context from: 10.11.15.80; type: RefreshTargetStates (1051)

[Waltar]

The quota messages are maybe just from quota checks before metadata service in online. 

Before trying to repair by a beegfs fsck always try to repair with the tools of the underlying filesystem first !!

You don't give your hw/sw configuration under the metadata service until now.

As mentioned earlier ext4 and zfs are quiet sensitive to a power outage if they were written too that time ... so I asume it's one of them ?

Try fsck.ext4 or zpool scrub <metadata-poolname> depending on you config and then try to restart this meta service again.

A USV is always a good thing to all beegfs servers as if having a distributed filesystem which is even more sensitive to errors to minimize such case of problems.