Beegfs 7.4.6 on a server with JBOD Rocky 9.6--Help

[slakkar...@gmail.com]

Hello, 

Im in the process of setting up BeeGFS 7.4.6 on Rocky 9.6 as a POC. Running into multimode setup issues.

Created 6 ZFS pools with 22TB each. 

Here is my df -h looks like:    

bgmeta                 539G  144K  539G   1% /bgmeta
bgpool4                 22T  1.0M   22T   1% /bgpool4
bgpool3                 22T  1.0M   22T   1% /bgpool3
bgpool5                 22T  1.0M   22T   1% /bgpool5
bgpool6                 22T  1.0M   22T   1% /bgpool6
bgpool1                 22T  1.0M   22T   1% /bgpool1
bgpool2                 22T  1.0M   22T   1% /bgpool2
bgmeta/beegfs-meta     539G  131M  539G   1% /bgmeta/beegfs-meta
bgpool1/beegfs-stor-1   22T  1.0M   22T   1% /bgpool1/beegfs-stor-1
bgpool2/beegfs-stor-2   22T  1.0M   22T   1% /bgpool2/beegfs-stor-2
bgpool3/beegfs-stor-3   22T  1.0M   22T   1% /bgpool3/beegfs-stor-3
bgpool4/beegfs-stor-4   22T  1.0M   22T   1% /bgpool4/beegfs-stor-4
bgpool5/beegfs-stor-5   22T  1.0M   22T   1% /bgpool5/beegfs-stor-5
bgpool6/beegfs-stor-6   22T  1.0M   22T   1% /bgpool6/beegfs-stor-6

beegfs_nodev            22T  1.0M   22T   1% /mnt/beegfs--> problem

[beegfssrv11-rk9 bgpool2.d]# beegfs-ctl --listtargets --nodetype=storage --longnodes --pools --storagepools --state
TargetID     Reachability  Consistency   St. Pool   Cap. Pool   NodeID
========     ============  ===========   ========   =========   ======
       1           Online         Good          1      normal   beegfs-storage beegfssrv11-rk9.corp.com [ID: 1]
       2           Online         Good          1      normal   beegfs-storage beegfssrv11-rk9.corp.com [ID: 1]
       3           Online         Good          1      normal   beegfs-storage beegfssrv11-rk9.corpcom [ID: 1]
       4           Online         Good          1      normal   beegfs-storage beegfssrv11-rk9.corpcom [ID: 1]
       5           Online         Good          1      normal   beegfs-storage beegfssrv11-rk9.corp.com [ID: 1]
       6           Online         Good          1      normal   beegfs-storage beegfssrv11-rk9.corp.com [ID: 1]

--------------------------------------------------------------------------------------------------------------------------------

Here is the output from beegfs-df:

[beegfssrv11-rk9 bgpool2.d]# beegfs-df
METADATA SERVERS:
TargetID   Cap. Pool        Total         Free    %      ITotal       IFree    %
========   =========        =====         ====    =      ======       =====    =
       1      normal     538.7GiB     538.6GiB 100%     1129.6M     1129.6M 100%

STORAGE TARGETS:
TargetID   Cap. Pool        Total         Free    %      ITotal       IFree    %
========   =========        =====         ====    =      ======       =====    =
 [ERROR from beegfs-storage beegfssrv11-rk9.corp.com [ID: 1]: Unknown storage target]
       1      normal       0.0GiB       0.0GiB   0%        0.0M        0.0M   0%
       2      normal   22169.8GiB   22169.8GiB 100%    46493.4M    46493.4M 100%
 [ERROR from beegfs-storage beegfssrv11-rk9.corp.com [ID: 1]: Unknown storage target]
       3      normal       0.0GiB       0.0GiB   0%        0.0M        0.0M   0%
 [ERROR from beegfs-storage beegfssrv11-rk9.corp.com [ID: 1]: Unknown storage target]
       4      normal       0.0GiB       0.0GiB   0%        0.0M        0.0M   0%
 [ERROR from beegfs-storage beegfssrv11-rk9.corp.com [ID: 1]: Unknown storage target]
       5      normal       0.0GiB       0.0GiB   0%        0.0M        0.0M   0%
 [ERROR from beegfs-storage beegfssrv11-rk9.corp.com [ID: 1]: Unknown storage target]
       6      normal       0.0GiB       0.0GiB   0%        0.0M        0.0M   0%

----------------------------------------------------------------------------------------------------------------------------------------------

Running 1beegfs-storage service for each target with different ports.

[beegfssrv11-rk9 /]# grep -E "connStoragePortTCP|connStoragePortUDP|tuneRdmaPort" /etc/beegfs/bgpool*.d/beegfs-storage.conf
/etc/beegfs/bgpool1.d/beegfs-storage.conf:connStoragePortTCP = 8003
/etc/beegfs/bgpool1.d/beegfs-storage.conf:connStoragePortUDP = 8003
/etc/beegfs/bgpool2.d/beegfs-storage.conf:connStoragePortTCP = 8103
/etc/beegfs/bgpool2.d/beegfs-storage.conf:connStoragePortUDP = 8103
/etc/beegfs/bgpool3.d/beegfs-storage.conf:connStoragePortTCP = 8203
/etc/beegfs/bgpool3.d/beegfs-storage.conf:connStoragePortUDP = 8203
/etc/beegfs/bgpool4.d/beegfs-storage.conf:connStoragePortTCP = 8303
/etc/beegfs/bgpool4.d/beegfs-storage.conf:connStoragePortUDP = 8303
/etc/beegfs/bgpool5.d/beegfs-storage.conf:connStoragePortTCP = 8403
/etc/beegfs/bgpool5.d/beegfs-storage.conf:connStoragePortUDP = 8403
/etc/beegfs/bgpool6.d/beegfs-storage.conf:connStoragePortTCP = 8503
/etc/beegfs/bgpool6.d/beegfs-storage.conf:connStoragePortUDP = 8503

----------------------------------------------------------------------------------------------------------------------------

beegs-check-servers returns:

[beegfssrv11-rk9 bgpool2.d]# beegfs-check-servers
Management
==========
beegfssrv11-rk9.corp.com [ID: 1]: reachable at 10.48.18.210:8008 (protocol: TCP)

Metadata
==========
beegfssrv11-rk9.corp.com [ID: 1]: reachable at 10.48.18.210:8005 (protocol: RDMA)

Storage
==========
beegfssrv11-rk9.corp.com [ID: 1]: reachable at 10.48.18.210:8103 (protocol: RDMA)

 Any help is greatly appreciated.

Regards

Sudha

[Guan Xin]

Hi Sudha,

Looks like you want multiple storage targets instead of pools?

That would be perfectly fine without an enterprise license.

In either case, there is no need to run multiple beegfs-storage services on the same node unless for fault tolerance.

A single beegfs-storage service is sufficient to manage multiple targets.

Regards,

Guan

[Waltar]

Hi Sudha,

don't forget that a POC just makes sense with at least a minimum of 2 beegfs servers otherwise it would always be slower than 1 single server without beegfs.

Don't overcomplicate a distribute filesystem like beegfs because in case of any hw/sw problems the admin of goes easy into hell,

so use 1 storage service with 1 target for each 1 beegfs server and preferable 1 meta service also for smoth usage of directory cmd's like ls/du/find as number of files grow with available space.

Regards

Waltar

[Yiwei Guo]

pls upload your beegfs-storage log and beegfs-mgmtd log

--
You received this message because you are subscribed to the Google Groups "beegfs-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/fhgfs-user/beabb258-0de3-432d-8d2a-c4d06462a18cn%40googlegroups.com.

[S L]

Guan,

I need 6 storage targets that are coming from 6 zfs pools. 

I’m attempting hyper converged  setup( packing all BeeGFS services on to one server). 

Fault tolerance would be ideal. 

Regards 

Sudha

--

You received this message because you are subscribed to the Google Groups "beegfs-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/fhgfs-user/5a475e3c-0a5d-41af-83ee-786523fa80c1n%40googlegroups.com.