Low Bandwith / Slow speed

42 views
Skip to first unread message

Farhad Saki

unread,
Aug 28, 2024, 8:19:03 PMAug 28
to beegfs-user
Hello,

i have a setup with a beegfs-mgmt and beegfs-metadata servers installed as VM's while the metadata server has a whole nvme disk for his own use.

The Storage Server is hosted on a bare metal machine equipped with 24x NVME's drive till yet i only have used one of the NVMe's for the storage pool.

I now have created a 100GB File for test purposes and copy this with pv 100GBfile >> /mnt/beegfs/100GBfile

unfortunately i only get speeds around 400MiB/s, when i do the same test on my NFSoRDMA mount path i can achieve speeds up to 2GiB/s.

I have installed the newest MLNX_OFED Drivers also and i am using 200GB/s Connect 6x cards.

I couldn't figure out why the speed is much slower than NFSoRDMA because if i copy fresh new files it couldn't be the caching which NFS is using.

Thanks in advance

Kind Regards

Kreefd

Waltar

unread,
Aug 30, 2024, 3:33:08 AMAug 30
to beegfs-user
What does "beegfs-check-servers" say ?

Waltar

unread,
Aug 30, 2024, 3:36:56 AMAug 30
to beegfs-user
Why have beegfs-metadata in a vm when you cannot move the vm as it's connected to your local nvme ? 
So it makes no sense to use the vm and run the meta direct onto that node nativ as it's even slower through the additional virtualisation layer with even restricted resources.

Farhad Saki

unread,
Aug 30, 2024, 6:36:24 PMAug 30
to beegfs-user
beegfs-check-servers says:

Management
==========
bee-mgmt [ID: 1]: reachable at 192.168.1.11:18463 (protocol: TCP)

Metadata
==========
bee-meta [ID: 2]: reachable at 192.172.1.101:8005 (protocol: RDMA)

Storage
==========
beegfs-storage1 [ID: 2]: reachable at 192.172.1.13:8003 (protocol: RDMA)


What i meant with the NVMe is that there is a kvmstorage on the proxmox host exclusively for the beegfs-metadata i can migrate the VM to other hosts etc.

Farhad Saki

unread,
Aug 30, 2024, 6:36:28 PMAug 30
to beegfs-user

What i mean is that the beegfs-metadata has the capacity/resources of the NVMe for it's own i can move the meta data server.


beegfs-check-servers says:

Management
==========
bee-mgmt [ID: 1]: reachable at 192.168.1.11:18463 (protocol: TCP)

Metadata
==========
bee-meta [ID: 2]: reachable at 192.172.1.101:8005 (protocol: RDMA)

Storage
==========
beegfs-storage1 [ID: 2]: reachable at 192.172.1.13:8003 (protocol: RDMA)

Waltar

unread,
Aug 31, 2024, 2:49:03 AMAug 31
to beegfs-user
Don't know your deployment but everythink (nfsordma+beegfs) looks too slow as you say you use 24 nvme's plus a single nvme.
When you compare writing a single 100GB file with nfsordma with beegfs which do "default" 512k chunks you write >200000 object-files for that,
so there is still a lot of extra inode allocations and communication overhead which is normally spread over too a couple of hosts/daemons
and 1 single storage pool is expected to be everytime slower.
You should look for your bottlenecks cpu, I/O, network from the ground and last go to beegfs to config nr daemons, (chunk+ stripe are useless in your single target case).

Waltar

unread,
Aug 31, 2024, 2:59:51 AMAug 31
to beegfs-user
Btw. a distributed filesystem - like beegfs - is designed to fullfill many file requests concurrently and not only 1 file/time.
Reply all
Reply to author
Forward
0 new messages