Slow NVME RAID0 performance

[ekha...@gmail.com]

I tried to test the ESOS v3 with 4 NVME drives in RAID0.  

First of all - I tested these drives under Windows 2022 (just physically installed them into server Dell R730 with Windows) and got the next results:

Single drive:

[Read]
  SEQ    1MiB (Q=  8, T= 4):  7028.650 MB/s [   6703.0 IOPS] <  4766.51 us>
  RND    4KiB (Q= 32, T= 8):   371.908 MB/s [  90797.9 IOPS] <  2816.88 us>

[Write]
  SEQ    1MiB (Q=  8, T= 4):  6922.579 MB/s [   6601.9 IOPS] <  4816.81 us>
  RND    4KiB (Q= 32, T= 8):   266.549 MB/s [  65075.4 IOPS] <  3927.88 us>

RAID0 from 4 drives:

[Read]
  SEQ    1MiB (Q=  8, T= 4):  9906.622 MB/s [   9447.7 IOPS] <  3379.70 us>
  RND    4KiB (Q= 32, T= 8):  1870.168 MB/s [ 456584.0 IOPS] <   543.58 us>

[Write]
  SEQ    1MiB (Q=  8, T= 4):  4929.566 MB/s [   4701.2 IOPS] <  6755.59 us>
  RND    4KiB (Q= 32, T= 8):  1461.543 MB/s [  356822.0 IOPS  ] <   694.08 us>

After this I installed these NVME drives on ESOS server (same Dell R730 server), created vdisk_fileio block device and connected it via dual Infiniband 56GB (SCSI over Infiniiband protocol) to Windows server and perform the same tests, single drive and MD0 RAID0 from 4 drives. I got the next performance:

Single drive:

[Read]
  SEQ    1MiB (Q=  8, T= 1):  6483.776 MB/s [   6183.4 IOPS] <  1291.25 us>
  RND    4KiB (Q= 32, T=16):   266.630 MB/s [  65095.2 IOPS] <  7587.72 us>

[Write]
  SEQ    1MiB (Q=  8, T= 1):  1277.712 MB/s [   1218.5 IOPS] <  6515.94 us>
  RND    4KiB (Q= 32, T=16):   130.528 MB/s [  31867.2 IOPS] < 16006.72 us>

RAID0 from 4 drives:

[Read]
  SEQ    1MiB (Q=  8, T= 1):  6462.454 MB/s [   6163.1 IOPS] <  1293.42 us>
  RND    4KiB (Q= 32, T=16):   276.984 MB/s [  67623.0 IOPS] <  7326.12 us>

[Write]
  SEQ    1MiB (Q=  8, T= 1):  1782.727 MB/s [   1700.1 IOPS] <  4668.70 us>
  RND    4KiB (Q= 32, T=16):   122.658 MB/s [  29945.8 IOPS] < 17047.25 us>

I attached table for better view of results.

Read SEQ 6452MB/sec is Ok, it close to PCI3 speed of Infiniband cards and I am Ok with this number.  It even shows that Infiniband connection is fast enough.

But why Write SEQ so slow -  1782MB/s vs 4929MB/s for RAID and 1277MB/s vs 6922 MB/s for single drive?

And why RND so slow - 456584 IOPS/356822 IOPS on Windows vs  67623 IOPS/29945 IOPS on ESOS?

Any ideas how to got the better performance from ESOS with NVME drives?

Thanks

 

[Marc Smith]

I would start with measuring performance of the block devices (first
NVMe drives by themselves, then the MD RAID array) using 'fio'
directly on the ESOS server. Then if those numbers look good, consider
tuning in the SCST layer. If the NVMe drives and/or MD RAID numbers
look poor in Linux (ESOS), then there is work there to trace/tune
(those block layers/drivers). Using the SCST 'vdisk_blockio' handler
may be a better fit for this type of setup.

--Marc


>
> Thanks
>
>
> --
> You received this message because you are subscribed to the Google Groups "esos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/20a68712-ab93-4988-8bb2-8a3d95ff63d8n%40googlegroups.com.

[ekha...@gmail.com]

Hi

Thanks for response.

1)  I can't use vdisk_blockio, this handler has a problem with multipath with ESXi. ESXi see different paths as different devices.. Vdisk_fileio works properly with multipath.

2) At least Write SEQ problem looks like related to MD.  I configured  vdisk_fileio handler with write cache in memory and got 5795 MB/s write speed with test file size smaller than memory size in ESOS.  It means that Infiniband and SCST is fast enough against cache. And only when ESOS has to begin to write to MD (test file size bigger than memory) the speed is going down.

3) Local speed test with FIO is a good idea. Let you know.

Thanks  

[Marc Smith]

On Sun, Feb 12, 2023 at 3:42 PM ekha...@gmail.com <ekha...@gmail.com> wrote:
>
> Hi
> Thanks for response.
> 1) I can't use vdisk_blockio, this handler has a problem with multipath with ESXi. ESXi see different paths as different devices.. Vdisk_fileio works properly with multipath.

I use 'vdisk_blockio' with ESXi initiators, and many, many others do
so as well ('vdisk_blockio' and 'vdisk_fileio' are nearly identical).
So you may want to check your configuration.

> To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/a24d8e9f-9468-471c-81b1-1195d020e180n%40googlegroups.com.

[ekha...@gmail.com]

Hi

Regarding  vdisk_blockio - you are right, the problem was related to Esos3. In Esos4 it works properly

[ekha...@gmail.com]

Regarding the performance. I performed few more tests.

In few words - looks like problem is in SCST.

The local fio test for RAID0 and for single drive shows the good numbers.

But for ESOS storage situation is different. 

To eliminate the influence of RAID engine, I decided to do a test on a single disk.

As you can see below - the SEQ read/write results are very close to local.

But RND read/write is about 10 times slower for vdisk_blockio and about 6 times slower for vdisk_fileio. 

Local single nvme:

 [Read]
  SEQ    1MiB (Q=  8, T= 1):  3 338 MB/s
  RND    4KiB (Q= 32, T=16):  366 457 IOPS
[Write]
  SEQ    1MiB (Q=  8, T= 1):  1 280 MB/s 
  RND    4KiB (Q= 32, T=16):  257 447 IOPS] 

ESOS  vdisk_blockio on single nvme  ->SCST->Infiniband 56GB:

 [Read]
  SEQ    1MiB (Q=  8, T= 1):  3 113 MB/s
  RND    4KiB (Q= 32, T=16):  36 076 IOPS
[Write]
  SEQ    1MiB (Q=  8, T= 1):  1 276 MB/s 
  RND    4KiB (Q= 32, T=16):  35 884 IOPS]

ESOS  vdisk_fileio single nvme  ->SCST->Infiniband 56GB:

 [Read]
  SEQ    1MiB (Q=  8, T= 1):  3 205 MB/s
  RND    4KiB (Q= 32, T=16):  53 253 IOPS
[Write]
  SEQ    1MiB (Q=  8, T= 1):  1 328 MB/s 
  RND    4KiB (Q= 32, T=16):  44 488 IOPS]

Is it possible to improve random read/write of ESOS? Any ideas?

What about to upgrade SCST from 3.5 to last 3.7?

Regards

E.