nvme_core multipath

[Nicholas Papadakos]

hello,

i am trying to build a VMware iscsi datastore so i can replace my quantastor because :

a) i cant use software raid with nvme devices(or any device)

b) i cant boot from USB 

c) i want to play with HA and cant afford the licensing cost.

d) esos its just awesome. I love the old school tui :) 

i setup 2 similar setups with the same hardware and used 4 esxi as initiators.

Setup A 

HP DL380G8 with 4 x 10GB eth and 1 x nvme intel pcie p3700 1.6 TB running osnexus quantastor. I exposed the raw device and added it to the esxi cluster with multipath. Run some tests got 160k iops with fio easily , while the disk alone gets 200k from the  quantastor console.

Setup B

HP DL380G8 with 4 x 10GB eth and 1 x nvme intel pcie p3700 1.6 TB running esos 4.

I exposed the raw device with vdisk_blockio and added it to the esxi with multipath configured. Run the same tests but this time i got 40k max. A single worker process would max it so it got me thinking there must be some problem with multipath. I went to console and tried :

cat /sys/module/nvme_core/parameters/multipath

i got an error saying that cat: can't open '/sys/module/nvme_core/parameters/multipath': No such file or directory so i figured out nvme multipath wasn't enabled.

 Is there any way to enable it? Is that the problem or i missed something ?

Thank you in advance.

[Marc Smith]

Looks like it's disabled in kernel config:
https://github.com/quantum/esos/blob/master/misc/linux-5.4.190.config#L1551

But I don't think this is the multipath setting you're looking for...
this would be for native NVMe multipath'ing (eg, on the local machine,
or if you connected to NVMe device via NVMeoF).


--Marc

>
> Thank you in advance.
>
> --
> You received this message because you are subscribed to the Google Groups "esos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/1706e8fa-5343-43ed-961b-1ae7d8b374b6n%40googlegroups.com.

[Nicholas Papadakos]

hello,

I never thought of NVMe multipathing in that way so i guess i should look elseware.

Is there a guide or even a short tutorial or something about setting up multipathing?

Do i need to set up ALUA ?

I haven't touched it in my config.

Best Regards,

[Nicholas Papadakos]

anyone ?

i could really use some insight as i can't manage to set multipath working corectly.

best regards,

[Marc Smith]

Sorry, I still don't fully understand what you're trying to setup...
so you have a single-head ESOS box, and you want to configure your
initiators for MPIO? In Linux, this is done using DM-multipath, and
with ESXi it has it's own built-in MPIO stack. You expose multiple
paths from the ESOS storage and then configure you're initiators for
multipath I/O.

--Marc

On Mon, May 30, 2022 at 3:16 PM Nicholas Papadakos

> To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/1dd233f8-61be-4a17-ae50-2a458bde6cf0n%40googlegroups.com.