pmem over rdma

[Anton Gavriliuk]

Hi all

I have 2 servers running RHEL 8.6

First server with Optane pmem, second server without.

Using NVMeoRDMAoROCEv2 I'm able to map pmem (in raw/sector/fsdax) to the second server.

However, when pmem is in devdax mode, I can't map pmem to the second server.
Pmem in raw/sector/fsdax modes is a block device and with devdax mode pmem is a character device, could it be the reason ??
On first server I got next errors

[root@memverge anton]# modprobe nvmet-rdma
[root@memverge anton]# nvmetcli restore rdma.json
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/nvmet/nvme.py", line 187, in set_enable
    file_fd.write(str(value))
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/sbin/nvmetcli", line 740, in <module>
    main()
  File "/usr/sbin/nvmetcli", line 722, in main
    funcs[sys.argv[1]](savefile)
  File "/usr/sbin/nvmetcli", line 678, in restore
    errors = nvme.Root().restore_from_file(from_file)
  File "/usr/lib/python3.6/site-packages/nvmet/nvme.py", line 382, in restore_from_file
    abort_on_error=abort_on_error)
  File "/usr/lib/python3.6/site-packages/nvmet/nvme.py", line 356, in restore
    Subsystem.setup(t, err_func)
  File "/usr/lib/python3.6/site-packages/nvmet/nvme.py", line 497, in setup
    Namespace.setup(s, ns, err_func)
  File "/usr/lib/python3.6/site-packages/nvmet/nvme.py", line 609, in setup
    ns._setup_attrs(n, err_func)
  File "/usr/lib/python3.6/site-packages/nvmet/nvme.py", line 230, in _setup_attrs
    self.set_enable(enable)
  File "/usr/lib/python3.6/site-packages/nvmet/nvme.py", line 190, in set_enable
    (self.path, e, value))
nvmet.nvme.CFSError: Cannot enable /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1: [Errno 22] Invalid argument (1)

[root@memverge anton]# dmesg
[10341.336442] memmap_init_zone_device initialised 789577728 pages in 9824ms
[10618.040835] nvmet: adding nsid 1 to subsystem testnqn
[10618.041010] nvmet: failed to open file /dev/dax0.0: (-22)
[10618.041577] nvmet: failed to open file /dev/dax0.0: (-22)
[root@memverge anton]#

Anton