Emulate CXL on qemu

[Raja]

Hi,

I am a  CSE undergrad at IIT Bombay. I am trying to emulate cxl on qemu and kvm for a research project. I am following the https://stevescargall.com/2022/01/20/how-to-emulate-cxl-devices-using-kvm-and-qemu/ blog.

I am getting Warning: no matching devices found at last step (cxl list -M). I have followed each steps mentioned in the blog but still getting the error.
Does anyone know how to resolve this?
Host OS - Ubuntu 22.04

Guest OS - Fedora 35

Thanks,
Raja Gond

[steve.s...@gmail.com]

Hi Raja,

My blog item is a little outdated. The intent was to show that - at that time - we could see `/dev/cxl/memN` and `/sys/bus/cxl/devices/memN` devices. You can't do anything with those devices since load/stores aren't supported. 

When `cxl list ..` returns "Warning: no matching devices found", it means there are no `/dev/cxl/memN` devices.

There is active work in the Kernel and QEMU communities to support volatile CXL Type 3 devices. The patches in those links get us further, but it's very much a work in progress, so keep watching.

/Steve

[Raja]

Hi Steve,

Thanks for the information.
I have utilized the latest QEMU documentation, kernel version, and your informative blog as references to emulate CXL. Now cxl list is showing the results. The updated instructions can be found at the following link: (Emulate cxl on qemu and kvm). Yeah I want to use these devices and will continue to keep an eye on the mentioned links.

Thanks & Regards,
Raja

[Anton Gavriliuk]

Hi Raja and Steve

I'm going to follow your link.

Requirements

• Host OS - Ubuntu 20.04

• Guest OS - Ubuntu 20.04

Should I use Ubuntu 20.04, or will the latest 22.04 be good too ?

Anton

ср, 1 февр. 2023 г. в 08:10, 'Raja' via pmem <pm...@googlegroups.com>:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/41f799c9-0a34-4e7c-849d-f5216223a009n%40googlegroups.com.

[Raja]

Latest 22.04 will work fine. I have verified it on my machine.

[Mohan Parthasarathy]

Hi Raja/Steve,

I am also getting the same error with cxl list returning empty list with the guest at Fedora 38. Do I need to update qemu ? Can you help with what to update to make things work ? I see the devices under /dev/cxl but they show up as disabled.

Regards,

Mohan

[root@cxl-test ~]# cxl list -M

  Warning: no matching devices found

[
]
[root@cxl-test ~]# ls -l /dev/cxl
total 0
crw-------. 1 root root 237, 0 Jun  8 06:23 mem0
crw-------. 1 root root 237, 1 Jun  8 06:23 mem1
[root@cxl-test ~]# cxl list -vvv
[
  {
    "memdev":"mem0",
    "pmem_size":268435456,
    "serial":0,
    "host":"0000:35:00.0",
    "state":"disabled",
    "partition_info":{
      "total_size":268435456,
      "volatile_only_size":0,
      "persistent_only_size":268435456,
      "partition_alignment_size":0
    }
  },
  {
    "memdev":"mem1",
    "pmem_size":268435456,
    "serial":0,
    "host":"0000:36:00.0",
    "state":"disabled",
    "partition_info":{
      "total_size":268435456,
      "volatile_only_size":0,
      "persistent_only_size":268435456,
      "partition_alignment_size":0
    }
  }
]

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/03a9482c-dd55-40bd-b3f0-8ce8f87fe60en%40googlegroups.com.

[steve.s...@gmail.com]

@pmonty,

A memdev typically auto enables during the initial device discovery at boot time.

Q) Do you see any *cxl*, *pci*, or other useful messages in `dmesg` inside the guest?

Q) What happens if you try to enable the memory devices inside the guest?

$ sudo cxl enable-memdev mem0 -v

[Mohan Parthasarathy]

Thanks, Steve and sorry for the late reply. If I run the enable command I get an error. See below :

[root@cxl-test ~]# sudo cxl enable-memdev mem0 -v
libcxl: __sysfs_device_parse: base: '/sys/bus/cxl/devices' dev: 'mem'
libcxl: add_cxl_memdev: mem1: base: '/sys/bus/cxl/devices/mem1'
libcxl: __sysfs_device_parse: base: '/sys/bus/cxl/devices/mem1' dev: 'pmem'
libcxl: __sysfs_device_parse: mem1: processed
libcxl: add_cxl_memdev: mem0: base: '/sys/bus/cxl/devices/mem0'
libcxl: __sysfs_device_parse: base: '/sys/bus/cxl/devices/mem0' dev: 'pmem'
libcxl: __sysfs_device_parse: mem0: processed
libcxl: __util_bind: mem0: bind failed
libcxl: cxl_memdev_enable: mem0: failed to enable
cxl memdev: cmd_enable_memdev: enabled 0 mem

The messages with CXL in the dmesg is as follows :

[root@cxl-test ~]# dmesg|grep -i cxl
[    0.582584] acpi ACPI0016:00: _OSC: OS supports [CXL11PortRegAccess CXL20PortDevRegAccess CXLProtocolErrorReporting CXLNativeHot]
[    0.586583] acpi ACPI0016:00: _OSC: OS now controls [CXLMemErrorReporting]
[    0.736837] ACPI: PCI Root Bridge [CXL0] (domain 0000 [bus 34-36])
[    0.740587] acpi ACPI0016:01: _OSC: OS supports [CXL11PortRegAccess CXL20PortDevRegAccess CXLProtocolErrorReporting CXLNativeHot]
[    0.744586] acpi ACPI0016:01: _OSC: OS now controls [CXLMemErrorReporting]
[    8.614003] cxl root0: unable to add dport0-pci0000:34 non-unique port id (pci0000:00)
[    8.639943] cxl_acpi: probe of ACPI0017:00 failed with error -16

Any suggestions ? Raja seemed to have similar issues but he appears to have resolved them so was hoping he can give some advice.

Regards,
Mohan

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/e8950855-1141-43d4-ac3c-6075e96b6965n%40googlegroups.com.

[Raja Gond]

Hi Mohan,

I had tried on fedora but it was not working. If you want to emulate it on ubuntu please refer to this link. I had tested the steps and it was working.

Regards,
Raja

[steve.s...@gmail.com]

Thanks for posting a good option/solution, Raja. I'll say that these instructions

```

mkdir ~/cxl ; cd ~/cxl
git clone https://gitlab.com/bwidawsk/qemu
cd qemu
git branch
* cxl-2.0v4

```

... can use the 'master' branch of the main QEMU project now as CXL support was added some time ago. See https://gitlab.com/qemu-project/qemu/-/tree/master. There's a lot of work currently going into CXL support in QEMU, so it's best to track the main project. If you want bleeding edge CXL features and bug fixes `git checkout` the 'staging' branch.

/Steve

[Mohan Parthasarathy]

Hi Steve,

If I use qemu from the master, I get errors when trying to instantiate cxl devices :

qemu-system-x86_64: -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-window-base=1,window-base[0]=0x4c00000000,memdev[0]=cxl-mem1: Property 'pxb-cxl.memdev[0]' not found

Am I missing something? Could you please give the exact instructions to build the latest qemu with CXL enablement ?

Regards,

Mohan

--

You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/555e4719-d336-4a94-8f4e-3cd50a5ca287n%40googlegroups.com.

[Anthony Dominic]

Hello All,

I've just tried the setup described in these  setup steps.

I used both the mainline version of QEMU and the branch described in the steps along with the mainline Linux kernel Version 6.5.7.

It did not work.  The CXL drivers were not loaded not loaded. I found no entries in /dev/cxl.

If I cut out everything and left with just one cxl device (a pxb-cxl entry, a crl-rp entry and a cxl-type3 entry), then I was able to see a mem0 device node in /dev/cxl.

sudo qemu-system-x86_64 -drive file=/var/lib/libvirt/images/CXL-ubuntu.qcow2,format=qcow2,index=0,media=disk,id=hd \
-m 4G,slots=8,maxmem=8G -smp 4 -machine type=q35,accel=kvm,nvdimm=on,cxl=on \
-enable-kvm -nographic -net nic,model=e1000 -net user,hostfwd=tcp::2222-:22 \
-object memory-backend-ram,id=vmem0,share=on,size=512M \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
-device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G

And yes, the syntax for -device pxb-cxl entry was changed.  the following described in the note no longer works:

-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=52,uid=0,len-window-base=1,window-base[0]=0x4c00000000,memdev[0]=cxl-mem1 \

Did you run into the same issues?

Thanks.

Regards,

Anthony Dominic.

[Gregory Price]

that is a very old guide, check out here:  Emulating CXL Shared Memory Devices in QEMU - MemVerge  

[Mohan Parthasarathy]

I agree !! Gregory’s guide is very comprehensive!! It helped me a lot.

Regards 

Mohan 

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/faf2a47f-3ccb-4322-b979-88f93cec065an%40googlegroups.com.

[Anthony Dominic]

Hello Gregory, Mohan,

The guide is indeed very comprehensive.  I did try the setup with both the latest QEMU and Kernel 6.6.0.  They all work.

I have 2 VMs with QEMU emulated CXL memory devices.  Have you ever tried accessing these devices from host?  Has there been any effort in doing this, enabling access to emulated CXL device from host?

Thanks for any links, insights.

Regards,

Anthony Dominic.

[chloi alverti]

Hello to all,

I am trying to reproduce the experiment of the MemVerge guide but with self-built Ubuntu lunar VM. 
I am also using kernel 6.6.4.

I have the following questions:

i) is the mainline qemu now sufficient or the dev brach of the guide is necessary?

ii) While using the latest cxl branch of the qemu repo referred in the guide, and ndctl version 78, and the qemu command from the guide -- the VM crashes during the "daxctl online-memory" command with the following error:

Did anyone have a similar issue?

Best Regards,

Alverti Chloe (UIUC post doc)