CXL1.1 and memory tiering

[Anton Gavriliuk]

Hi all 

With upcoming CXL 1.1 I’m very interested in CXL.mem and memory tiering between local and CXL DRAM.

 

While there is no CXL hardware yet available, is it possible to simulate CXL DRAM using Optane DCPMM or any other way ?

Anton

[Andy Rudoff]

Hi Anton,

Sure, using Optane is an excellent way to prepare for dealing with multiple memory tiers.  Most of the memory tiering work done in recent years was developed with Optane as one of the tiers.  By configuring Optane in App Direct mode, using device DAX, and configuring it as system-ram, the Optane tier will appear as a separate NUMA node just like CXL-attached memory will.

Of course, Optane has different performance characteristics than DRAM.  Another way to simulate CXL-attached DRAM is simply to use remote DRAM (i.e. on a remote NUMA node from where your code is running).

Either way, the APIs, libraries built on them, and even the application-transparent tiering solutions will be identical to those you'd use with CXL memory.

[steve.s...@gmail.com]

Hi Anton,

QEMU has features that can emulate most CXL 2.0 features including devices and switches. I wrote a blog about this - How To Emulate CXL Devices using KVM and QEMU. QEMU has some great documentation on the CXL features - https://www.qemu.org/docs/master/system/devices/cxl.html. I need to update the article to use the latest QEMU main branch code, Linux Kernel 6.0, and ndctl v74 which gives us more features and functionality than were available when I wrote the blog. This approach is good if you want to see what a CXL environment will look like with real hardware, but would not be good for benchmarking.

Linux has the tiered memory feature in Kernel v5.15 or later, though this is only the demotion feature - DRAM -> CXL/PMem. The promotion feature - CXL/PMem -> DRAM - is still pending upstream. I wrote a blog to explain how the feature works - Using Linux Kernel Memory Tiering. Alternatively, you can contact MemVerge (or me) for info on our Tiering, Snapshot, Replication, and Cloning features in Memory Machine. 

Kernel 6.1 is set to get MGLRU features that should benefit tiering performance - https://www.phoronix.com/news/Linux-6.1-Improve-Memory-Tiers.

HTH

    Steve

[Anton Gavriliuk]

Thank you Andy

> Of course, Optane has different performance characteristics than DRAM. 

I still like Optane DCPMM.  I believe perf characteristics will be improved in Crow Pass, but even now Optane has higher latency only for reads, for writes it is very close to DRAM (~90ns)

> Another way to simulate CXL-attached DRAM is simply to use remote DRAM (i.e. on a remote NUMA node from where your code is running).

I missed this nice and simple idea.Thank you!

Anton

пт, 23 сент. 2022 г. в 18:28, Andy Rudoff <an...@rudoff.com>:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/a6a318e7-6f99-418b-9ed2-bb896df9100bn%40googlegroups.com.