CXL1.1 and memory tiering

89 views
Skip to first unread message

Anton Gavriliuk

unread,
Sep 22, 2022, 4:32:38 PMSep 22
to pmem

Hi all 


With upcoming CXL 1.1 I’m very interested in CXL.mem and memory tiering between local and CXL DRAM.

 

While there is no CXL hardware yet available, is it possible to simulate CXL DRAM using Optane DCPMM or any other way ?


Anton

Andy Rudoff

unread,
Sep 23, 2022, 11:28:22 AMSep 23
to pmem
Hi Anton,

Sure, using Optane is an excellent way to prepare for dealing with multiple memory tiers.  Most of the memory tiering work done in recent years was developed with Optane as one of the tiers.  By configuring Optane in App Direct mode, using device DAX, and configuring it as system-ram, the Optane tier will appear as a separate NUMA node just like CXL-attached memory will.

Of course, Optane has different performance characteristics than DRAM.  Another way to simulate CXL-attached DRAM is simply to use remote DRAM (i.e. on a remote NUMA node from where your code is running).

Either way, the APIs, libraries built on them, and even the application-transparent tiering solutions will be identical to those you'd use with CXL memory.

steve.s...@gmail.com

unread,
Sep 23, 2022, 11:50:18 AMSep 23
to pmem
Hi Anton,

QEMU has features that can emulate most CXL 2.0 features including devices and switches. I wrote a blog about this - How To Emulate CXL Devices using KVM and QEMU. QEMU has some great documentation on the CXL features - https://www.qemu.org/docs/master/system/devices/cxl.html. I need to update the article to use the latest QEMU main branch code, Linux Kernel 6.0, and ndctl v74 which gives us more features and functionality than were available when I wrote the blog. This approach is good if you want to see what a CXL environment will look like with real hardware, but would not be good for benchmarking.

Linux has the tiered memory feature in Kernel v5.15 or later, though this is only the demotion feature - DRAM -> CXL/PMem. The promotion feature - CXL/PMem -> DRAM - is still pending upstream. I wrote a blog to explain how the feature works - Using Linux Kernel Memory Tiering. Alternatively, you can contact MemVerge (or me) for info on our Tiering, Snapshot, Replication, and Cloning features in Memory Machine. 

Kernel 6.1 is set to get MGLRU features that should benefit tiering performance - https://www.phoronix.com/news/Linux-6.1-Improve-Memory-Tiers.

HTH
    Steve

Anton Gavriliuk

unread,
Sep 23, 2022, 2:59:29 PMSep 23
to Andy Rudoff, pmem
Thank you Andy

> Of course, Optane has different performance characteristics than DRAM. 

I still like Optane DCPMM.  I believe perf characteristics will be improved in Crow Pass, but even now Optane has higher latency only for reads, for writes it is very close to DRAM (~90ns)

> Another way to simulate CXL-attached DRAM is simply to use remote DRAM (i.e. on a remote NUMA node from where your code is running).

I missed this nice and simple idea.Thank you!

Anton

пт, 23 сент. 2022 г. в 18:28, Andy Rudoff <an...@rudoff.com>:
--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/a6a318e7-6f99-418b-9ed2-bb896df9100bn%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages