CXL Ping-Pong Programming

[Zach O'Donnell]

I’m experimenting with CXL and trying to figure out how to use it as shared memory between two different hosts/instances. The goal is to write a small C program where:

• Instance A writes data into CXL memory

• Instance B reads that same data from the shared CXL memory region

Ideally, I’d like to do this entirely from user space (no kernel module).

Does anyone know of sample code, libraries, or documentation on how to mmap() or otherwise access CXL memory from user space?

[Zach O'Donnell]

*Configured as a dax device

[Jeff Moyer]

Hi, Zach,

Zach O'Donnell <zodon...@gmail.com> writes:

> I’m experimenting with CXL and trying to figure out how to use it as *shared
> memory* between two different hosts/instances. The goal is to write a small
> C program where:

The following mailing list thread provides some insight into what may or
may not be possible with shared CXL memory:

https://lore.kernel.org/linux-cxl/a571be12-2fd3-e0ee...@easystack.cn/T/#m56476606140cb6966824ee27a2292747bbc5a040

Cheers,
Jeff

[Dongha Yoon]

Here is simple C code. I did not test this code but I believe you can fix some bugs.

```
#include <sys/mman.h>

#include <fcntl.h>

#define SERVER_ID 0 /* 0: writer, 1: reader */

#define DAX_PATH "/dev/dax0.0"

#define CXL_SHARED_SIZE (4096UL)

#define DAX_OFFSET (0UL)

int main() {

  int fd = open(DAX_PATH, O_RDWR);
  if (fd < 0) {

    perror("open error");

    return -1;

  }

  void *addr = mmap(NULL,  CXL_SHARED_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED, fd, DAX_OFFSET);

  if (addr == MAP_FAILED) {

    perror("mmap failed");

    return -1;

  }

  

  if (SERVER_ID  == 0) {

    /* writer, initialization */

    memset(addr, 0, CXL_SHARED_SIZE);

    cache_line_flush(addr); 

    /* write a data */

    *((int  *)addr = 0x1234;

     cache_line_flush(addr);

  }

  else {

      /* reader, wait for initialization */

      sleep(5);

      cache_line_flush(addr);      

      /* wait for data */

      while (*((int  *)addr == 0) {

        cache_line_flush(addr);

      }

      printf("Data: %d\n", *((int *)addr));

  }

   return 0;

}
```

2025년 11월 6일 목요일 오전 11시 23분 7초 UTC-8에 jmo...@redhat.com님이 작성:

[Zach O'Donnell]

Thank you for your answer. I will take a look at that, it seems really interesting!

[Zach O'Donnell]

Hi,

Thank you for your answer. First of all, I apologize if I make any obvious mistakes. I am just a noob here :)

I get "Error memory-mapping DAX device: Invalid argument" when I use CXL_SHARED_SIZE (4096UL) in mmap, but it works fine with 2 * 1024 * 1024. Do you know why that might be?

Using 2 MiB for CXL_SHARED_SIZE and flushing with clflushopt, I get the following output:

• Writes from host a -- mmaps successfully!
• reader: Data: -1 - does not read a's write
  
  

Could the size or alignment still be an issue? The device is in daxdev mode. Am I missing a configuration step or using the wrong flush instruction?

Thanks again,
ZO

[Dongha Yoon]

1. About mmap error: Can you show me output of '$cxl list -Ru' ? I assume 'interleave_granularity' affects the minimum mmap size.
2. About data correctness: could you try clflushopt before and after every accesses? And check the value before and after write in the writer's side. 

Thank you
Dongha

2025년 11월 6일 목요일 오후 4시 25분 19초 UTC-8에 zodon...@gmail.com님이 작성:

[John Groves]

> > <https://lore.kernel.org/linux-cxl/a571be12-2fd3-e0ee...@easystack.cn/T/#m56476606140cb6966824ee27a2292747bbc5a040>
> >
> > Cheers,
> > Jeff
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/pmem/89fd8b84-77ef-4ffc-9d56-462f2957a089n%40googlegroups.com.

Famfs (the fabric-attached memory file system) formats disaggregated shared
memory as a file system. Check out https://famfs.org for code, docs, youtube
links, etc. If you mmap a famfs file, it's just memory - no faulting into
the page cache.

One of the test programs in the famfs source, pcq.c, implements shared memory
producer-consumer queues as file pairs in famfs - without using atomics, which
CXL doesn't support.

John

[Zach O'Donnell]

Dongha,

Thank you for your response! 

Here is the output for cxl list -Ru

{

  "region":"region1",

  "resource":"0x8080000000",

  "size":"128.00 GiB (137.44 GB)",

  "interleave_ways":1,

  "interleave_granularity":4096,

  "decode_state":"commit"

}

When CXL_SHARED_SIZE is 4096: Error memory-mapping DAX device: Invalid argument. However, using 2 MiB (2 * 1024 * 1024) mmap works. 

Now, making some edits to the script - clflushopt-ing before and after every access-, here are the outputs:

From host A, writer: 

sudo ./w

DAX device just mapped at address 0x7f5188400000

Data before writing: -1

Data after writing completed: 4660

From host B, reader:

sudo ./r

DAX device just mapped at address 0x7f02c6400000

Data: -1

As I said, I'm pretty new to this, so I might be doing something wrong. 

Really appreciate your help!

[Zach O'Donnell]

John,

Thank you for your response! I actually came across your work before, it's super interesting!

Best,

ZO