Persistent memory in device dax mode

[Jerrin Shaji George]

Hi All,

I had a question regarding persistent memory programming. I have a test program

in which in which I map a persistent memory device into memory and write some

bytes into the first 8 bytes.

I am then using the linux pagemap interface to get the page frame number of the

mapped address, and see if I can actually read the data written into the pmem

device from /dev/mem.

I was suprised to see that I could read back the written data from /dev/mem. My

understanding regarding this is limited, but I was not expecting to read the

data from /dev/mem as I thought writes to the pmem device wouldn't go to /dev/mem

first. Is there something that I am missing here?

Please find the program that I am referencing here - 

https://gist.github.com/jerrinsg/31582cab3b7e47e63503736a0a6ffe53

Output from running the program:

# ./pmemTest 

pmemaddr is 0x7f5e2943d000

page present:1 page swapped:0 page exclusively mapped:0 file page or shared-anon:0 pfn:14893568

read from device: 8877665544332211

I am running the program on a server running Ubuntu 18.04. The persistent memory

hardware used is Intel 3DXpoint.

# uname -a

Linux jerrin-ubuntu-server 4.15.18+ #2 SMP Mon Jul 2 23:41:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

# ndctl create-namespace --mode devdax -e namespace0.0 -f -a 4K

# ndctl list

[

  {

    "dev":"namespace1.0",

    "mode":"raw",

    "size":536870912000,

    "uuid":"4b1eae70-2ee2-3152-b1d1-9b793a7c80e4",

    "sector_size":512,

    "blockdev":"pmem1",

    "name":"NvDimmVol2",

    "numa_node":0

  },

  {

    "dev":"namespace0.0",

    "mode":"devdax",

    "map":"dev",

    "size":528480206848,

    "uuid":"e19bbba6-c361-425d-be04-7dcd96d4247d",

    "raw_uuid":"265b90ee-5ae3-4480-b324-b58869ada005",

    "chardev":"dax0.0",

    "name":"NvDimmVol1",

    "numa_node":0

  }

]

Thanks,

Jerrin

[Andy Rudoff]

Hi Jerrin,

Just walking through your program...

When you mmap() /dev/dax0.0, or when you mmap a DAX-mounted file system, you get direct access to the persistent memory.  The virtual address you get back from mmap() gets translated by the page tables to the physical address of the persistent memory.  After storing directly to the persistent memory, your program then looks up the translation from virtual to physical in the same page tables, and uses the /dev/mem driver which takes the offset you read at as a physical address and returns the contents at that physical address.  So you are writing and reading back the exact same physical address, writing through the mappings setup by DAX, and reading through the /dev/mem driver logic that converts the offset into the mem device into the same physical address.

So I'm not sure what the confusion is -- your program proves what you stored is at the exact address where you stored it.  Your conclusion about "first going to /dev/mem" isn't correct -- the store goes to a physical address and /dev/mem allows access to that same physical address.  It didn't go anywhere else first and there's no kernel page cache or DRAM copy of your stores  (that's what DAX is all about).

Hope that helped clarify it a bit.

-andy

[Jerrin Shaji George]

Thanks Andy for the clarification. My confusion stemmed from wrongly assuming

/dev/mem represented the main memory (DRAM) and not the entire physical address

space.

[Dan Williams]

Note that the ability to do this dual access through /dev/mem and
/dev/dax is disabled by the CONFIG_IO_STRICT_DEVMEM kernel
configuration symbol. Hopefully more distributions turn that on in the
future.

> --
> You received this message because you are subscribed to the Google Groups "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
> To post to this group, send email to pm...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/9b149590-c43b-4dbf-acdf-86671de4719b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.