DAX (load/store/flush)

[Anton Gavriliuk]

Hi team

I would ask you about pmem DAX (load/store/flush).  This is quite new for me because it's not I/O we known over the years, so stupid questions possible 😊

Load - means load data from pmem to cpu.  But where exactly on cpu ?, L3, L2, L1, load/store buffers, registers ??

The same for store - store data from where to where exactly ?? and then flush to pmem.

Anton

[Andy Rudoff]

Hi Anton,

Just to clarify the terminology a bit.  When a file is exposed via DAX, that means it doesn't use the page cache.  Syscalls like open/close/read/write will work as expected, but inside the kernel the operations are being turned into memory copies directly with the media.  If you then use mmap() to map the DAX file into your address space, you then have direct load/store access.  Typically, the loads and stores are just like any other memory locations, meaning they are cached by the CPU caches.  For example, assume you have a pointer to the file contents you got from mmap(), like this (using C and Linux for this example):

    char *pmembase;

    pmembase = (char *)mmap(...);  /* map the pmem */

Now you do a load to fetch data directly from the media, like this:

    char c;

    c = *pmembase;    /* fetch a character directly from the media */

Just like any other load from memory, this will use the CPU caches.  Now store the character 'X' to pmem:

    *pmembase = 'X';   /* store a character to pmem */

This will store the value to the media, but it will land in the CPU cache and if you lose power before the value was evicted from the cache, it won't be persistent.

To ensure the value is persistent, you can either use syscalls like msync()/fsync(), or if you used the MAP_SYNC flag with mmap(), then it is safe to use instructions like CLWB to make the change persistent.  The libpmem routine pmem_persist() will do this for you (see http://pmem.io for more info on the PMDK libraries).

-andy

[Anton Gavriliuk]

Now it's better for my understanding, thank you Andy.

>  Syscalls like open/close/read/write will work as expected, but inside the kernel the operations are being turned into memory copies directly with the media. 

That's like if I create and mount xfs/ext4 with "-o dax" option and then will use standart I/O system calls ?

>  Now store the character 'X' to pmem:

> *pmembase = 'X';   /* store a character to pmem */

> This will store the value to the media, but it will land in the CPU cache and if you lose power before the value was evicted from the cache, it won't be persistent.

With non-temporal stores we still use CPU L1-3 caches ?, I was thought it should be directly registers (or store buffers) -> pmem.

Anton

вт, 30 апр. 2019 г. в 19:10, Andy Rudoff <an...@rudoff.com>:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/f37b2a16-521a-46bf-ad47-fcfb7f8b9de8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andy Rudoff]

>  Syscalls like open/close/read/write will work as expected, but inside the kernel the operations are being turned into memory copies directly with the media. 

That's like if I create and mount xfs/ext4 with "-o dax" option and then will use standart I/O system calls ?

Right.  Currently dax is a mount option.  (In the future, it may turn into a per-file attribute, but for now it is a mount option.)

 

>  Now store the character 'X' to pmem:

> *pmembase = 'X';   /* store a character to pmem */

> This will store the value to the media, but it will land in the CPU cache and if you lose power before the value was evicted from the cache, it won't be persistent.

With non-temporal stores we still use CPU L1-3 caches ?, I was thought it should be directly registers (or store buffers) -> pmem.

Non-temporal stores will bypass the CPU caches.  In my example, I was showing you the normal stores you get with a typical assignment statement.

-andy

[steve]

Hi all,

I have gotten my new Optane PM-equipped server up and running with Windows 10 LTSC, and I'm getting over 500K 400-byte records per second throughput
with about 1 usec latency when the program parameters are tuned properly.

I was wondering whether there is any way to get at the Optane PM devices from a Linux virtual machine under Windows. I realize this may not be
possible, in which case I'll have to dual-boot Linux instead, but it would be a lot more convenient to be able to do it with a virtual machine.

Any assistance would be appreciated.
------------
Steve Heller

[Steve Scargall]

Hi Steve,

If you wanted to use Hyper-V, there is some documentation for using "Cmdlets for configuring persistent memory devices for Hyper-V VMs".  If you install Linux within the guest, it should see the pmem (type 12?).  (Note: I've not tested this for myself yet, though it is on my TODO list.  If you do get it working, feel free to document it and we'd be happy to host it on docs.pmem.io).  With the recent announcement that Windows will deliver a Linux Kernel, this will open new pmem configuration opportunities.

As far as I can see, neither VMWare Player nor Oracle Virtualbox support DAX.  You could still present pmem as a storage device to the guest.

Regards,

    SteveS

[steve]

On Mon, 13 May 2019 08:42:50 -0700 (PDT), Steve Scargall <steve.s...@gmail.com> wrote:

>Hi Steve,
>
>If you wanted to use Hyper-V, there is some documentation for using "Cmdlets
>for configuring persistent memory devices for Hyper-V VMs

><https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/persistent-memory-cmdlets>".

>If you install Linux within the guest, it should see the pmem (type 12?).
>(Note: I've not tested this for myself yet, though it is on my TODO list.
>If you do get it working, feel free to document it and we'd be happy to
>host it on docs.pmem.io).

I haven't been able to get this to work. Here is what I have done:

1. Create a Generation 2 Hyper-V VM.

2. Install Ubuntu 18.04.

3. Follow the steps in that article, after correcting a typo in the command:
"Add-VMHardDiskDrive ProductionVM1 PMEM -ControllerLocation 1 -Path D:\VPMEMDevice1.vhdpmem"

which I think should be

"Add-VMHardDiskDrive ProductionVM1 -ControllerType PMEM -ControllerLocation 1 -Path D:\VPMEMDevice1.vhdpmem"

although it doesn't seem to need the "-ControllerLocation" argument because it seems to work when I do this:

"Add-VMHardDiskDrive -VMName UbuntuPMTest -ControllerType PMEM -Path F:\VMPMEMDevice1.vhdpmem"

At least, the command:
"Get-VMHardDiskDrive -VMName UbuntuPMTest"

shows this result:

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
UbuntuPMTest SCSI 0 0 C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks\UbuntuPMTest.vhdx
UbuntuPMTest PMEM 0 0 F:\VMPMEMDevice1.vhdpmem


4. Start the VM, I get this:

[Window Title]
Hyper-V Manager

[Main Instruction]
An error occurred while attempting to start the selected virtual machine(s).

[Content]
'UbuntuPMTest' failed to start.

[Expanded Information]
(Virtual machine ID 0D2E2FE2-635B-43B0-93E1-9A8A5CDC261E)

'UbuntuPMTest' failed to start worker process: One or more arguments are invalid (0x80070057). (Virtual machine ID
0D2E2FE2-635B-43B0-93E1-9A8A5CDC261E)

[^] Hide details [Close]

5. Note that if I remove the hard disk via
"Remove-VMHardDiskDrive -VMName UbuntuPMTest -ControllerType PMEM -ControllerNumber 0 -ControllerLocation 0"

then the VM does start, so that demonstrates that it is actually the PM drive that is causing the problem.
Obviously that doesn't help me get the PM drive working.

Any suggestions as to how to proceed from here?
------------
Steve Heller

[Pratim Bose]

Hi Steve,

I had created a SLES VM a while back on Hyper-V and had used:

(i) Add-VMPmemController <VM-Name>

(ii) Add-VMHardDiskDrive <VM-Name> PMEM -ControllerLocation 1 -Path <vhdpmem path>

I have not used the ControllerType switch you are using, but i can check that out too. Additionally -  when it does start - without the PMEM Controller Type, are you able to see any PMEM device unsder /dev/? 

I am using a Windows Server and not a Windows 10. Can you please share the steps of VM creation as well?

Regards,

Pratim

On Friday, May 31, 2019 at 8:09:41 PM UTC+5:30, steve wrote:

[Pratim Bose]

Some more attachments.