PMEM file on Windows RAM Disk

[SeanW]

 

Goal is to find a reasonably fast alternative in Windows (10) similar to emulating PMEM in Linux, for development purposes.

 

At the bottom of the original Linux emulation blog post there is a tip – https://pmem.io/2016/02/22/pm-emulation.html:

It is additionally worth mentioning you can emulate persistent memory with ramdisk (i.e. /dev/shm), or force pmem-like behavior by setting environment variable PMEM_IS_PMEM_FORCE=1, that would eliminate performance hit caused by msync(2).

 

There might be an option to create an iSCSI based RAM Disk on Windows Server but doesn’t seem available in Windows 10.

 

The first one I tried was ImDisk https://sourceforge.net/projects/imdisk-toolkit/, however when trying to open a pmem file here, PMDK was throwing an exception “GetVolumePathNameW: Success” and wouldn’t open the file. This had some useful features though, like being able to save and reload an image at shutdown/restart.

 

The next one I tried was Starwind RAM disk https://www.starwindsoftware.com/high-performance-ram-disk-emulator. Using this I was able to create a RAM Disk of 18GB, reformat as NTFS (default was FAT32), then create a pmem file of 16GB.

 

Time taken on SSD: (Samsung 960 PRO 512GB on M.2 socket): 31 sec (baseline)

[2019-08-04 08:20:21.189] [main] [info] Creating new pool c:\temp\test.pmem with layout 'myapp_v0.0.1' and size 17179869184
[2019-08-04 08:20:52.790] [main] [info] Pool successfully created.

 

Time taken on RAM Disk: 18 sec

[2019-08-04 08:24:05.177] [main] [info] Creating new pool e:\test.pmem with layout 'myapp_v0.0.1' and size 17179869184
[2019-08-04 08:24:23.943] [main] [info] Pool successfully created.

 

However then initialising the root object still took ~7 seconds on Ram Disk:

[2019-08-04 08:24:23.943] [main] [info] Pool successfully created.
[2019-08-04 08:24:31.127] [main] [info] AppManager: App initializing after first persistence  

After SET PMEM_IS_PMEM_FORCE=1

 

Time taken on SSD: Pool creation 24s, object initiation 2ms

[2019-08-04 08:56:26.042] [main] [info] Creating new pool c:\temp\test.pmem with layout 'myapp_v0.0.1' and size 17179869184
[2019-08-04 08:56:50.784] [main] [info] Pool successfully created.
[2019-08-04 08:56:50.786] [main] [info] AppManager: App initializing after first persistence

 

Time taken on RAM Disk: Pool creation 15s, object initiation 2ms

[2019-08-04 08:49:47.364] [main] [info] Creating new pool e:\test.pmem with layout 'myapp_v0.0.1' and size 17179869184
[2019-08-04 08:50:02.691] [main] [info] Pool successfully created
[2019-08-04 08:50:02.693] [main] [info] AppManager: App initializing after first persistence

The initialization time has significantly reduced for both tests.

 

For comparison, on Linux on Emulated PMEM: Pool creation 35ms, object initiation <1ms

[2019-08-04 09:07:39.537] [main] [info] Creating new pool /mnt/pmem0/test.pmem with layout 'myapp_v0.0.1' and size 8589934592
[2019-08-04 09:07:39.572] [main] [info] Pool successfully created.
[2019-08-04 09:07:39.572] [main] [info] AppManager: App initializing after first persistence

Linux (Fedora 29) using  memmap emulation. Unfortunately my test Linux server doesn’t have 16GB RAM free, so used 8GB – but still it is significantly faster

 

Using PMDK 1.6 and libpmemobj-c++ v1.7.

 

Needs more rigorous testing and testing for larger transactions, larger reads, concurrency etc. But maybe some improvement. 

PMDK team - is there anything on the horizon that might allow byte-addressable emulation on Windows similar to Linux memmap? These options are still all block-based I believe.

  

Sean Whiteley - AxoMem.io

axomem.io | @axomemio | @seanwitx

[steve]

Could I ask if you have considered buying a server with actual Optane persistent memory in it?

On Sat, 3 Aug 2019 18:47:23 -0700 (PDT), SeanW <sdw.g...@axomem.io> wrote:

>
>
>
>
>Goal is to find a reasonably fast alternative in Windows (10) similar to
>emulating PMEM in Linux, for development purposes.
>
>
>
>At the bottom of the original Linux emulation blog post there is a tip –
>https://pmem.io/2016/02/22/pm-emulation.html:
>

>*It is additionally worth mentioning you can emulate persistent memory with

>ramdisk (i.e. /dev/shm), or force pmem-like behavior by setting environment
>variable PMEM_IS_PMEM_FORCE=1, that would eliminate performance hit caused

>by msync(2).*

>
>
>
>There might be an option to create an iSCSI based RAM Disk on Windows
>Server but doesn’t seem available in Windows 10.
>
>
>
>The first one I tried was ImDisk
>https://sourceforge.net/projects/imdisk-toolkit/, however when trying to
>open a pmem file here, PMDK was throwing an exception “GetVolumePathNameW:
>Success” and wouldn’t open the file. This had some useful features though,
>like being able to save and reload an image at shutdown/restart.
>
>
>
>The next one I tried was Starwind RAM disk
>https://www.starwindsoftware.com/high-performance-ram-disk-emulator. Using
>this I was able to create a RAM Disk of 18GB, reformat as NTFS (default was
>FAT32), then create a pmem file of 16GB.
>
>
>
>Time taken on SSD: (Samsung 960 PRO 512GB on M.2 socket): 31 sec (baseline)
>[2019-08-04 08:20:21.189] [main] [info] Creating new pool c:\temp\test.pmem
>with layout 'myapp_v0.0.1' and size 17179869184
>[2019-08-04 08:20:52.790] [main] [info] Pool successfully created.
>
>
>
>Time taken on RAM Disk: 18 sec
>[2019-08-04 08:24:05.177] [main] [info] Creating new pool e:\test.pmem with
>layout 'myapp_v0.0.1' and size 17179869184
>[2019-08-04 08:24:23.943] [main] [info] Pool successfully created.
>
>
>
>However then initialising the root object still took ~7 seconds on Ram Disk:
>[2019-08-04 08:24:23.943] [main] [info] Pool successfully created.
>[2019-08-04 08:24:31.127] [main] [info] AppManager: App initializing after
>first persistence
>
>

>*After SET PMEM_IS_PMEM_FORCE=1*

>
>
>
>Time taken on SSD: Pool creation 24s, object initiation 2ms
>[2019-08-04 08:56:26.042] [main] [info] Creating new pool c:\temp\test.pmem
>with layout 'myapp_v0.0.1' and size 17179869184
>[2019-08-04 08:56:50.784] [main] [info] Pool successfully created.
>[2019-08-04 08:56:50.786] [main] [info] AppManager: App initializing after
>first persistence
>
>
>
>Time taken on RAM Disk: Pool creation 15s, object initiation 2ms
>[2019-08-04 08:49:47.364] [main] [info] Creating new pool e:\test.pmem with
>layout 'myapp_v0.0.1' and size 17179869184
>[2019-08-04 08:50:02.691] [main] [info] Pool successfully created
>[2019-08-04 08:50:02.693] [main] [info] AppManager: App initializing after
>first persistence
>
>
>The initialization time has significantly reduced for both tests.
>
>
>

>For comparison, on Linux on Emulated PMEM: Pool creation 35*ms*, object

>initiation <1ms
>[2019-08-04 09:07:39.537] [main] [info] Creating new pool /mnt/pmem0/test.pmem
>with layout 'myapp_v0.0.1' and size 8589934592
>[2019-08-04 09:07:39.572] [main] [info] Pool successfully created.
>[2019-08-04 09:07:39.572] [main] [info] AppManager: App initializing after
>first persistence
>

>*Linux (Fedora 29) using memmap emulation
><https://docs.pmem.io/getting-started-guide/creating-development-environments/linux-environments/linux-memmap>. **Unfortunately

>my test Linux server doesn’t have 16GB RAM free, so used 8GB – but still it

>is significantly faster*

>
>
>
>Using PMDK 1.6 and libpmemobj-c++ v1.7.
>
>
>
>Needs more rigorous testing and testing for larger transactions, larger
>reads, concurrency etc. But maybe some improvement.
>
>
>PMDK team - is there anything on the horizon that might allow
>byte-addressable emulation on Windows similar to Linux memmap? These
>options are still all block-based I believe.
>
>
>

>*Sean Whiteley - AxoMem.io*
>
>axomem.io | @axomemio <https://twitter.com/axomemio> | @seanwitx
><https://twitter.com/seanwitx>
------------
Steve Heller

[SeanW]

Considered yes, purchased not yet. Has anyone on the list worked out the cheapest server one could buy with Persistent Memory recently? 

I had access to an Intel server with Intel® Optane™ DC Persistent Memory as part of the Private Preview https://builders.intel.com/private-preview program before the launch in April. Yes it is fast, but it is very different when you are writing an app using PMDK and C++ with byte-level access vs just comparing media speeds doing sequential or even random block read/writes. 

I found the largest speed limitation was inefficient code (that’s being polite) and missing/mismanaged parallelism. After that, there wasn’t a significant difference in using emulation in RAM and real PMEM – not enough to worry about anyway when you are operating at those speeds. 

If your code runs bad on emulated PMEM on 4 cores its going to run bad on real PMEM on 196 cores. 

The main difference is capacity, cores and sockets. I designed my apps so that I could test on a fraction of the total data capacity, either by reducing the number of entities loaded or reducing the history stored. I lose the data when I turn off the server. But since I am using a subset, it doesn't take long to reload. Or I don't turn the server off (its Linux after-all)...

At present I generally work on emulated PMEM. My "server" is a Linux PC with an older i7 CPU and 32GB DRAM running Fedora 29. I run CLion in Windows using their remote SSH support. I've also run the Linux server on Hyper-V on my laptop while on a long haul flight. I am now doing some work targeting cross-platform support, hence why I am looking at Windows emulation options.

There’s still much to learn in this space that access to real PMEM at present is not really a limitation IMHO. Topics to focus on are efficient C++ code, C++17 concurrency support, NUMA awareness, Intel PSTL/TBB, profiling and optimizing, data oriented design, on top of PMDK and libpmem-c++. All these can be learned/improved without real PMEM, but one needs to know these if you want to program for an 8 socket server with persistent memory. 

Its perhaps similar to the way Quantum computing is developing. Few have direct access to real quantum hardware, but simulators can already run up to 30 qubits on a regular laptop. This is more than enough to learn the fundamentals. And now we have quantum inspired algorithms too. 

[SeanW]

https://www.omgubuntu.co.uk/2019/05/ubuntu-support-windows-subsystem-linux-2

Has anyone tried getting Linux PMEM emulation working in WSL v2 yet? Apparently it includes full kernel...

With Edge now moving to Chromium, maybe this is start of the same thing for Windows? Ok perhaps not. 

[Łukasz Plewa]

Windows on non-pmem filesystems are very slow. On pmem, performance should be similar to the Linux except "deep flush"(on windows is much slower)

deep flush is used during pool open/create and close and it's performance unfortunetly is scales with pool size on windows.

All of this performance problems are caused by need to call FlushViewOfFile() and FlushFileBuffers() to flush data to the disk (non-pmem) and perform deep flush.

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/77a93a5e-7fcf-421b-8ec9-73da8abae526%40googlegroups.com.

[Łukasz Plewa]

To disable deep_flush() at pool create you can use ctl sds.at_create = 0, (but you shouldn't use it in production enviroment).

http://pmem.io/pmdk/manpages/windows/master/libpmemobj/pmemobj_ctl_get.3

Also with this flag ImDisk ramdisk should work.