How to check if my server supports eADR?

[Jungsik Choi]

Hello everyone,

My server has 3rd gen Intel Xeon scalable processors (Platinum 8353H) and 2nd Optane memory. I expected my server to support eADR, but it does not seem to support that.

When using PMDK, the following debug messages are output:

<libpmem>: <1> [out.c:205 out_init] pid 6774: program: /home/chjs/Workplace/libfs/benchmarks/fio/fio
<libpmem>: <1> [out.c:208 out_init] libpmem version 1.1
<libpmem>: <1> [out.c:212 out_init] src version: 1.9
<libpmem>: <1> [out.c:220 out_init] compiled with support for Valgrind pmemcheck
<libpmem>: <1> [out.c:225 out_init] compiled with support for Valgrind helgrind
<libpmem>: <1> [out.c:230 out_init] compiled with support for Valgrind memcheck
<libpmem>: <1> [out.c:235 out_init] compiled with support for Valgrind drd
<libpmem>: <1> [out.c:240 out_init] compiled with support for shutdown state
<libpmem>: <1> [out.c:245 out_init] compiled with libndctl 63+
<libpmem>: <3> [mmap.c:39 util_mmap_init]
<libpmem>: <3> [libpmem.c:27 libpmem_init]
<libpmem>: <3> [pmem.c:827 pmem_init]
<libpmem>: <3> [init.c:490 pmem2_arch_init]
<libpmem>: <3> [init.c:413 pmem_cpuinfo_to_funcs]
<libpmem>: <3> [init.c:416 pmem_cpuinfo_to_funcs] clflush supported
<libpmem>: <3> [init.c:424 pmem_cpuinfo_to_funcs] clflushopt supported
<libpmem>: <3> [init.c:437 pmem_cpuinfo_to_funcs] clwb supported
<libpmem>: <3> [init.c:468 pmem_cpuinfo_to_funcs] WC workaround = 1
<libpmem>: <3> [init.c:291 use_avx_memcpy_memset] avx supported
<libpmem>: <3> [init.c:299 use_avx_memcpy_memset] PMEM_AVX enabled
<libpmem>: <3> [init.c:372 use_avx512f_memcpy_memset] avx512f supported
<libpmem>: <3> [init.c:380 use_avx512f_memcpy_memset] PMEM_AVX512F enabled
<libpmem>: <3> [init.c:514 pmem2_arch_init] using clwb
<libpmem>: <3> [init.c:523 pmem2_arch_init] using movnt AVX512F
<libpmem>: <3> [auto_flush_linux.c:86 check_domain_in_region] region_path: /sys/bus/nd/devices/region2
<libpmem>: <3> [auto_flush_linux.c:30 check_cpu_cache] domain_path: /sys/bus/nd/devices/region2/persistence_domain
<libpmem>: <3> [pmem.c:851 pmem_init] Flushing CPU cache
<libpmem>: <3> [pmem_posix.c:78 pmem_os_init] 

With IPMCTL, I can get the following information:

$ sudo ipmctl show -system NFIT
...
   ---TableType=0x7
      Length: 16 bytes
      TypeEquals: PlatformCapabilities
      HighestValidCapability: 0x02
      Capabilities: 0x00000002
      Capabilities.CPUCacheFlushToNVDIMM: 0
      Capabilities.MemoryControllerFlushToNVDIMM: 1
      Capabilities.MemoryMirroring: 0

I don't know whether I set my server wrong or it doesn't support eADR. Could you give me any advice?

Thanks,

Jungsik Choi

[Steve Scargall]

Hello Jungsik,

eADR is a platform feature, so it's best to check with your server provider. I am not aware of any eADR platforms available on the market today. You may still be able to experiment with eADR if the BIOS supports it. Be aware that if the host loses power or crashes, you could lose or corrupt data. On some vendor BIOS's, there's an option for eADR that you can set to Enable, Disable, or Auto. 

To determine whether the platform supports ADR or eADR, you see from the logs that libpmem checks the /sys/bus/nd/devices/region{N}/persistence_domain which will have values of 'memory_controller' (ADR) or 'cpu_cache' (eADR). In your case, the platform is currently in ADR mode, so we'll need to flush the CPU caches. 

<libpmem>: <3> [auto_flush_linux.c:30 check_cpu_cache] domain_path: /sys/bus/nd/devices/region2/persistence_domain
<libpmem>: <3> [pmem.c:851 pmem_init] Flushing CPU cache

/Steve 

[Jungsik Choi]

Hello Steve,

Thank you so much for your detailed explanation. It was very helpful.

I have another question.

I know that platforms that support eADR store additional power for power failure. Are there any other differences between eADR and ADR platforms? If there is no significant difference, I want to experiment with the code assuming eADR on my server even though my server does not support eADR. I wonder how much eADR affects the performance of applications. Could you advise on this?

Thanks,

Jungsik Choi

2021년 7월 8일 (목) 오후 11:03, Steve Scargall <steve.s...@intel.com>님이 작성:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/c75349c9-be77-48d1-9ac9-bc3bf069aa5bn%40googlegroups.com.

[Steve Scargall]

It all depends on how frequently the application writes, and how large those writes are. There's a great writeup and video explaining the benefits of eADR for applications on https://software.intel.com/content/www/us/en/develop/articles/eadr-new-opportunities-for-persistent-memory-applications.html. Watch the video at the bottom of the page for even more info and context. In the Cassandra example, you'll see there's significant performance improvements for eADR platforms vs CLWB and non-CLWB environments. Ice Lake is the first CPU to fully implement CLWB (write, and don't invalidate the cache line). Cascade Lake had the instruction but it was a wrapper/pointer to CLFLUSHOPT which invalidated the cache line. 

[Jungsik Choi]

Thank you for the good information. I am studying the video.

Thanks,

Jungsik Choi

2021년 7월 9일 (금) 오전 12:16, Steve Scargall <steve.s...@intel.com>님이 작성:

It all depends on how frequently the application writes, and how large those writes are. There's a great writeup and video explaining the benefits of eADR for applications on https://software.intel.com/content/www/us/en/develop/articles/eadr-new-opportunities-for-persistent-memory-applications.html. Watch the video at the bottom of the page for even more info and context. In the Cassandra example, you'll see there's significant performance improvements for eADR platforms vs CLWB and non-CLWB environments. Ice Lake is the first CPU to fully implement CLWB (write, and don't invalidate the cache line). Cascade Lake had the instruction but it was a wrapper/pointer to CLFLUSHOPT which invalidated the cache line. 

--

You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/d5071ab8-c1e9-4912-af2c-9d99a0bc0cb7n%40googlegroups.com.