Regions

[Charlie Dong]

Hi

I have 4 PM DIMMs in one server and some how they are grouped into 2 regions: region0 and region1. I did not find a region creation command in ndctl. My question is that how to create only one region, that includes all dimms: nmem0-3?

Regards

Charlie

Sent from my iPhone

[Andy Rudoff]

Hi Charlie,

How DIMMs are configured into interleave sets is vendor-specific, so you configure it with a vendor-specific tool.  If your persistent memory is Intel Optane DC persistent memory, you use the "ipmctl" command to configure it (or you can do it from the BIOS UI).  Your choices are to have one interleave set per socket (the system will not interleave persistent memory across sockets), or to turn off interleaving completely and put each memory module in its own set.

If the modules are from another vendor, you will need to ask them what configurations are supported and how to configure them.

Anyway, it sounds like you want to put them all into a single interleave set, and like I said above, that's not supported.  But if you're using Linux you can essentially do what you want at the SW level using the Linux Device Mapper.  Steve Scargall wrote a very detailed description about it here: http://pmem.io/2018/05/15/using_persistent_memory_devices_with_the_linux_device_mapper.html

-andy

[Charlie Dong]

Hi Andy,

Thank you very much! That’s explained.

One thing is not clearly defined in Intel’s namespace spec. Namespace can be built across multiple dimms through interleaving set. As you pointed, the interleaving set can not across multiple sockets, region can not across multiple interleaving sets and namespace can not across multiple regions. Am I right?

Regards

Charlie

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/384a7938-ff9c-4ea9-bc01-6a75c8e41bdf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andy Rudoff]

Hi Charlie,

The Linux term "region" is really the same thing as what we call "interleave set" at the HW level.  They always match up 1:1.  An interleave set (or region) can be sub-divided into namespaces.  You are correct that namespaces cannot be used to combine multiple interleave sets together.  To do that, you need to either use the device mapper or a library like PMDK which combines them together via the "poolset" feature where you create a PMDK pool from multiple files

-andy.

On Tuesday, March 26, 2019 at 9:39:13 AM UTC-6, Charlie Dong wrote:

Hi Andy,

Thank you very much! That’s explained.

One thing is not clearly defined in Intel’s namespace spec. Namespace can be built across multiple dimms through interleaving set. As you pointed, the interleaving set can not across multiple sockets, region can not across multiple interleaving sets and namespace can not across multiple regions. Am I right?

Regards

Charlie

Sent from my iPhone

On Mar 26, 2019, at 05:41, Andy Rudoff <an...@rudoff.com> wrote:

Hi Charlie,

How DIMMs are configured into interleave sets is vendor-specific, so you configure it with a vendor-specific tool.  If your persistent memory is Intel Optane DC persistent memory, you use the "ipmctl" command to configure it (or you can do it from the BIOS UI).  Your choices are to have one interleave set per socket (the system will not interleave persistent memory across sockets), or to turn off interleaving completely and put each memory module in its own set.

If the modules are from another vendor, you will need to ask them what configurations are supported and how to configure them.

Anyway, it sounds like you want to put them all into a single interleave set, and like I said above, that's not supported.  But if you're using Linux you can essentially do what you want at the SW level using the Linux Device Mapper.  Steve Scargall wrote a very detailed description about it here: http://pmem.io/2018/05/15/using_persistent_memory_devices_with_the_linux_device_mapper.html

-andy

On Tuesday, March 26, 2019 at 12:00:36 AM UTC-6, Charlie Dong wrote:

Hi

I have 4 PM DIMMs in one server and some how they are grouped into 2 regions: region0 and region1. I did not find a region creation command in ndctl. My question is that how to create only one region, that includes all dimms: nmem0-3?

Regards

Charlie

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "pmem" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+unsubscribe@googlegroups.com.

[Charlie Dong]

Hi Andy,

Understood! Thank you very much!

Charlie

Sent from my iPhone

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/ccffb560-78a9-47ce-9841-9760fcf07d79%40googlegroups.com.

[J Kim]

I'm Junghan from Korea and PhD student. 

If I want to use the entire PM as single storage, Device Mapper in Linux is the best option? I'm so curious about why not supported by processor and IMC? The bigger problem I think is that even when using Device Mapper, PMDK does not allocate memory considering SW-based interleaved PM. It allocates the memory to only Socket-0's PM. Please let me know. Thanks. 

2019년 3월 26일 화요일 오전 8시 41분 25초 UTC-4, Andy Rudoff 님의 말:

[Steve Scargall]

On Thursday, October 31, 2019 at 9:03:24 AM UTC-6, J Kim wrote:

I'm Junghan from Korea and PhD student. 

If I want to use the entire PM as single storage, Device Mapper in Linux is the best option?

Software RAID is one option, but may not be the 'best' option.  It depends on the application requirements.  'Best' is not necessarily the 'optimal' solution.  The 'optimal' solution is usually not to use SW RAID.  This includes Device Mapper, mdadm, or LVM.  Note that only the stripe and linear DeviceMapper code has DAX enablement.  So only 4- or 8-socket systems, doing more complex RAID levels will not work with DAX.

Similar to the SW RAID solution are 'poolsets'.  PMDK supports persistent memory pool sets, where a larger pool can be created from two or more smaller pools.  Specifically, those smaller pools may be created on different NUMA nodes/CPU Sockets.  The following example poolset config file creates a 400GiB memory pool using 2 x 200GiB files, one from CPU0 and one from CPU1 (assumes you have created the regions, namespaces, and mounted them on /pmemfs0 and /pmemfs1).  At this time, we do lose NUMA locality information as PMDK will concatenate the smaller pools so you can lose performance depending on which CPU the thread is running when accessing the data.

PMEMPOOLSET
OPTION NOHDRS
200G /pmemfs0/myfile.part0
200G /pmemfs1/myfile.part1

See the poolset(5) man page for more info.

 

I'm so curious about why not supported by processor and IMC?

Intel platforms are NUMA (Non-Uniform Memory Architecture).  To have CPUs access any address with the same latency requires UMA (Uniform Memory Architecture).  

Simplistically, Intel CPUs have an integrated memory controller (IMC) that manages memory physically located on that CPU sockets.  If a request occurs on a CPU that is out of range, the CPU has a directory it can lookup to forward the request to the CPU that does manage the requested memory address.  The request then goes over the UPI interface to satisfy the request.  To learn more about how the UPI and directory updates work, take a look at these videos:

Choose the Right Persistent Memory Mode: Part 1

Choose the Right Persistent Memory Mode: Part 2

The BIOS has several tunables to manage the directory behaviour.

 

The bigger problem I think is that even when using Device Mapper, PMDK does not allocate memory considering SW-based interleaved PM. It allocates the memory to only Socket-0's PM. Please let me know. Thanks. 

When you introduce RAID, we lose the NUMA locality information.  For this reason, it is recommended to make the application NUMA aware, if not already, meaning it can memory map files (persistent memory pools) from any of the CPU sockets on the host and ensure threads run on the appropriate CPU.  There are several techniques to achieve this including, but not limited to, some of the following:

• The app creates pools of worker threads to handle the data requests.  Each thread pool is assigned to a specific CPU socket using a local persistent memory pool
• The app assigns specific data structures to specific persistent memory pools

In summary, apps should be NUMA aware and be designed to access data optimally by running threads on the CPUs closest to the data to which they need access (read or write).

HTH

   Steve