how to integrate two DCPMM regions into one?

[Emeth Obando]

Hi All, 

I have a machine with 2 sockets, 24 DIMMs. 

I have 8 128GB DCPMM modules and each socket is equipped with 4 DCPMM, so each socket NVM size is 512GB. I would like to configure all these modules into App Direct Mode. I followed the following steps:

1. ndctl create-namespace --region region0 -–mode fsdax
2. ndctl create-namespace --region region1 -–mode fsdax
3. mkfs.ext4 /dev/pmem0
4. mkfs.ext4 /dev/pmem1
5. mount -o dax /dev/pmem0 /pmemfs0
6. mount -o dax /dev/pmem1 /pmemfs1

So now I have two mounting points with 512GB each. Considering 512GB size, it is not very large. Therefore, I am wondering if there is a way to integrate these two mounting points into one?

Many thanks for the help :)

[Anton Gavriliuk]

Hi 

> I am wondering if there is a way to integrate these two mounting points into one?

With your setup with 2 mount options you may have NUMA advantage if your Db/App understands and supports NUMA advantage.

If you want a single mount point, it may cause some perf degradation due to no NUMA access.

Several ways here

1. https://pmem.io/2018/05/15/using_persistent_memory_devices_with_the_linux_device_mapper.html

Creating dm-linear Devices

Creating dm-striped Devices

2. Or you can use LVM stripe or concat.

Anton

вт, 2 мар. 2021 г. в 17:32, Emeth Obando <oband...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/afb54350-0af4-4ed6-b9c1-c650451b4c38n%40googlegroups.com.

[Steve Scargall]

I agree with Anton, and advise against using SW RAID with memory as it's not designed to work at memory speeds, nor is it NUMA aware so you'll get a very high CPU Cache and local memory data miss rate which forces the CPUs to perform a lot of UPI operations. The overhead of software raid and CPU/Memory directory maintenance is substantial (relatively speaking). There are BIOS tunable options to reduce the overhead, but it's not zero. Performance is significantly impacted using RAID with Memory as demonstrated in Storage Redundancy with Intel® Optane™ Persistent Memory Modules. For the best performance, you should layout your app to be NUMA local. If the app isn't NUMA aware, use taskset or numactl to bind the app to the CPU socket where the PMem is (your data).