Ideas on how to support multiple object pools on the same devdax device.

[Daniel Waddington]

Hi,

I need to be able to sub-partition a devdax device, so that I can map a pool to some sub-region (i.e. support multiple object pools on the same /dev/dax0.xx device).  I'm not sure how to go about this, for example, is there a way to create a pool without a file but with just a region of memory?  Having two-layer allocation is sort of fundamental.

BTW, I need devdax because its being used for RDMA, otherwise fsdax would solve the problem.

Daniel

[Steve Scargall]

Hi Daniel,

The intent of DEVDAX was to provide a single contiguous memory mapped region to the application.  libpmem sees this as a single device/file and so can only support a single root object per pool.  The idea of multiple root objects per pool has come up in the past.  The idea of supporting multiple KV Trees per pool is also being discussed for pmemKV.  Since the root object should point to an application data structure, can you use this to point to a custom data struct that in turn points to multiple objects/structs within the application space?

As an FYI, There is an open feature request to support RDMA registration on FSDAX by the Linux community.  On-Demand Paging (ODP) with FSDAX is also being looked at.

One option, if it's feasible for your situation, is to create multiple DEVDAX devices per region and keep the one memory pool/root object per devdax device, eg to create 4 x ~54GB DEVDAXdevice instances on the same region:

$ ndctl create-namespace --region=region0 --mode=devdax --size=54G

$ ndctl create-namespace --region=region0 --mode=devdax --size=54G

$ ndctl create-namespace --region=region0 --mode=devdax --size=54G

$ ndctl create-namespace --region=region0 --mode=devdax --size=54G

$ ls -l /dev/dax*

crw-------. 1 root root 250,  7 Nov 21 05:10 /dev/dax0.0

crw-------. 1 root root 250,  8 Nov 21 05:10 /dev/dax0.1

crw-------. 1 root root 250,  9 Nov 21 05:10 /dev/dax0.2

crw-------. 1 root root 250, 10 Nov 21 05:10 /dev/dax0.3

Would that satisfy the requirement?

- Steve

[Daniel Waddington]

Thanks Steve,

Yes, I know about these options.  Although in theory I could use one big pool, it makes it harder to allocate an amount of the resource (i.e. do admission control) due to issues around fragmentation etc.  I am using namespaces at the moment.  I think that I will write a crash-consistent heap allocator that I can overlay on a pmemobj allocated contiguous region.

Daniel

[Piotr Balcer]

Namespaces are the correct solution for this problem. Is there any problem which prevents you from using them?

Doing this in libpmemobj just seems redundant.

Piotr