extending the pool by appending parts with headers is not supported!

[tal wertheimer]

Hi,

I have a Rust program that uses the pmdk library (1.6 and 1.8).

the program is ran on ubuntu with intel cpus and nvme flash disk (not persistent memory).

After creating a pmem pool with capacity of 128 objs (pool size is 128 * (obj _size + 64) * 3), when I'm calling pmemobj_alloc from more then 11 threads (from rust cpupool - a thread pool intended to run CPU intensive work), and the obj size is between 24021bytes - 184317bytes  get this error consistently on the 12'th allocation: "extending the pool by appending parts with headers is not supported!"

(I tried obj_size of 184341 and more, and  8000 and less,  and it passed)

With other thread pool (tokio_threadpool) is less consistent:

It fails after about 100 allocations if I create the file in /tmp : /tmp/tal/__dirname/filename.pmemobj (probably ext4 file system),

and succeeds if I create the file in the local dir: ./__dirname/filename.pmemobj.

the call stack is probably this:

 util_pool_extend
   heap_extend
     heap_ensure_huge_bucket_filled
        heap_get_bestfit_block
           palloc_reservation_create
              palloc_operation/
                obj_alloc_construct
                   pmemobj_alloc

I didn't see in the code where I could set this OPTION_SINLGLEHRD flag.

What does this flag stand for? And how can I set it if I want to?

Do you have any idea why I got this error anyway?

why does the pool need to be expended after just 11 parallel obj allocations even though it is much bigger than that?

more information:

when there is less than 12 cpus in the machine the error didn't occur.

I tried to reproduce the issue in c code: created 100 threads that ran 100 allocation in loop each. it didn't reproduce.

thanks!

[P B]

Hi,

Attempt at a pool extension is simply the last step in the allocation process if there's no available memory in the heap.

This message is telling you that the heap couldn't be extended because of how the pool was created. This is expected.

TBH, I think we should change the message because it's confusing as hell.

As for your actual problem, this is the effect of arena allocation. The heap is over-provisioning memory for individual

threads for the purpose of scalability, but this has the side-effect of reducing the globally available memory.

See description of arenas in CTL namespace man page: https://pmem.io/pmdk/manpages/linux/master/libpmemobj/pmemobj_ctl_get.3

Piotr

[wer...@gmail.com]

Thanks for the quick answer!

Well, It seems that set this OPTION_SINGLHDR will solve me issue. the problem is i can't find how can I set this flag. should I change it manually after I get the pool from calling pmemobj_create\open?

Thanks!

ב-יום חמישי, 2 ביולי 2020 בשעה 10:41:27 UTC+3, ppbb...@gmail.com כתב/ה:

[Piotr Balcer]

It will solve your issue by simply extending the heap, if that's what you want, then you need to use a poolset:

https://pmem.io/pmdk/manpages/linux/master/poolset/poolset.5

Something like this should work:

PMEMPOOLSET
OPTION SINGLEHDR
500G /mnt/pmem/my_pool/

[Kit]

Thanks! It does work, but now, I get sometimes this message:

"exceeded reservation size"

When I try to allocate in parallel 128 objs with size of 49152B in a pool with this configuration:

PMEMPOOLSET

OPTION SINGLEHDR

18898944B /tmp/tal/__store

While 18898944B came from the calculation: 3 * (obj_size + 64) * capacity = 3 * (49152 + 64) * 128

Why do I get this message?

BTW, double the pool size solves the problem

[ppbb...@gmail.com]

Well, this is simply telling you that the heap cannot be extended anymore since the reservation size has been reached.

Reservation in this context means the size of the virtual address space reserved for use in the pool. Reservation

can be much much larger than the whole heap, since this is only related to address space the pool will use.

Set it to 1TB and things will work. If the reservation size is larger than the file system capacity, then the heap will

return ENOMEM only once the capacity is exhausted.

The sizes you are using are too small to be practically usable in a multi-threaded application with the default

allocation classes. To reduce space usage you can reduce the number of arenas used by the allocator and

use custom allocation classes tailored for your use case.

Also, a libpmemobj pool has about 7 megabytes of fixed metadata, so the usable size of the pools you are using

is much smaller.