pmemobj allocation failure while there is still space left
55 views
Skip to first unread message
Yue Li
Jun 11, 2018, 9:14:36 PM6/11/18
to pmem
Hi everyone,
We tried to test the object allocation limitation of PMEM using dax device and encountered the following issue:
We first create a 4K-aligned dax device /dev/dax5.0 and create a new pool. The pool size is 33820770304:
/usr/local/bin/pmempool info -s /dev/dax5.0
Part file:
path : /dev/dax5.0
type : device dax
size : 33820770304
alignment : 4096
Then we tried to create as many 2MB-object as possible, and we saw failure after creating object #8061. (The code is at the end of this mail)
./a.out /dev/dax5.0
ERROR:alloc 8061 failed: 12 <==== Got NOMEM here
/usr/local/bin/pmempool info -s /dev/dax5.0
Part file:
path : /dev/dax5.0
type : device dax
size : 33820770304
alignment : 4096
POOL Header:
Signature : PMEMOBJ
Major : 4
Mandatory features : 0x0
Not mandatory features : 0x0
Forced RO : 0x0
Pool set UUID : 8eb69225-ed2e-41a4-8b1d-fc090ff5bfe2
UUID : 8d3585d3-d552-4170-8dc7-c275736896c0
Previous part UUID : 8d3585d3-d552-4170-8dc7-c275736896c0
Next part UUID : 8d3585d3-d552-4170-8dc7-c275736896c0
Previous replica UUID : 8d3585d3-d552-4170-8dc7-c275736896c0
Next replica UUID : 8d3585d3-d552-4170-8dc7-c275736896c0
Creation Time : Tue Jun 05 2018 03:11:51
Alignment Descriptor : 0x000007f737777310[OK]
Class : 64
Data : 2's complement, little endian
Machine : AMD X86-64
Checksum : 0xc6a82aa2b670b739 [OK]
PMEM OBJ Header:
Layout : (null)
Lanes offset : 0x2000
Number of lanes : 1024
Heap offset : 0x302000
Heap size : 33817616384
Checksum : 0x2f83780000304400 [OK]
Root offset : 0x0
Statistics:
Objects:
Number of objects : 8061
Number of bytes : 19717850880
Objects by type:
Type number : 0
Number of objects : 8061 [100 %]
Number of bytes : 19717850880 [100 %]
Heap:
Number of zones : 2
Number of used zones : 2 [100 %]
Zone 0:
Number of chunks : 4096
free : 1 [0.024414 %]
run : 4095 [99.975586 %]
Total chunks size : 65528
free : 8 [0.012209 %]
run : 65520 [99.987791 %]
Zone's allocation classes:
Unit size : 262144
Units : 8
Used units : 0 [0 %]
Bytes : 2097152
Used bytes : 0 [0 %]
Total bytes : 2097152
Total used bytes : 0 [0 %]
Zone 1:
Number of chunks : 3967
free : 1 [0.025208 %]
run : 3966 [99.974792 %]
Total chunks size : 63471
free : 15 [0.023633 %]
run : 63456 [99.976367 %]
Zone's allocation classes:
Unit size : 262144
Units : 15
Used units : 0 [0 %]
Bytes : 3932160
Used bytes : 0 [0 %]
Total bytes : 3932160
Total used bytes : 0 [0 %]
Total zone's statistics:
Chunks statistics:
Number of chunks : 8063
free : 2 [0.024805 %]
run : 8061 [99.975195 %]
Total chunks size : 128999
free : 23 [0.017830 %]
run : 128976 [99.982170 %]
Allocation classes:
Unit size : 262144
Units : 23
Used units : 0 [0 %]
Bytes : 6029312
Used bytes : 0 [0 %]
Total bytes : 6029312
Total used bytes : 0 [0 %]
Here's the code that we use for doing the allocation tests:
int main(int argc, char** argv)
{
int i = 0, size;
PMEMobjpool *pool = pmemobj_open(argv[1], NULL);
if (!pool) {
err("Open pool failed %d\n", errno);
return 1;
}
size = 2<<20; /* Here to adjust the size */
PMEMoid oid;
while (true) {
int rc = pmemobj_zalloc(pool, &oid, size, 0);
if (rc < 0) {
err("alloc %d failed: %d\n", i, errno);
return 1;
}
i++;
}
pmemobj_close(pool);
return 0;
}
Thanks!
Best,
Yue
Piotr Balcer
Jun 12, 2018, 10:14:10 AM6/12/18
to pmem
Hi Yue,
libpmemobj uses what I call a two-level segregated fit algorithm for small and medium allocations, this means that a single allocation class, contrary to common designs, serves multiple sizes from a single "run" of memory. To illustrate, let's say that we have an allocation class of 100 bytes, and a single run has 10 units (each of 100 bytes). Allocating 700 bytes from that run would take 7 units and leave 3 "leftover" units. Those leftover units are still available for allocation, and if you were to try to allocate 300 bytes, you would get three of those leftover units.
This is exactly what's happening in your example.
To solve this problem in the general case, we've developed a mechanism to create allocation classes in runtime:
http://pmem.io/pmdk/manpages/linux/master/libpmemobj/pmemobj_ctl_get.3 (see `heap.alloc_class.new.desc`).
I *strongly* encourage to use this mechanism for any workloads that use a limited number of allocation sizes. It entirely eliminates fragmentation.
Having said that, we target internal fragmentation to be below 5% on average, and your example exceeds that significantly, so I have decided to
tune the allocation class selection algorithm slightly to favor classes with smaller "leftover memory". You will find the change here:
Thanks for reporting this.
Piotr
Yue Li
Jun 12, 2018, 1:32:33 PM6/12/18
to pmem
hi Piotr,
Thanks for these insights, they are really helpful!
While waiting for the PR to be merged, we will try to register a 2MB-slab allocator you mentioned for a quick test. Thanks!
Best,
Yue
Reply all
Reply to author
Forward
0 new messages