Allocating memory in PMDK vs frequent program restarts

[Jan K]

Hello,

Recently I encountered a strange problem with memory allocation. (By memory allocation I mean calling make_persistent<>() within a transaction.) Namely, when I allocate memory multiple times during a single run, I can allocate much more memory then in case I run a program multiple times, each time allocating few objects.

I expected that if, for instance, I can fit 1000 objects in a pool when I allocate all objects in one run, then I should be able to run a program 1000 times, each time allocating a single object. But this seems not to hold.

To illustrate the problem, I wrote a simple program (named 017, source code attached as well as posted in https://pastebin.com/Cxv2cK4T) that adds to a 'linked list' as many elements as requested in the first argument, obviously allocating memory for each element. If it fails to add an element, it prints out current list length and the transaction error, and exits with an error code.

Now when I run the program once attempting to allocate memory 100000 times, I get:

./017; ./017 100000
Count: 75664 (err: failed to allocate persistent memory object)
But when I run the program 100000 times, each time requesting a single memory allocation, I get:

./017; let i=100000; while let i--; do ./017 1 || break; done
Count:    36 (err: Could not add an object to the transaction.)

And when I try to run the program 10000 times allocating memory 10 times in each run, I get:

./017; let i=10000; while let i--; do ./017 10 || break; done
Count:   360 (err: Could not add an object to the transaction.)

(for ×100 I get 3600, for ×1000 I get 36000)

As far as I tracked down the "failed to allocate persistent memory object" and  "Could not add an object to the transaction", both seems to indicate running out of pool memory.

Now there is a vast gap between 75664 successful allocations and 36 successful allocations.

Am I doing something very wrong, or is it a flaw of PMDK?

Regards,

Jan

[OpenData Synth]

Block size of object?
...Subject|Depending
PMDK vs frequent program restarts
Kinda the same thing, Does the Object preform the functions in all but one run?
Is it a fail on last run to stop the app?
Could be an Auto-done or appkill

[Piotr Balcer]

Hi,

You've hit an already addressed problem where the first small allocation in an application does not reuse already existing runs for the allocation class,

which results in that first small allocation creating its own run, and all subsequent allocations in that instance of the application using that new run.

This makes the heap quickly run out memory for different allocation classes if your workload for a single instance of application is very small.

Notice that you are not running out of memory for allocating the objects, but rather your application is failing to create a snapshot - this is where the

problem is. pmemobj tries to allocate the space for snapshots, but it's unable to do so.

I recommend upgrading to a newer version of libpmemobj. This problem does not occur on 1.5.

Piotr

[Jan K]

Thanks for the reply. I'm kind of relieved to know it's not me missing
something easy to spot in my code, and that it's going to go away.
I'll give a try to 1.5-rc1.

> You've hit an already addressed problem

I must have missed that. Did I oversee a list o known issues / bug tracker?

> Notice that you are not running out of memory for allocating the objects,
> but rather your application is failing to create a snapshot - this is where
> the problem is. pmemobj tries to allocate the space for snapshots, but
> it's unable to do so.

If I get it right, it's trying to allocate memory for transaction logs
and fails to do so.
I assume transaction logs and objects use the same memory space, so it
essentially boils down to running out of space, doesn't it?

>> ./017; *./017 100000*
>> Count: *75664* (err: failed to allocate persistent memory object)

>> But when I run the program 100000 times, each time requesting a single
>> memory allocation, I get:

>> ./017; let i=*100000*; while let i--; do *./017 1* || break; done
>> Count: *36* (err: Could not add an object to the transaction.)

>> And when I try to run the program 10000 times allocating memory 10 times
>> in each run, I get:

>> ./017; let i=*10000*; while let i--; do *./017 10* || break; done
>> Count: *360* (err: Could not add an object to the transaction.)

>> (for ×100 I get 3600, for ×1000 I get 36000)
>>
>> As far as I tracked down the "failed to allocate persistent memory object"
>>
>> and "Could not add an object to the transaction", both seems to indicate
>>
>> running out of pool memory.
>>
>> Now there is a vast gap between 75664 successful allocations and 36
>> successful allocations.
>>
>> Am I doing something very wrong, or is it a flaw of PMDK?
>>
>> Regards,
>> Jan
>>
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pmem+uns...@googlegroups.com.
> To post to this group, send email to pm...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pmem/e3a443ec-3ab6-4f8f-9609-1904d1d15b42%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Piotr Balcer]

Thanks for the reply. I'm kind of relieved to know it's not me missing
something easy to spot in my code, and that it's going to go away.
I'll give a try to 1.5-rc1.

This was behavior that was introduced during allocator changes, and then subsequently
addressed once we've noticed the problem during regular development, so I don't think
there was an issue created. But since you've asked, here's our issue tracker:
https://github.com/pmem/issues/issues

 

> You've hit an already addressed problem
I must have missed that. Did I oversee a list o known issues / bug tracker?

> Notice that you are not running out of memory for allocating the objects,
> but rather your application is failing to create a snapshot - this is where
> the problem is. pmemobj tries to allocate the space for snapshots, but
> it's unable to do so.
If I get it right, it's trying to allocate memory for transaction logs
and fails to do so.
I assume transaction logs and objects use the same memory space, so it
essentially boils down to running out of space, doesn't it?

They do use the same heap, but due to how the allocation algorithms work,
the request to allocate the log (~32kilobytes) failed but you could have still allocated
thousands of list objects (16 bytes).
The problem isn't that the heap ran out of memory to give, but rather that
a single freelist (bucket in libpmemobj's terms) consumed all of the still
available memory. On top of that, the single allocations that were successfully
allocated prevented the memory blocks from being coalesced.

[Adrian Jackson]

Is that a haiku? 😉

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/46f909fe-9ec2-4b0e-871c-d8d86968b7b0%40googlegroups.com.

[Piotr Balcer]

I was wondering that myself.

My best guess is some machine learning experiment in the wild ;)