Free Entire PMEM Pool

[Davi Braga]

Hello everyone,

I'm developing an application where I have some data that's inserted into a persistent data structure using libpmemobj. In my use case, when it's no longer possible to allocate more persistent memory to insert new entries, I have to serialize all data and write it into disk using a traditional file API. After the data is transferred to disk, I have to free all the objects in the persistent data structure in order to insert new entries again. Considering my use case, I'd like to know if there is any efficient way to free all data in the pmempool without the need to iterate over all the persistent objects and free each one individually.

Thanks!

--

Davi Braga

DEVELOPER

davi....@lsbd.ufc.br

+55 (85) 3366 9235

+55 (85) 99773 1992

   davi.gomes64

[Andy Rudoff]

Hi Dave,

Pool creation is pretty fast.  Would it work to just delete the pool and create a fresh one?  Or are there some items in the pool that you don't delete?

-andy

[Steve Scargall]

Hi Davi,

Q) Do you currently use multiple pools or one giant pool?

Q) Are you able to continue working while you serialize and free or does the app have to wait for this operation to complete?

One thought would be to take Andy's idea and apply a log/pool rotation model to keep the IOs flowing.  Using 3 memory pools, one would be the active pool that your app writes to, one would be in an immutable state while you serialize the data to disk and perform compaction if necessary (then it gets destroyed and recreated - or freed), and the other is a brand new, or previously freed pool ready to take writes when the active pool gets full.  You could then assign different threads or processes to handle these activities. Switching between pools would be very quick.

For a single pool implementation, a potential approach would be to define a low-water and high-water mark (how full the pool is before you take action).  If you reach the low-water mark, you can kick off the serialization thread to start writing data out to disk and freeing the objects when the write is complete.  You can choose any number of well-known algorithms - FIFO, LRU, etc to decide which objects are written and removed.  If the ingestion of data into the pmem pool exceeds how fast you can write out to the SSD/HDD you'd hit the high-water mark which would either use a more aggressive serialization approach or queue the ingestion until you get caught up.

Just thought I'd share some ideas as I can imagine the current freeing approach can take several seconds to complete with large pools.

Regards,

    Steve