libpmemblk question

[Don Doerner]

Hello,

I have the intent to use libpmemblk on a flash-based SSD to implement a block device cache.  When a line is evicted from cache (i.e., no longer has relevant content), I'd like to discard it, to optimize the garbage collection on the SSD, and in general, prolong it's life.

I don't see an interface for doing this, though I imagine it might be buried in pmemblk_set_zero() or pmemblk_set_error().  Any idea how to get where I need to go?

Thanks in advance,

Don Doerner

[Andy Rudoff]

Hi Don,

A flash-based SSD is does not fit our definition of "persistent memory" since it is not byte-addressable and flash media is too slow for direct load/store access.  All the NVM Libraries, including libpmemblk, are optimized for persistent memory.

Of course, if you just want to try out the APIs they will work on any file, like a file on a flash SSD.  Just use pmemblk_create() on a normal file system and it will work, it just won't perform very well because each time the library wants to flush a cache line, it will use msync() which can only flush entire pages.

Anyway, it sounds like a mismatch of use cases.  The libraries are optimized algorithms for persistent memory, and it sounds like you're trying to build something to minimize the impact of flash media management.  So I'm not sure there's a good fit here.

Thanks,

-andy

[Don Doerner]

HI Andy,

Thanks for your support on this.  Let me put a bit more around what I am trying to do, to see if it changes your opinion that it is a mismatch...

I am building a cache for storage -- conceptually, a cache above a block device to keep a working set available at reduced latency and improved throughput.  The cache is intended to be large (a significant fraction of the block device, perhaps a handful of TB) and non-volatile (prototyping will be based on Intel P3700 NVMe SSDs).  Along with the cached data, some small amount of metadata must also persist.

I would find it convenient to address this storage as memory - and the size of the cache lines (a few MB at most) is easily manageable in memory, the processors involved support a 48-bit-sized virtual address space, so mapping a large device doesn't present a problem, and so on.  The stars seem aligned.

Moreover, the design of the cache is such that I'll need transactions -- the design of the NVM library's transactions is pretty much exactly what I need.

Occasionally (;-}) I'll want to allocate a "new" cache line from a ready-to-use pool.  That pool is produced by evictions (based on lack of access).  That's where my desire to TRIM comes up.  When I evict the cache line, the data is irrelevant, no reason for me to go through a garbage collection process -- my modelling suggests that this may improve the performance and extend the life of the flash-based devices that I use.

[Andy Rudoff]

Hi Don, thanks for the clarification.

If you're building a cache and the cache is pmem (byte-addressable persistent memory) then libpmemblk is the library for you.  But in your description you mention using an NVMe SSD as the cache.  To access that as memory, you'll have to do paging -- an NVMe device only speaks blocks.  You could, for example, put a file system on the NVMe SSD, create a big file, and mmap() it.  Now you have byte-addressable storage, but not persistent memory by my definition, because you are actually doing loads and stores to the page cache, as the system transfers pages to/from the NVMe device.  If you used libpemblk for this, each time we update a block atomically, we do an additional three writes (as part of making the write atomic).  These writes are small and on persistent memory they result in three cache lines being flushed from the CPU caches.  But if you used that same algorithm on an NVMe SSD, it will work correctly, but those three extra writes will turn into three page writes from the system page cache to the NVMe device!  That's not what you want.  Instead, for your use case, you want an algorithm designed to provide atomic updates on storage instead of persistent memory.  There are lots of those, but working on top of block storage is not a goal of NVML, so it probably isn't a good choice for you.

Hope that helps,

-andy

[Don Doerner]

Yes, it does.  It looks like the reasonable thing is to find another way.  Thanks again for your assistance.

-don-