Samsung's KV-SSD alternative to pmem

[Niall Douglas]

Wearing my "WG21 Elsewhere Memory" standards hat, I've been dialoging recently with a Samsung rep for KV-SSD, which is an alternative approach to solving many of the problems which persistent memory solves. KV-SSD exposes the internal key value store which top end SSDs implement directly to the userspace program, avoiding any kernel, filesystem or database overhead entirely. For the key value store use case which many problems can be reduced down into, this approach has many strengths.

Quick summary of KV-SSD: https://www.snia.org/sites/default/files/SDC/2017/presentations/Object_ObjectDriveStorage/Ki_Yang_Seok_Key_Value_SSD_Explained_Concept_Device_System_and_Standard.pdf

Samsung's equivalent to the Intel PMDK for KV-SSD: https://github.com/OpenMPDK/KVSSD

I'd be interesting in seeing what this list's members think of this approach. Of course there are some things which only NV-DIMM pmem can do, but those are not in common use practice right now, whereas the key value store has tons of big iron users already, ones which might have been expected to enthusiastically adopt pmem combined with a lightweight, pmem-native, embedded database to manage the raw mapped non-volatile memory. Those big iron users may now not look as favourably on pmem as a solution to their needs, as they have a hardware accelerated key value store now.

Low latency storage just keeps on getting more exciting! We're on the cusp of a brave new world right now.

Niall

[Daniel Waddington]

It's not load/store addressable, so in my mind still a storage device albeit with atypical interface. It might be more interesting on the fsb, e.g. ultrapath.

Daniel

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/04d2987a-d87b-4b34-95a7-9b98a0b9adc0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Niall Douglas]

On Thursday, October 25, 2018 at 3:13:50 AM UTC+1, Daniel Waddington wrote:

It's not load/store addressable, so in my mind still a storage device albeit with atypical interface. It might be more interesting on the fsb, e.g. ultrapath.

Ah, well, maybe it's much more than that.

As some here may remember, the C++ standard committee is working (very slowly, due to this happening in people's outside-of-work free time) on getting mapped memory to not be undefined behaviour, and with a bit of luck, perhaps eventually some understanding within the C++ abstract machine for non-volatile RAM.

What nobody here likely knows is that that work actually began from the need for an embedded key-value store, which we need for C++ Modules, and in a number of other areas, to progress the language and ecosystem. This embedded key-value store is highly unfancy - you store some key-values durably, you later look up key-values. No indexing, no other databasey stuff. But it would be absolutely core to the C++ and C language. As in, no future C or C++ program could run without this key-value store, because it literally implements program bootstrap i.e. what happens between kernel launching the process, and execution reaching main(). Also, the toolchain would be embedded into this store as well, as the linker would write Modules into the store, as well as the program.

We sorely need one of those, and about six years of evolution and two empirical prototypes has gone into the design we have in mind so far. So when Samsung's KV-SSD comes along, we are naturally very, very interested, as this solves our actual need whereas persistent memory was merely an avenue to getting us to a solution.

Unfortunately, Samsung KV-SSD in its current form falls well short of what we need, and if I am to be honest, it's well short of what is useful. However from reading the KV-SSD kernel driver source files, 95% of what's lacking can be fixed in the kernel driver, with gen 2 KV-SSD fixing the remaining shortcomings on the device. So I believe it is doable that gen 2 KV-SSD and the standard C++ key value store can meet up in the middle.

Obviously for those without KV-SSD devices the C++ key value store uses your filing system, and on embedded devices, a block of flash. So future C++ will work fine on all existing and legacy hardware, but considerably better on machines with a hardware key-value device.

I have yet to formally run this stuff past WG14 the C programming language committee, but I have heard generally warm noises so far. So if they also approve this, all C programs would be similarly based.

Niall

[Adrian Jackson]

Interesting, what's the requirement to use a hardware KV store to implement modules? Is it storing compiled routines for later import by applications?

cheers

adrianj

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/90c12d80-f011-4e29-9150-b0baf1651ac7%40googlegroups.com.

[Niall Douglas]

On Thursday, October 25, 2018 at 10:35:26 AM UTC+1, adrianj.jackson wrote:

Interesting, what's the requirement to use a hardware KV store to implement modules? Is it storing compiled routines for later import by applications?

Firstly, there is no need for a hardware KV store. But if you have one to make things go much faster, then great. Otherwise it falls back onto software.

Modules in C++ 23 will be statically linked time affairs quite like object files are today. So your linker inspects the filesystem, and yanks in Modules you imported, and performs optimisation passes on the merged ASTs to make a shared library or program.

Modules for C++ 26 are hoped to be rather more useful. Gabriel dos Reis via Microsoft is working on a new linker table format through which symbols get resolved. It is a partial AST based format, and includes programming languages far outside C++. Modules will be able to be bound into a single file with this standard interface definition format (currently called IFC), and can be dynamically loaded into multiple programs at once. If this looks like shared libraries, that's because they are exactly that. Latest MSVC in experimental mode can already generate these shared Modules with IFC bindings, so if you'd like, you can give that a whirl with your own code.

When linking a program which uses shared libraries, you need to write into your program which shared library and which version thereof. Same goes for Modules. Difference is that the C and C++ program will no longer need the kernel or OS to dynamically load shared libraries for it, instead it'll be built in to C++ and the C++ standard.

Hence the need for an embedded key value store, so each C++ program can bootstrap itself into existence.

Down the line we also want to be able to fetch missing dependency shared Modules from an internet based central repository. The standard C++ key value store will be able to do that as well, fetch missing dependencies from the internet and place them into the local store on first run. I know languages like Python had that fifteen years ago, but we are very conservative in C++. We take our time!

Does this make sense? Everything I have just said above is current best plans, and anything or everything may change between now and the relevant standards. Whilst me and some of the committee leadership have informally discussed these plans, none of the above has seen a formal committee vote, and therefore it should be considered only as informed conjecture.

Questions are welcome, of course.

Niall

[Adrian Jackson]

Thanks, very interesting. I have a natural aversion to automatically downloading stuff but I'm sure the security side of it can be handled. The challenge going from filesystems to kv stores or persistent memory, from my perspective, is that filesystems do at least 3 things rolled into one:

1) persistent storage

2) namespace

3) permissions and security

I can imagine its straight forward to use the KV for persistent storage, but namespace and permissions/security become more challenge in multi-user, multi-install systems (i.e. where do you go to look for Modules, which versions to you choose, who is allowed to update them, how do you look for user specific ones and manage the versions there). Not insurmountable challenges but may mean KV stores end up looking a lot like filesystems in the end ;)

My experience issues with dynamic linking of libraries on systems with lots of users and lots of software installs is that it can become really problematic so I imagine careful design work is needed to ensure the same doesn't happen for Modules.

Good luck with it.

cheers

adrianj

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/5cf99e14-fc09-4627-b727-8126643943c0%40googlegroups.com.

[Niall Douglas]

I can imagine its straight forward to use the KV for persistent storage,

Maybe. A major difference between the C++ and SNIA KV stores is atomic transactions (we have those, they need to gain them).

 

but namespace and permissions/security become more challenge in multi-user, multi-install systems (i.e. where do you go to look for Modules, which versions to you choose, who is allowed to update them, how do you look for user specific ones and manage the versions there). Not insurmountable challenges but may mean KV stores end up looking a lot like filesystems in the end ;)

In our proposal, that's gated by the URI. So you open our KV store on a URI, and permissions checks happen at that time. Once opened, you have free reign. Note read only stores, and attribute only access is possible.

 

My experience issues with dynamic linking of libraries on systems with lots of users and lots of software installs is that it can become really problematic so I imagine careful design work is needed to ensure the same doesn't happen for Modules.

As with everything in C++, we serve a hideously complex menu of choice to the developer and leave them shoot themselves in their foot!

Niall 

[Brian Bulkowski]

Hi.

The KV-SSD work is loosely derived from the KV Device work at FusionIO, which I and my database company ( Aerospike ) worked on with FusionIO, which if memory serves turned into OpenNVM. ( And had nothing to do with the C++ standards work ).

I showed the prototype of Aerospike with KV underlying systems at a conference in 2011.... it was in santa clara.... forget the name....

The Samsung interfaces are slightly better than the OpenNVM interfaces we created, but doesn't solve a few harder problems - like fixed key sizes. We tried to solve iteration issues, but the interface was never quite good enough. And replication becomes hard, because you never just store your data in literally one device.

Now.... I think you're seeing similarity because PMEM has also created an object storage API.

OpenNVM / KV Drives solve an interesting problem: how do you integrate defragmentation at the database layer with the FTL required in the device itself. If you support a KV interface, the database doesn't need to defragment, allowing lower wear and higher performance. PM doesn't have a defragmentation problem, so ( theoretically ) a device written to the One True Object Layer wouldn't have to build a defrag layer in either case.

It was my original intent ( when I cooked up these ideas with David Flynn in 2009 ) that we would change the read() and write() block interfaces at the kernel layer into object storage APIs. I know, big dream.

In reality, I've learned that the interface abstraction of Block probably is the right layer for OSes, because there are too many extra bits you want in the block layer ( such as automatic expiration ). therefore I am less hopeful in the promise of systems like KV and pmem's object layers. While I may be wrong someday, when we did the Aerospike PM integration we looked at PMEM objects, and they are far slower than what we were able to achieve with libpmem and our own code.

Thus we have two problems.

One is API. If device people are going to try this problem, they should all be coding to a single API, instead of libpmem and KVSSD and NVM and C++ and RocksDB ( which is a library despite the "DB" ) and whatnot.

Second is implementation. If device companies are going to promote that API, they need to have a good implementation.

-brian

[Niall Douglas]

The KV-SSD work is loosely derived from the KV Device work at FusionIO, which I and my database company ( Aerospike ) worked on with FusionIO, which if memory serves turned into OpenNVM. ( And had nothing to do with the C++ standards work ).

Firstly thank you for your service and contribution to computer science and engineering.

 

One is API. If device people are going to try this problem, they should all be coding to a single API, instead of libpmem and KVSSD and NVM and C++ and RocksDB ( which is a library despite the "DB" ) and whatnot.

You can see the current draft prototype API at https://github.com/ned14/llfio/blob/master/include/kvstore/kvstore.hpp#L484. It is not implemented yet, it is merely there to aid standards committee discussion.

You will note very significant differences to the SNIA proposed API, though I believe that this API is implementable with the SNIA API.

Keys are always fixed sized integers. If people want arbitrary keys, they can hash them themselves into a fixed size integer.

 

In reality, I've learned that the interface abstraction of Block probably is the right layer for OSes, because there are too many extra bits you want in the block layer ( such as automatic expiration ). therefore I am less hopeful in the promise of systems like KV and pmem's object layers. While I may be wrong someday, when we did the Aerospike PM integration we looked at PMEM objects, and they are far slower than what we were able to achieve with libpmem and our own code.

Right now the C and C++ memory and object model does not support memory shared with other C and C++ programs, including those running on storage devices, NICs, GPUs or other processes. It also requires that all memory is equally accessible, and has the same characteristics (latency etc).

The compiler is required to optimise based on this model, which means that memory operation reordering takes no account of persistent write ordering requirements, that memory in a different 4Kb chunk could have much higher first access latencies, that memory being DMAed can be read from but must not be written to etc. All of this leads to a lot of inefficiency, programmer surprise, and makes heterogeneous compute unstable as toolchains evolve.

Where I'd like to get to is that there is no block vs object distinction in future POSIX, because the memory and object model has absorbed the need for such a distinction. What I'd like is that at the language level, there is simply memory with varying extents of latency nearness and latency farness, and latency far memory may asynchronously change over time. I appreciate this looks like good old segmented memory reinvented, but I don't mean near vs far pointers. We are probably actually looking at extended precision pointer types, so they'd be user definable.

As an example of micro-level gains, realloc() would no longer copy memory for large reallocations, but would simply remap the old memory to a new address. std::vector<> would no longer write zeros into already known zero bits memory. That sort of thing.

As an example of the macro-level gains, i/o would never touch the kernel, and would 99% of the time occur exclusively from user space via ordinary reading and writing of memory. read() and write() would be userspace emulations for backwards compatibility only.

That's where I'd like to get to. I don't know if it's possible, but I'll give it a try.

Niall