pmem + Boost Memory-Mapped Files

[Jiri Jetmar]

Hi guys, 

I looked at pmem and it might be interesting for our use case. 

A database called Chainbase (https://github.com/eosio/chainbase/tree/4724baf2095cdc1bb1722254874b51070adf0e74)

is a combination of Boost Memory-Mapped Files and some Boost Multi-index Container 'acrobatics'.

The database is used within the https://eos.io/ blockchain. 

Currently, Chainbase uses RAM for transactions and the filesystem for persistence. As the amount of 

data grows, so do the RAM requirements. The idea is now to use e.g. Intel Optane to persist transactions. 

It is simply a matter of cost to store 1 TB in RAM or in a PCIe NVMe device(s). 

For this reason, I would like to ask if I can give anywhere an example of Boost Memory-Mapped Files implementation

where pmem.io is used. 

Any other feedback or advice is welcome as well!

Best Regards, 

jj

[Niall Douglas]

I am unaware of any persistent-memory native database which is mature, and I say that as the lead for WG21 P1031 Low level file i/o which does have some support for pmem. 

Persistent memory storage is very different to kernel paged storage. Your kernel paged storage will usually run a good bit faster on persistent memory if the end user has booted their kernel and formatted their storage right. But a database which performs as well as persistent memory could enable is a very different proposition.

Right now, pmdk relies heavily on UB in C and C++ to work, and uses a lot of inefficient algorithms forced on it by backwards compatibility and OS kernel quirks. WG21 P1026 A call for an Elsewhere Memory study group proposes a SG to work on refactoring the C++ and C memory models to better support memory elsewhere such as persistent memory. But I really must emphasise this work is a decade out from standardisation, without commercial sponsorship to prototype a compiler with the proposed memory model, the work happens in people's free time, and it will take the time it takes.

So for sure, you could build a much better Chainbase which requires persistent memory and which wouldn't work well on kernel paged storage. But the hardware available to run it will be very limited, and you'll be locked into probably a single compiler on a single architecture (Intel) and on a single OS kernel. If you're keen on building such a thing, I'm very interesting in hearing more. Everybody here would love to see how it would go if we're not the ones doing it! :)

In terms of more practical approaches, it looks like Chainbase is a simple append-only database. They're quite easy to build, I rolled one myself with the LLFIO reference library myself without much effort. But they're expensive on bandwidth, and don't scale too well with growth as GC and fragmentation take hold. I have a B*-tree based design in mind which makes much better use of pmem if it's available via a large page granularity on pmem and 4Kb when without, but it's merely scribbled notes accumulated over time. As with all these things, this stuff gets prototyped only when I'm between contracts, which is not often. Otherwise my free time is filled by family and standards committee stuff.

Piotr may jump in here and say that the pmdk already implements this stuff. If so, you should use that instead. Intel do the heavy lifting wrt compatibility and platform specific hacks.

Niall

[Piotr Balcer]

Hi,

libpmem implements more or less what "Boost Memory-Mapped Files" provides, but with added persistence APIs.

You can't easily plug PMDK into a mmap-based application and expect things to be automagically persistent ;)

In your case, an append-only database could be easily implemented based on the primitives provided by

libpmem. Using cacheline aligned non-temporal memcpy into a DAX-based mapping will *significantly*

outperform writing to DRAM and then synchronously flushing.

Sadly, we don't have a library that solves *exactly* this use case ;) libpmemlog is close though.

https://github.com/pmem/pmdk/blob/master/src/libpmemlog/libpmemlog.c

Piotr

[Niall Douglas]

In your case, an append-only database could be easily implemented based on the primitives provided by

libpmem. Using cacheline aligned non-temporal memcpy into a DAX-based mapping will *significantly*

outperform writing to DRAM and then synchronously flushing.

That excellently summarises the kinds of changes we want to make to the C and C++ memory and object model!

Right now it's safest if you build out or serialise your object representation, and do a non-temporal memory copy.

However in the future, we'd like the ability to give the CPU and compiler the freedom to reorder and combine object modification, but annotated with constraints to ensure sudden power loss persistence.

This is very similar to, but orthogonal to, std::atomic<T> which constrains apparent reordering wrt other CPU cores. We need a new annotation which can specify constraints for reordering of reads and writes to memory elsewhere.

Also, do bear in mind that non-temporal writes may currently be sufficient for pmem persistence on current Intel CPUs, but other CPUs and maybe future Intel CPUs may not make that guarantee.

Niall

[Jiri Jetmar]

Hello guys,

I would like to thank you for the extensive and valuable input. I didn't understand everything down to the last detail and need to understand pmem and its structure & function better. 

On the Chainbase topic - it is not just an append-only database, but represents the (last) state of a transaction and able to manage a number of versions (multiversion) of the states. 

Assume you have blocks (just containers) with the following transactions on the accounts B, C, D.

#B1: Trx(A -> B, 5 XYZ)

#B2: Trx(B -> C, 2 XYZ)

#B3: Trx(C -> D, 1 XYZ)

The final in-memory state on Chainbase for the above accounts would be, B=3, C=1, D=1. 

The above principle is even more generic. Accounts are not just structures for managing balances, but so-called "Smart Contract" can be deployed that uses Boost Multi-index to manage states and therefore arbitrary structures are possible. 

typedef eosio::multi_index<N(accounts), account>accounts; // <-- this here gets stored to a chainbase. 

accounts from_acnts( _self, owner );

from_acnts.modify( from, owner, [&]( auto& a ) { // <-- is is compiled to WASM and interpreted as byte-code

          a.balance -= value;

});

To confuse you even more, the above Boost example is compiled down to Webassembly and interpreted as WASM byte-code by a WASM engine, that is embedded within a C++ binary executable and where the Chainbase is used to store the states in RAM. 

Let me please sketch a picture, how the above described parts are working together. Maybe we can do something together. 

Back on the initial use case, RAM on the EOS Blockchain is expensive (1000 USD per 1,5 MB), because it is a commodity of speculation and it is known, that the offer is limited since more than 3-6 TB cannot be installed in a server. 

Best Regards, 

Jiri

[Niall Douglas]

On the Chainbase topic - it is not just an append-only database, but represents the (last) state of a transaction and able to manage a number of versions (multiversion) of the states. 

Assume you have blocks (just containers) with the following transactions on the accounts B, C, D.

#B1: Trx(A -> B, 5 XYZ)

#B2: Trx(B -> C, 2 XYZ)

#B3: Trx(C -> D, 1 XYZ)

The final in-memory state on Chainbase for the above accounts would be, B=3, C=1, D=1. 

I think we all understood that. In append only we were referring to your transaction implementation mechanism. The set of root notes of the directed acyclic graph you generate from your transactions reflects the version in time of account balances. It's similar to git, in that the contents of versions of files specified by HEAD would be similar to account balances.

 

The above principle is even more generic. Accounts are not just structures for managing balances, but so-called "Smart Contract" can be deployed that uses Boost Multi-index to manage states and therefore arbitrary structures are possible. 

typedef eosio::multi_index<N(accounts), account>accounts; // <-- this here gets stored to a chainbase. 

accounts from_acnts( _self, owner );

from_acnts.modify( from, owner, [&]( auto& a ) { // <-- is is compiled to WASM and interpreted as byte-code

          a.balance -= value;

});

To confuse you even more, the above Boost example is compiled down to Webassembly and interpreted as WASM byte-code by a WASM engine, that is embedded within a C++ binary executable and where the Chainbase is used to store the states in RAM. 

Piotr may be able to comment more, but I'd say at this stage that persistent memory is now out of scope for you. Persistent memory requires very specific configuration and use (e.g. alignment) to perform well, and just getting WebAssembly to issue clflushopt efficiently would be hard. I suppose you could expose native pmdk into the WASM engine, but at some point you've got to ask if you're solving the right problem here.

In my opinion, if you used a less bandwidth wasteful storage algorithm, you'd get far better performance than throwing expensive persistent memory at the problem. Most simple databases such as SQLite use an append-only delta log like you're currently exclusively using. From time to time, that write ahead log is expensively merged into a B*-tree store which is the main database. The databases which don't care so much about data integrity tend to use Log Structured Merge (LSM) instead of a B*-tree. There are lots of strategies, it's what makes storage so fun!

I would finally mention that filing systems implement a lot of this stuff already for you. Rather than reinventing the wheel, you might just consider being dumb with ext4 and see how bad things are. You might well be surprised.

Niall

[Piotr Balcer]

That's actually a really interesting use case. Your WASM byte-code is a mini transaction for the database, and you'd ideally want to do execute that directly on persistent memory which can potentially have huge capacities. Is my understanding correct?

I agree with Niall that this might be tricky from the point of view of the WASM engine. But... if you just use NVDIMMs for their capacity, and don't care about persistence, you could just use your existing application with very few modifications (I'm assuming you can use your own memory-mapped region for the WASM byte code, which is not normally possible if my quick look at WebAssembly spec is correct).

If you'd want to use persistence and run your small WASM "transactions" fail-safe atomically, then that's an entirely different story, and you'd probably need to change your code quite a lot.

I missed something in your initial email - the entire PMDK/pmem.io project is focused on NVDIMM hardware (on Intel platforms, this: https://newsroom.intel.com/editorials/re-architecting-data-center-memory-storage-hierarchy/). PMDK is not optimized to be used on regular NVMe drivers.

Piotr

[Jiri Jetmar]

Hi guys, 

thanks again for the valuable input. 

Ok, I got the context to implement the storage mechanism. I do find only few informations on how the Boost Multiindex indexing algorithm really works, but a DAG makes sense. 

My first idea was to use the B-Tree based LMDB (http://www.lmdb.tech/doc/). I'm gonna do some benchmarks

and see if that's a possible way. 

"That's actually a really interesting use case. Your WASM byte-code is a mini transaction for the database, and you'd ideally want to do execute that directly on persistent memory which can potentially have huge capacities. Is my understanding correct?"

Yes. EOSIO is supporting a number of different WASM implementations, e.g. the wabt (https://github.com/WebAssembly/wabt). WASM memory is a contiguous byte-addressable linear memory spanning from offset 0 and extending up to a varying memory size. Whenever a byte is written or fetched from memory by using the corresponding WASM opcode, Chainbase is used. So every memory operation is a mini transaction to the RAM based Chainbase, together with a number of metadata, like the account who is paying for the used RAM, etc. I estimate (but would have to measure it exactly) that about 40k-100k byte level operations (mini transactions) are processed per second. This corresponds in about 4K business transactions per second. 

"I missed something in your initial email - the entire PMDK/pmem.io project is focused on NVDIMM hardware"

I think I read somewhere that Optane devices are also supported. Probably that was just marketing. The idea is to have a scaling strategy. At the moment there is no and a hard limit for available RAM. And it's also a costly thing. Not only one node must have the appropriate RAM, but at least 21 nodes, as well as failover and backup.

If we get an increase in the storage density by e.g. factor 10x and save some money, then the case is interesting. 

Also, a further adoption of the EOS Blockchain is not always easy to promote. If you sell 1.5 MB for 1000 USD, people will either think you are brilliant or totally crazy. An already purchased byte is a "lifelong-byte", yet we need a scaling strategy. 

Thank you. 

Regards, 

Jiri

Am Montag, 8. Oktober 2018 22:25:45 UTC+2 schrieb Jiri Jetmar: