Storage used to store hdfs metadata

[158010...@gmail.com]

Why use NDB-Cluster to store the metadata ?
Is other storage considered or better than NDB such as KV store ?

[Jim Dowling]

(same answer as on gitter)

The concurrency model in hopsfs requires that the DB backend supports high performance cross-partition transactions (by partition, i mean shards). No serialization of cross-partition transactions - it would kill performance. It also requires row-level locks, because we implement our concurrency model using row-level locks. The beauty of open-source is that if TiKV is good enough, somebody will implement the driver in it for HopsFS. 

(My initial analysis of TiKV (from reading front page) is that it has most things - but its using paxos for each shard. Are cross-shard transactions supported - it says it has "Consistent distributed transactions", which could be that? Are they serialized or high performance though? Is there row-level locking? What is the transaction isolation level supported for cross-partition transactions?

This is not promising for TiKV:

https://www.quora.com/How-does-TiDB-compare-with-MySQL