MatrixOne: a hyperconverged and one-size-fits-most database in Go

[Nan DENG]

Hi,

I’d like to introduce a hyperconverged and one-size-fits-most database named MatrixOne, written in Go. 

Github: https://github.com/matrixorigin/matrixone

It’s a redesigned database with a combined architecture of CockroachDB, Clickhouse and Apache Flink. Our motivation is to bring a simple and one-size-fits-most data processing framework and significantly reduce the complexity of building modern data platform.

Key Objectives:

• Simple usage by standardized SQL with support of multiple dialects.
• Supports transactional, analytical, streaming workloads in one database. 
• Blazing fast query performance with strong consistency and ACID transactional property.
• Large scalability across different infrastructures, including public/private cloud, on-premise and edge devices. 

It’s still a very early stage project, the latest 0.3.0 release version is mainly a MPP OLAP engine, with distributed strong consistency based on Raft.

A SSB benchmark test is run over MatrixOne and it outperforms Clickhouse at 50%, with some constraints on. 

Feel free to test MatrixOne and any feedback is welcome. 

[Zhaoxun Yan]

Does it support sequence and upsert in PostgreSQL?

[Brian Candler]

On Friday, 15 April 2022 at 17:47:48 UTC+1 nicolas...@gmail.com wrote:

any feedback is welcome. 

The only feedback I can give is from your reading your documentation, which appears at this point to be very vague.

I was looking for a proper architectural white paper that says exactly what CAP and transaction serializability tradeoffs you're making. The nearest I could find was this one-page "tech design":

https://docs.matrixorigin.io/0.3.0/MatrixOne/Overview/MatrixOne-Tech-Design/matrixone-techdesign/

It's full of general statements like "consistency protocols are proposed to ensure the consistency of replicate logs. Common consistency protocols include Paxos and Raft" - rather than saying exactly what MatrixOne has chosen to do.

There is a link that says "Please refer to AOE Technical Design for more details", but that link gives a 404 error.

Searching for the word "isolation", it appears in the glossary only, and nowhere else.

In comparison, CockroachDB and FoundationDB have a ton of detailed documentation saying *exactly* how they handle things like overlapping transactions, consistency, node failures, shard splitting etc; and these change over time as they revisit the design decisions and trade-offs they made.  I'm not saying you are positioning yourself against those two particular databases. I just give them as examples where the level of documentation is good enough to evaluate the database design for a particular application.

Otherwise, I think it's a dangerous claim to make that MatrixOne suits almost all database use cases.  For example, many use cases value data consistency and durability far above performance.  If you're saying MatrixOne can match the consistency and durability of the best SQL databases, *and* give higher performance, then that is a bold claim to make without detailed justification.  What level of stress and regression testing are you doing? Are you running any Jepsen-like scenarios? The happy path is easy to get right - dealing with all possible failure modes is hard.

[Yingfeng Zhang]

Currently we are working on the update features, and upsert has not been considered initially and will be done in future.

[Yingfeng Zhang]

Hi, Brian,

Thanks for your comments.

Actually the MatrixOne is still in its infant stage, so it's far from being able to be compared with other alternatives in terms of ACID,...,etc. Current codebase is just a demo such that MPP sql execution engine could work together with both columnar storage as well as multi-raft architecture, while the columnar storage engine adopts a name with AOE which means Append Only Engine, that's why we call it a demo stage.

Currently we are working on transactional support to deliver a TAE engine(Transactional Analytical Engine),  the initial version would be visible during June this year which is still a single node version, and distributed TAE would be initially ready during Q3 of this year. If you are interested in the draft design of TAE, you may take a look here .

After these works have been ready, could we start with such works as Jepsen-like testing.

Thank you ~

[Nan DENG]

Hi Brian,

Thanks for your feedback. Indeed we are aware of the doc problem. Many technical design docs are still under edition. As for the AOE technical design, sorry for the wrong link in the doc website, you may refer to this good RFC link.

For the bold claim problem, as Yingfeng specified, MatrixOne is still at a very early stage, but our objectives of suiting most use cases were carefully thought over, we are also moving towards that objective. But I agree with you that before convincing proof and testing comes out, we'd improve the claim about what are final objectives and what are already implemented and proven. 

As we believe in open source, we'd like to share ideas and works as early as possible with community. We appreciate for your thoughtful and constructive feedback.

Thanks,