levigo vs sqlite wrappers performance comparison ?

[Sankar]

Hi,

I am trying to write a program which will need an in-process db and will deal with a few million records. If leveldb go reimplementation was ready, I would have chosen it blindly as my server is going to be in Go. I saw that levigo https://github.com/jmhodges/levigo is a Go wrapper for leveldb. I have seen numerous sqlite wrappers too. But I am not sure what will be the fastest solution out there. I do not mind doing SQL or storing gob strings in leveldb and doing the comparisons myself. Has anyone done a performance comparison of levigo and the various sqlite go wrappers ?

Thanks.

[Caleb Spare]

It would help if you were more specific about your needs. LevelDB is
quite a different (and lower-level) abstraction from an SQL db like
sqlite. It's not clear that you can do an apples-to-apples comparison
between the two. Additionally, LevelDB is an LSM DB whereas sqlite
uses a typical b-tree architecture (I think), so I expect that a
particular type of workload (reads vs writes) will favor one over the
other.

I would advise you not to worry too much about performance benchmarks
before you have a clearer picture of exactly what your needs are (and
maybe not until you know your DB is the bottleneck).

One of my favorite projects in this space is Bolt:

https://github.com/boltdb/bolt

It has the advantage of being pure-Go, unlike the other two projects
you listed. It also has a small, simple API that may fit your use case
or may be too limited.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Sankar]

On Friday, 15 August 2014 12:39:06 UTC+5:30, Caleb Spare wrote:

It would help if you were more specific about your needs. LevelDB is
quite a different (and lower-level) abstraction from an SQL db like
sqlite. It's not clear that you can do an apples-to-apples comparison
between the two. Additionally, LevelDB is an LSM DB whereas sqlite
uses a typical b-tree architecture (I think), so I expect that a
particular type of workload (reads vs writes) will favor one over the
other.

My usecase will be very much read centric with occasional writes. May be I would say 80% of the time will be spent in reads and 20% in writes.

 

I would advise you not to worry too much about performance benchmarks
before you have a clearer picture of exactly what your needs are (and
maybe not until you know your DB is the bottleneck).

Sure. I need to measure the performance before I assume :)

But my server implementation will be a little different if I were to store in a relational database or if I am going to store in a key-value pair db. So I wanted to see the performance numbers of any of these nosql datastores (using Go wrapper) before I go down this path. If the performance numbers are not too compelling, I will happily write code for the relational model (basically avoiding a few compress multiple-columns into a single value code) .

 

One of my favorite projects in this space is Bolt:

https://github.com/boltdb/bolt

It has the advantage of being pure-Go, unlike the other two projects
you listed. It also has a small, simple API that may fit your use case
or may be too limited.

Actually, this seem to be very nice based on a first glance. I have heard highly of the lmdb after which this is modelled. Are you happy with the performance of this database in general ? The pure-Go part is really appealing instead of the wrappers elsewhere.

Thanks.

Sankar

[Tamás Gulácsi]

And of course there is https://github.com/cznic/ql .

[Jakob Borg]

2014-08-15 8:49 GMT+02:00 Sankar <sankar.c...@gmail.com>:
> I saw that
> levigo https://github.com/jmhodges/levigo is a Go wrapper for leveldb. I
> have seen numerous sqlite wrappers too.

There's also https://github.com/syndtr/goleveldb, a native leveldb
implementation.

//jb

[Dobrosław Żybort]

Only key/value but maybe you will find it useful:
https://github.com/jimrobinson/kvbench

[Siddon Tang]

Well, LedisDB uses different database as the backend storage: leveldb, rocksdb, goleveldb, hyperleveldb, boltdb, lmdb, but no sqlite, lmdb and boltdb use b+tree and others use LSM.

I think the benchmark result can help you. 

Benchmark is here : https://github.com/siddontang/ledisdb/wiki/Benchmark or you can see the diagram here http://ledisdb.com/

1. lmdb is the fastest for read and write if you use some dangerous async config, even you use sync config, its write performance is well too.
2. boltdb is very good for read, especially for range read, but it's not good at write, you can see why here https://github.com/boltdb/bolt/issues/237
3. others are good at write but range read is not good.
   

in your scene, 80% is read and 20% is write, so I recommend boltdb or lmdb. boltdb is pure go and you can use it directly. lmdb may not be used in windows because I meet some compile error.

if you want to use others, I recommend goleveldb which is pure go too. Using leveldb, or rocksdb, or hypherleveldb is not easy, you must first compile them, then set GO_CFLAGS and LD_CFLGAS to build associated go package. 

Btw: config rocksdb is very painful, even the rocksdb developer can not know them all https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#final-thoughts

在 2014年8月15日星期五UTC+8下午2时49分13秒，Sankar写道：