Added reducing memory consumption with leveldb or boltdb!

[ChrisLu]

The git head has the change adding leveldb and boltdb support! This reduces memory consumption even more.

To use it, "weed server -volume.index=[memory|leveldb|boltdb]", or "weed volume -index=[memory|leveldb|boltdb]". You can switch between the 3 modes any time, as often as possible. If the files for leveldb or boltdb is outdated or missing, they will be re-generated as needed.

boltdb is fairly slow to write, about 6 minutes for recreating index for 1553934 files. Boltdb loads 1,553,934 x 16 = 24,862,944bytes from disk, and generate the boltdb as large as 134,217,728 bytes in 6 minutes.
To compare, for leveldb, it recreates index in leveldb as large as 27,188,148 bytes in 8 seconds.

To test the memory consumption, the leveldb or boltdb index are created. There are 7 volumes in benchmark collection, each with about 1553K files. The server is restarted, hen I start the benchmark tool to read lots of files.
For leveldb, the memory starts at 142,884KB, and stays at 179,340KB.
For boltdb, the memory starts at 73,756KB, and stays at 144,564KB.
For in-memory, the memory starts at 368,152KB, and stays at 448,032KB.

To test the write speed, I use the benchmark tool with default parameters.
For boltdb, the write is about 4.1MB/s, 4.1K files/s
For leveldb, the writes is about 10.4MB/s, 10.4K files/s
For in-memory, it is a tiny bit faster, not statistically different. But I am using SSD, and os buffer cache also affect the numbers. So your results may be different.

[Tamás Gulácsi]

Just for the record: BoltDB is not a good idea on 32bit, as that limits the max db size to 256MB.

[ChrisLu]

Thanks for the note! Very good to know!

Chris

[Chris Lu]

I just found that Boltdb has a fix on this, https://github.com/boltdb/bolt/issues/280 , changing from 256MB to 2GB on 32bit.

Chris

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weed-file-syst...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gulácsi Tamás]

Apparently that fix was tested by me, but I've already forgotten... :)

Anyway, BoltDB has size limits, but I don't know how sever for seaweedfs:

In theory, how many files could be in a volume? So can it exceed that 2GB?

--
You received this message because you are subscribed to a topic in the Google Groups "Seaweed File System" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weed-file-system/T-eVFTzyCkk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weed-file-syst...@googlegroups.com.

[Chris Lu]

In Seaweed volume, the storage limit 32GB is a tighter bound. There is no limit in terms of number of files.

Chris

[Gulácsi Tamás]

I know that, but too lazy to do the math.

32GB of 8byte aligned storage is at most 32Gb/8b = (32<<30) >> 3 = 2<<32 = 4Gb of files. Now I remember: with CDB, this theoretic limit would've needed 2 2GB file.

OK, the same applies to BoltDB on 32bit: MAYBE you can reach the 2Gb limit, but now I just cannot find a viable scenario for it. That "store more than 2 billion, but less-than 8 byte files" scenario screams for something simpler: just store those "files" in a key-value store, say BoltDB.