Leveldb level more

[Fabien S.]

Hello all,

I would like to understand the usage of leveldb in the Warp10 standalone.

I created 315360000 points in the database (1 signal sampled ate 1 sample / second for 10 years) and it creates a lot of level 3 files (LOG file indicate it is level 3, all files are arround 2.1 Mb) but it never reach upper levels.

I got 2118 files of level 0 to 3.

In leveldb folder there is 4Gb data:

 ./warp10/leveldb$ ls -l | more
total 4331748
-rw-r--r-- 1 warp10 warp10 2111426 Jan 18 16:52 002708.sst
-rw-r--r-- 1 warp10 warp10 2112858 Jan 18 16:52 002710.sst
-rw-r--r-- 1 warp10 warp10 2110567 Jan 18 16:52 002714.sst
-rw-r--r-- 1 warp10 warp10 2112793 Jan 18 16:52 002718.sst
-rw-r--r-- 1 warp10 warp10 2111804 Jan 18 16:52 002720.sst
-rw-r--r-- 1 warp10 warp10 2111263 Jan 18 16:52 002724.sst
...
-rw-r--r-- 1 warp10 warp10 2111727 Jan 18 16:55 004641.sst
-rw-r--r-- 1 warp10 warp10 2112784 Jan 18 16:55 004643.sst
-rw-r--r-- 1 warp10 warp10 2110597 Jan 18 16:55 004647.sst
-rw-r--r-- 1 warp10 warp10 2112826 Jan 18 16:55 004649.sst
-rw-r--r-- 1 warp10 warp10 2111476 Jan 18 16:55 004654.sst
-rw-r--r-- 1 warp10 warp10 2111578 Jan 18 16:55 004656.sst
...
-rw-r--r-- 1 warp10 warp10 2111374 Jan 18 17:20 020859.sst
-rw-r--r-- 1 warp10 warp10 2031616 Jan 18 17:29 020865.log
-rw-r--r-- 1 warp10 warp10 2113084 Jan 18 17:20 020867.sst
-rw-r--r-- 1 warp10 warp10 1661397 Jan 18 17:20 020868.sst
-rw-r--r-- 1 warp10 warp10 2112236 Jan 18 17:20 020869.sst
-rw-r--r-- 1 warp10 warp10 2107283 Jan 18 17:20 020870.sst
-rw-r--r-- 1 warp10 warp10      16 Jan 18 16:35 CURRENT
-rw-r--r-- 1 warp10 warp10       0 Jan 18 16:35 LOCK
-rw-r--r-- 1 warp10 warp10 4240723 Jan 18 17:20 LOG
-rw-r--r-- 1 warp10 warp10      57 Jan 18 16:35 LOG.old
-rw-r--r-- 1 warp10 warp10 2031616 Jan 18 17:20 MANIFEST-000004

After a delete with the WebAPI, the files are not deleted, others are created (for tombstone I think).

Then when I FETCH data with WarpScript, it is very long.

It seams DELETE Warpscript or FETCH ask time to time a clean on the database.
It takes a lot of seconds.

When I run a manual compaction, it seams compaction is ok.

(in docker, manual compaction command is

/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -cp /opt/warp10/bin/warp10-2.7.2.jar -Dfile.encoding=UTF-8 io.warp10.standalone.WarpCompact /data/warp10/leveldb "" ""

)

Then FETCH command is 200ms.

Why Warp10 stop compaction at level3 ?
Is it possible to run a compaction manually ?
It is possible to raise the compaction level of leveldb with Warp10 ?

If I create a Function Extension of WarpScript to manually start a compaction, does it freeze the ingestion possibilities of Warp10 ?

What does it means "key range" parameters of the function compactRange(byte[] begin, byte[] end)?

Best regards,

Fabien.

[mathias....@gmail.com]

Hi,

Warp 10 leverages LevelDB which manages the .sst files. I highly recommend you read https://blog.senx.io/demystifying-leveldb/ which explains how LevelDB works.

Warp 10 does not stop compaction at level3, the LevelDB library only creates files at a deeper level when the number of files at a given level is more than 10 times that of the lower level, so as more files will be created, your number of levels will grow. Here is for example the dispatching of files in the various levels for one of our instances with over 840,000 files:

[ 1 11 85 903 8389 83509 751592 ]

You can run compactions manually as you did using WarpCompact, be aware though that compacting hundreds of thousands of files can take eons.

As stated above, the number of levels is dependent on the number of files.

You could create a function which performs a LevelDB compaction, but you would have to execute it with a closed backend, see WarpDB#doOffline.

The key range in the compactRange function determines the start and end keys of the LevelDB records to compact.

Regarding the deletions, if you intend to do frequent deletions, SenX offers a commercial extension to drop SST files instantly to help speed up the deletion process. Do not hesitate to reach out to sa...@senx.io to learn more.

Mathias.

[Fabien S.]

Ok, I understand, Thanks a lot.

At a compaction of one level, it creates multiple files of xxx Mo.

In my case, files max size are 2.1 MB. I think it is the same as the blog article demystifying-leveldb

In the example you gave, we have 903 files in level 3.

So depending the threshold (says 40 MB for level 1 ), we have  a max of 4 * 10^LEVEL / 2.1 Files .

In the example, there is 844490 files, so the database size is less than 1.8 TB !

It is clear, Thank you for your answer.

Best regards,

Fabien.