How to optimize my write operations of levelDB in my case

35 views
Skip to first unread message

程洋

unread,
Sep 14, 2021, 3:18:23 AM9/14/21
to leveldb
I tries to use `kythe` tools developed by google on AOSP project. it turns out there are lots of data to write into levelDB. And i encounter really big problem on performance.

Here is my use case
1. I need to write database every week.
2. each time i have 3 TB data to write, averagely 500 bytes per key-value entry
3. Unfortunately, maybe 80% of the data are duplicated.
4. after all 60TB data is written, there is only read operations after. until the next 60 TB write in the future.
5. i have machine with 512GB memory and 64 cores.
6. my disk reading benchmark is ~150MB/s,  so reading all 3TB data takes about 6 hours.  however it takes about 20 hours to finish all write. so bottle neck is on levelDB

Currently i adjust WriteBufferSize to 2GB. Though it still need 24 hours to write.
Is there any chance i can optimize write performance in my case?
Reply all
Reply to author
Forward
0 new messages