I tries to use `kythe` tools developed by google on AOSP project. it turns out there are lots of data to write into levelDB. And i encounter really big problem on performance.
Here is my use case
1. I need to write database every week.
2. each time i have 3 TB data to write, averagely 500 bytes per key-value entry
3. Unfortunately, maybe 80% of the data are duplicated.
4. after all 60TB data is written, there is only read operations after. until the next 60 TB write in the future.
5. i have machine with 512GB memory and 64 cores.
6. my disk reading benchmark is ~150MB/s, so reading all 3TB data takes about 6 hours. however it takes about 20 hours to finish all write. so bottle neck is on levelDB
Currently i adjust WriteBufferSize to 2GB. Though it still need 24 hours to write.
Is there any chance i can optimize write performance in my case?