rocksdbjni & jemalloc

Henning Lohse

unread,

Mar 2, 2024, 1:30:16 AM3/2/24

to rocksdb

Hi everyone,

We are using RocksDB in our Java application via rocksdbjni. We saw improvements in memory usage during high update rate scenarios when using jemalloc via LD_PRELOAD.

I have two questions here:

Would we expect more memory usage improvements if, instead of using jemalloc via LD_PRELOAD, we had a build of RocksDB with some jemalloc flags set during compilation, and jemalloc statically linked in?
There seems to be a specialized allocator in the code base, and there are some options in the build files regarding jemalloc, but I did not find documentation for them regarding their impact exactly.

Also, are there any resources regarding how to build your own rocksdbjni fat jar?
I am wondering if it would be possible to build it with jemalloc statically linked in and enabled.

Best regards,

Henning

Jon Groff

unread,

Mar 2, 2024, 4:58:37 PM3/2/24

to rocksdb

I am betting the resource is: grab your favorite IDE and build environment and create a JNI project.

Radek Hübner

unread,

Mar 6, 2024, 4:50:21 AM3/6/24

to rocksdb

Hello Henning,

I don't know if compiling directly with jemalloc will improve performance, but some people mentioned previously they are getting better performance with jemalloc and considering there is support in RocksDB for jemalloc, you may get better results directly compiling with jemalloc.

How to compile a fat jar. For Linux you have two options:
1) make command. "make rocksdbjava JEMALLOC=1", see section compiling from source on RocksDB wiki : https://github.com/facebook/rocksdb/wiki/RocksJava-Basics#compiling-from-source

2) CMake - Primary used for compilation on Windows, but you can use it also on Linux. There is direct support for jemalloc with CMake parameter -DWITH_JEMALLOC=1 Example of full command : cmake -B build -S . -DJNI=1 -DWITH_JEMALLOC=1

More documentation how to compile RocksDB you can find in INSTALL.md : https://github.com/facebook/rocksdb/blob/main/INSTALL.md

Radek

--
You received this message because you are subscribed to the Google Groups "rocksdb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/607e85eb-65de-4e39-951b-0b1befe43cc0n%40googlegroups.com.

Henning Lohse

unread,

Mar 6, 2024, 5:32:30 AM3/6/24

to rocksdb

Hi Radek,

thanks for the insights and the documentation links.

Your answer confirms that it would be quite the work to e.g. make our CI build a fat jar like the "official" one, but with jemalloc enabled and statically linked in, including all platforms. (macOS, Linux 32/64 bits + musl, Windows).

Do you know if there are any plans by the RocksDB team to provide a fat jar with jemalloc enabled and statically linked in already? If not, I suppose we will recommend using jemalloc via LD_PRELOAD for the time being.

Best regards,

Henning

Mark Callaghan

unread,

Mar 6, 2024, 10:59:56 AM3/6/24

to rocksdb

I can't think of perf improvements from compiling so that RocksDB knows it is using jemalloc. But doing that provide at least 2 features:
1) enables monitoring specific to jemalloc, see db/malloc_stats.cc, I have used this for debugging perf issues
2) enables the no-dump allocator, which I have never used

$ find . -type f -print | xargs grep ROCKSDB_JEMALLOC
./thirdparty.inc: set(JEMALLOC_CXX_FLAGS "-DROCKSDB_JEMALLOC -DJEMALLOC_EXPORT= -DJEMALLOC_NO_RENAME")
./CMakeLists.txt: add_definitions(-DROCKSDB_JEMALLOC -DJEMALLOC_NO_DEMANGLE)
./Makefile: PLATFORM_CXXFLAGS += -DROCKSDB_JEMALLOC -DJEMALLOC_NO_DEMANGLE
./Makefile: PLATFORM_CCFLAGS += -DROCKSDB_JEMALLOC -DJEMALLOC_NO_DEMANGLE
./db/malloc_stats.cc:#ifdef ROCKSDB_JEMALLOC
./db/malloc_stats.cc:#endif // ROCKSDB_JEMALLOC
./examples/Makefile: PLATFORM_CXXFLAGS += -DROCKSDB_JEMALLOC -DJEMALLOC_NO_DEMANGLE
./utilities/memory_allocators.h:// JeMallocMemoryAllocator can define these methods if ROCKSDB_JEMALLOC is
./port/win/port_win.h:#ifdef ROCKSDB_JEMALLOC
./port/win/port_win.h:#ifdef ROCKSDB_JEMALLOC
./port/win/port_win.h:#ifdef ROCKSDB_JEMALLOC
./port/win/win_jemalloc.cc:#ifndef ROCKSDB_JEMALLOC
./port/jemalloc_helper.h:#ifdef ROCKSDB_JEMALLOC
./port/jemalloc_helper.h:// MSVC does not have weak symbol support. As long as ROCKSDB_JEMALLOC is
./port/jemalloc_helper.h:#endif // ROCKSDB_JEMALLOC
./util/compression.h:#if defined(ROCKSDB_JEMALLOC) && defined(OS_WIN) && \
./util/compression.h:#endif // defined(ROCKSDB_JEMALLOC) && defined(OS_WIN) &&
./util/bloom_test.cc:#ifdef ROCKSDB_JEMALLOC
./util/bloom_test.cc:#endif // ROCKSDB_JEMALLOC
./memory/jemalloc_nodump_allocator.h:#if defined(ROCKSDB_JEMALLOC) && defined(ROCKSDB_PLATFORM_POSIX)
./memory/jemalloc_nodump_allocator.h:#define ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.h:#endif // ROCKSDB_JEMALLOC && ROCKSDB_PLATFORM_POSIX
./memory/jemalloc_nodump_allocator.h:#ifdef ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.h:#endif // ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.h:#ifdef ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.h:#endif // ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.h:#ifdef ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.h:#endif // ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.h:#ifdef ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.h:#endif // ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/memory_allocator_test.cc:#ifdef ROCKSDB_JEMALLOC
./memory/memory_allocator_test.cc:#endif // ROCKSDB_JEMALLOC
./memory/jemalloc_nodump_allocator.cc:#ifdef ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#endif // ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#ifndef ROCKSDB_JEMALLOC
./memory/jemalloc_nodump_allocator.cc: *why = "Not compiled with ROCKSDB_JEMALLOC";
./memory/jemalloc_nodump_allocator.cc:#ifndef ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#endif // ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#ifdef ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#else // ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#endif // ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#ifdef ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#endif // ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#ifdef ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#endif // ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#ifdef ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#endif // ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
./memory/jemalloc_nodump_allocator.cc:#ifndef ROCKSDB_JEMALLOC

On Friday, March 1, 2024 at 10:30:16 PM UTC-8 hennin...@hivemq.com wrote:

Radek Hübner

unread,

Mar 12, 2024, 10:56:38 AM3/12/24

to rocksdb

Hello Henning,

Unfortunately there is no plan to release a fat jar with jemalloc in near future. Our goal is to maximize compatibility and there are still sometimes problems with jemalloc in some specific cases and on some platforms is near impossible to compile and release.

At the moment I can only suggest that you compile "lite jar" only for your production systems. Another advantage which you may get is better optimisation for your CPU architecture. Our build doesn't use the latest vector instruction on modern CPUs. For example if you are using compression, you should get some performance boost with AVX/AVX2 instructions with your own builds.

Best regards,

Radek

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/d142dd4a-4109-47a7-8425-ef6452ceb696n%40googlegroups.com.

Henning Lohse

unread,

Mar 13, 2024, 12:44:33 AM3/13/24

to rocksdb

Hi Radek,

thanks for the insights.

I will forward your hint regarding an architecture-specific build, we might have use for that.
Would you be able to tell me more about cases and/or problem where jemalloc led to problems?

Best regards,

Henning

Adam Retter

unread,

Mar 13, 2024, 4:13:13 AM3/13/24

to Henning Lohse, rocksdb

Hi Henning,

For enabling all architecture specific optimisations, you should use
the flag PORTABLE=0.

Thanks Adam.

On Wed, 13 Mar 2024 at 05:44, 'Henning Lohse' via rocksdb

> To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/4a930081-f537-488b-9d77-2125ae123f9en%40googlegroups.com.

--
Adam Retter

skype: adam.retter
tweet: adamretter
http://www.adamretter.org.uk

Reply all

Reply to author

Forward