Memory leak after manual compactions

[Rahul Arora]

RocksDB 8.5.4 (RocksDBJni)

Hi, I'm using RocksDB for a read-heavy workload. After several tests, I found out that the reads work best if the DB is in the best state (compacted). So, I'm running manual compaction by running db.compactRange(CF) every 60 mins.

I also keep monitoring my system's memory usage by "sar -r 1" command. And, it clearly shows an increase in memory after every compaction which is never reclaimed.

Is this a known issue? or am I doing something wrong here?

Please let me know if more details are required.

[MARK CALLAGHAN]

I am not aware of a serious memory leak. What allocator do you use -- jemalloc and tcmalloc are far better at reducing RSS for RocksDB processes versus glibc malloc.
https://smalldatum.blogspot.com/2023/08/rocksdb-and-glibc-malloc-dont-play-nice.html

--
You received this message because you are subscribed to the Google Groups "rocksdb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/d5b8978c-1092-4d4b-bb5a-bc901bc4819fn%40googlegroups.com.

--

Mark Callaghan
mdca...@gmail.com

[Rahul Arora]

I've been using the default one. The blogs gave some good insight on the memory allocators for Rocks. Will try using jemalloc / tcmalloc.

Thanks.

[Rahul Arora]

So, although running compactions manually does increase the memory by a small % I also tried turning the manual ones off completely and found out that the memory was still increasing at an almost constant rate. 

In my app, I'm only doing read-modify-write type of operations. Earlier, I was not closing the WriteBatch object which was leading to a memory leak every time I was writing something. Not sure what is it this time.

During the reads, I only use the ColumnFamilyHandle object (which inherited RocksObject that can caused the problem during writes earlier) which is only defined once and is reused multiple times. One important thing to note here is that I'm using multi-get operations. So, with all the keys, I also have to pass on their respective ColumnFamilyHandles to the method call. In my case, it's the same handle so before the call, I just make 'n' copies of the ColumnFamilyHandle object.

I'm guessing that it should not be the problem. 

I;m not using iterators. My filter and index blocks are cached in the block cache. My memtable size is 100 MB * 6. 

Any suggestions please?

[Rahul Arora]

After further analysis, I found nothing wrong while checking the properties for the various things like memtables, block cache, etc

I don't see unnecessary (deleted) files also using the 'lsof' command.

[Adam Retter]

Have you made sure that you are calling close on any object that inherits from RocksObject (once you are finished with it)?

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/c5aaacd2-c0f4-450a-b0a5-43db9c7e5060n%40googlegroups.com.

[Rahul Arora]

Hi, yes, I'm closing all the objects that inherit RocksObject. From what I've figured out, the issue is in the read path somewhere. Writing doesn't take extra memory apart from the memtables.

[Rahul Arora]

I'll probably try using some other memory allocator once like Mark mentioned. Can someone suggest how can I do that though for JNI?

[Ricardo Fonseca]

Hello,

This might be a long shot, but I think there could be a memory leak if RocksDB::multiGet is called without passing a ReadOptions object as argument, since on the C++ side of rocksdbJNI there is an allocation of a ReadOptions object without a subsequent memory free (see here).

A quick test to see if it resolves your memory leak, is to allocate a single ReadOptions object and pass it as argument for RocksDB::multiGet each time.

Cheers,

Ricardo

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/d515e8e5-0ed8-4fbc-95d9-a2fc2f0ed362n%40googlegroups.com.

[Adam Retter]

It looks to me like ReadOptions is allocated on the Stack in
RocksDB::multiGet, and so it should be cleaned up when the scope
exits.

> To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/CAFpd3Woxh1-FR-RiitatvDzOG3PmCFjZt-LVBzY66E5_dZ6NtA%40mail.gmail.com.



--
Adam Retter

skype: adam.retter
tweet: adamretter
http://www.adamretter.org.uk

[Ricardo Fonseca]

Ah yes, you are right. I misread the & as a *.

In that case there is no leak caused by ReadOptions.

[Rahul Arora]

I've ensured all RocksObjects are closed in the code after use. I'm adding a trimmed-down code of my application that mimicks how I interact with RocksDB. 

Hopefully, this will give a better insight.

[Rahul Arora]

Found out that this PR (https://github.com/facebook/rocksdb/pull/11221) addresses the memory leaks after version 7.

Can someone please tell me if it Is gonna be pushed to Maven? else I can fork it and create my own build

[Adam Retter]

That PR is from my colleague Alan, as far as I can see it doesn't fix any memory leaks 

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/708744ce-c280-4722-b7e0-2fc207de76e9n%40googlegroups.com.

[Alan Paxton]

What Adam says, with a very minor exception. The PR https://github.com/facebook/rocksdb/pull/11221 is designed to make it more difficult to add leaks in future code.

--Alan

[Rahul Arora]

Oh got it. There does exist a memory leak in the versions >= 7 though. I've not been able to figure out why though. I tried using the latest 6.x version of Rocksdbjni but the performance was at least 20x worse than the latest rocksdbjni (8.5.4) version.

Any suggestions please?

[Rahul Arora]

Mentioning again that it's happening in the read flow only. And I don't see any RocksObject being used in the read flow except ReadOptions which, in my code, is defined only once.

Attaching my trimmed-down code here. Can someone please take a look at it and suggest something?

[Adam Retter]

You would need to use appropriate profiling tools to identify where the memory leak is

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/7c6ebe66-16fa-4f12-a90b-5c3abe2e9081n%40googlegroups.com.