Minimum memory requirement for Persistit

33 views
Skip to first unread message

simon.b...@sonarsource.com

unread,
Jun 11, 2013, 5:23:04 AM6/11/13
to akiba...@akiban.com
Hi there,

The integration of Persistit in the next version of Sonar (www.sonarsource.org) is a nice improvement. It allows to deal with big amount of temporary data during source code analysis without requiring as much heap memory. So thanks to the Askiban team for this great project.

After some memory profiling, we found that Persistit pre-allocates at least 30Mb of heap mem, whatever the buffer pool configuration. In our case ten buffers of 8192 bytes are supposed to be used (+ an expected overhead of 14%):

Properties props = new Properties();
props.setProperty("datapath", tempDir.getAbsolutePath());
props.setProperty("logpath", "${datapath}/log");
props.setProperty("logfile", "${logpath}/persistit_${timestamp}.log");
props.setProperty("buffer.count.8192", "10");
props.setProperty("journalpath", "${datapath}/journal");
props.setProperty("tmpvoldir", "${datapath}");
props.setProperty("volume.1", "${datapath}/persistit,create,pageSize:8192,initialPages:10,extensionPages:100,maximumPages:25000");
persistit.setProperties(props);
persistit.initialize();
volume = persistit.createTemporaryVolume();

The increase of 30Mb appears when Persistit is initializing, so indexes are empty at this time. Could you help us to understand this behavior ?

Thank you,
Simon

Peter Beaman

unread,
Jun 11, 2013, 9:46:53 AM6/11/13
to simon.b...@sonarsource.com, akiban-user
Hi Simon,

I'm glad Persistit is working well in your application.  I look forward to investigating Sonar - looks like an interesting product.

The com.persistit.JournalManager class allocates a 16MB byte buffer (called _writeBuffer) which is where records that will be written to the journal are held before writing.The JournalManager's JOURNAL_COPIER thread also allocates a 16MB buffer used in sorting pages during the journal page copying process.  I believe these two buffers are the source of the 30MB allocation you mentioned. Unfortunately both sizes are currently of fixed size and untunable.

Are you running into memory constraints? If so you could modify the constants com.persistit.JournalManageMXBean#DEFAULT_BUFFER_SIZE and #DEFAULT_COPY_BUFFER_SIZE. In a future release we could make these parameters tunable. These predefined sizes work well with some of our largest configurations and tests - e.g., several hundred threads running TPC-C-like load with a million or more 16KB buffers. I'm sure that reducing their size for a configuration that otherwise runs acceptably with 10 8KB buffers would be fine.

Please feel free to send any more questions or comments, and good luck with Sonar.

Peter Beaman








--
You received this message because you are subscribed to the Google Groups "Akiban User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akiban-user...@akiban.com.
To post to this group, send email to akiba...@akiban.com.
Visit this group at http://groups.google.com/a/akiban.com/group/akiban-user/?hl=en-US.



simon.b...@sonarsource.com

unread,
Jun 13, 2013, 3:59:31 AM6/13/13
to akiba...@akiban.com, simon.b...@sonarsource.com
Hi Peter,

Thank you for the explanation. It exactly confirms our profiling results.

About the memory constraints, an overhead of 32Mb is accepted, even if you think that it could be reduced regarding our 10 8k buffers. So configuring the size of these journal buffers is not mandatory for our needs.


> Please feel free to send any more questions or comments, and good luck with Sonar.

Thanks, and good luck for Persistit and Akiban too !

Reply all
Reply to author
Forward
0 new messages