Optimal BDB checkpoint configuration

[Feng Wu]

Hi,

Does anybody has any recommendations on what the optimal BDB
checkpoint configuration settings might be? Right now we are using
the default settings on Project Voldemort's configuration page, i.e.,
20 MB interval and 30000 ms interval. What we noticed is that lots of
Voldemort threads are blocked when there are active checkpoint
activities, and that coincides with our application showing really bad
max response (multi seconds), and Voldemort client complaining about
operation timed out.

Our current test settings:

je.cleaner.minUtilization=25
je.cleaner.threads=10
je.cleaner.readSize=1024000
je.cleaner.lockTimeout=100000
je.checkpointer.highPriority=true
je.env.backgroundReadLimit=5
je.env.backgroundWriteLimit=5

A few notes about these settings. We were currently using %5
minUtilization and 1 cleaner threads, but we are running out disk
space very quickly so one of our goal is to increase the
minUtilization to reclaims some disk space back. The other settings
were per Oracle recommendations to reduce disk IO and hopefully reduce
response time.

In our test environment we have a two node Voldemort cluster with
required read and write at 2. Our BDB log size is 10 MB. We tested 1
GB, 256 MB, 50 MB, and 10 MB log file sizes and 10 MB gave the best
response time. Our test starts with about 10GB data file, bdb file
utilization stays around 25% through out the test.

We are going to run a test with all the above settings but with
checkpoint high priority off to see whether it will make any
difference. In the meantime I'd appreciate any advice in terms of how
we might adjust the checkpoint intervals.

I'll cross post to BDB mailing list to see whether I might get some
pointers over there as well.

Thanks.

-Feng

[Feng Wu]

I apologize if this gets posted twice. Resending below email as
previous attempt seemed to have failed.

Thanks.

-Feng

[Geir Magnusson Jr.]

On Aug 25, 2009, at 3:21 PM, Feng Wu wrote:

>
> Hi,
>
> Does anybody has any recommendations on what the optimal BDB
> checkpoint configuration settings might be? Right now we are using
> the default settings on Project Voldemort's configuration page, i.e.,
> 20 MB interval and 30000 ms interval. What we noticed is that lots of
> Voldemort threads are blocked when there are active checkpoint
> activities, and that coincides with our application showing really bad
> max response (multi seconds), and Voldemort client complaining about
> operation timed out.
>

This is interesting. What's the load such that you get these pauses?

> Our current test settings:
>
> je.cleaner.minUtilization=25
> je.cleaner.threads=10
> je.cleaner.readSize=1024000
> je.cleaner.lockTimeout=100000
> je.checkpointer.highPriority=true
> je.env.backgroundReadLimit=5
> je.env.backgroundWriteLimit=5
>
> A few notes about these settings. We were currently using %5
> minUtilization and 1 cleaner threads, but we are running out disk
> space very quickly so one of our goal is to increase the
> minUtilization to reclaims some disk space back. The other settings
> were per Oracle recommendations to reduce disk IO and hopefully reduce
> response time.
>
> In our test environment we have a two node Voldemort cluster with
> required read and write at 2. Our BDB log size is 10 MB. We tested 1
> GB, 256 MB, 50 MB, and 10 MB log file sizes and 10 MB gave the best
> response time. Our test starts with about 10GB data file, bdb file
> utilization stays around 25% through out the test.

I did a lot of work on similar aspects a while ago as I too was
concerned about getting predictable space usage. I found similar
behavior and settled on using a 5MB file (although I'm guessing that a
10MB file will be no different). I also am being very aggressive with
utilization, going for the 50% max.

I've been very happy with this so far, but it's still early for us.

>
> We are going to run a test with all the above settings but with
> checkpoint high priority off to see whether it will make any
> difference. In the meantime I'd appreciate any advice in terms of how
> we might adjust the checkpoint intervals.
>
> I'll cross post to BDB mailing list to see whether I might get some
> pointers over there as well.

Let us know what they say please :)

geir

>
>
> Thanks.
>
> -Feng
>
> >

[Feng Wu]

Sorry for the long layoffs after my last posts.  Did not mean to only ask never give back.  Things got really busy, we managed to stabilize our application.  Then I had to go out of country for a week.  Anyhow, enough excuses already, I'll try to summarize what we found and hopefully these information maybe helpful to others.

1. Regarding load, we were testing at about 200 requests per second across four instances of our application, which talks to a two-node Voldemort cluster.

2. As for threads being blocked, it is not necessarily checkpointer thread blocking cleaner and Voldemort threads.  We also found one cleaner threads blocking other cleaner threads, and one application thread blocking many other application threads and/or cleaner threads.  It is when many application threads gets blocked when we always see our app's response time took a dive. 

3. je.checkpointer.highPriority=true did not make a meaningful difference.

4. Log cleaning definitely affects BDB/Voldemort performance.  When we were testing changing minUtilization from 5% to 25% and log file from 1GB to 10 MB, we restarted Voldemort and it started to convert/clean log files.  During this converting/cleaning period our application essentially became totally unresponsive. 

5. In our test environment we were able to trigger checkpoint and log cleaning via JMX, this was with about 12 GB log file at about 10% overall utilization.  However in production with greater than 60GB log files and about 10% overall utilization, JMX calls timed out and we were not able to trigger log conversion/cleaning.  Only by restart Voldemort did we get it to finally clean/convert all log files to 10 MB and overall utilization at about 25%.  Again, during this massive log cleaning process our app was not responsive at all. 

6. The rest of those je.xyz settings provided some improvement.  However the main setting that helped us is 10 MB log file size.  So that's the only change that went and remained in production.  Even when we reverted back to 5% minUtilization and a single cleaner thread ( don't ask why, it was an accident :-) ), we were able to achieve close to 20% overall log file utilization.  Though we may still go ahead and change the minUtilization to 25% explicitly, as the overall utilization percentage is gradually going down.

7. The biggest lesson we learned is that our test environment was not a sufficient analog to our production environment.  We ran a whole bunch of tests, however they are not always reproducible, wasting lots of our time and energy.  The testing boxes got lots of beating and eventually pretty much died and had to be rebooted.  In the future we'll probably explore Amazon AWS for our testing environment.

So far our app is relatively stable, meeting our interal SLA, essentially with one change, log file size changed from 1Gb to 10 MB.  We'll probably change minUtilization to 25%, from current 5%, to ensure our disk space usage don't go totally crazy.

Enough rumbling for now.  I probably missed some stuff that we have tested.  Hopefully someone will find something useful here :-)

-Feng

[Geir Magnusson Jr.]

This is good useful info, and mirrors much of what I found. I use 50%
util and 5MB or 10MB log files, and I also had noted that at small
util percentages, I got times when bringing V down that it just spent
serious time doing cleanup before it exited.

I did all my testing in EC2 using a 4 node cluster and about 4 client
machines. That was very effective.

geir