How to reduce my db size ?

[jam+]

Hi,

Currently my disk size is almost used over 75%, maybe I should do something to reduce my db size.

I am trying to figure out if I wanna reduce my db size by deleting some garbage records, does it work ?

Or is there any scenario can recommend ?

Thanks.

[Esteban Donato]

you can set the retention-days property to keep your db capped.  Alternatively, if your keys or values are long strings you can set data compression.  Check out the section "store configuration" here http://www.project-voldemort.com/voldemort/configuration.html

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at http://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/groups/opt_out.

[Brendan Harris (a.k.a. stotch on irc.oftc.net)]

Hi Jam,

If this is the same configuration from your previous post about CPU, then you could have a few problems going on. You have two different on-disk storage engines running, which manage data differently and have different read/write patterns. You also have the JVM spending too much time on CPU, which could be impacting the bdb compaction, causing the bdb structure to grow. And both storage engines will be competing for I/O on the same volume.

Can you do us a favor and get us a du of /opt/vdm/service/data/voldemort/read-only and /opt/vdm/service/data/voldemort/bdb and get us the average I/O wait of /dev/md0. Could you also tell us what kind of configuration is /dev/md0 (number of disks, type of disks and RAID type)?

Also, how many keys are you storing in bdb and how many in read-only?

Thanks,

Brendan

[jam+]

Hi Brendan,

I think I have made some mistakes. 

After checking stores.xml I found only "bdb" using in "persistence", so base on current settings, it's make sense that I can't find the read-only folder.

Then I should modify my config as below:

storage.configs=voldemort.store.bdb.BdbStorageConfiguration (remove voldemort.store.readonly.ReadOnlyStorageConfiguration & voldemort.store.memory.CacheStorageConfiguration)

We are running VDM cluster on AWS EC2 and using RAID0 (EBS x 8) for /dev/md0, and because we restarted the cluster yesterday so currently I have no information about the I/O wait... 

BTW, du -h in /opt/vdm/service/data/bdb is 37G, and I roughly calculate the total key number is almost reach one hundred million (Needs to be confirmed by my DBA).

Thanks.

Brendan Harris (a.k.a. stotch on irc.oftc.net)於 2013年11月13日星期三UTC+8上午12時39分53秒寫道：

[Brendan Harris (a.k.a. stotch on irc.oftc.net)]

Hi Jam,

Chances are, if you optimize your JVM and remove those extra storage engines (as discussed in the other thread), that your bdb structure will reduce in size. Your cleaner threads may well be falling behind. Unless the situation is that you are either storing very large values/keys or are never overwriting keys and only ever creating new keys. In the latter situation, you simply need to have a retention policy. In the former situation, you'll need compression enabled (and perhaps to rethink what you want to store in voldemort). Try increasing the size of the JVM and removing the extra storage engines and see if that helps first.

There is also a bdb stats mbean that can show you whether or not you have a cleaner backlog, which will indicate whether or not the cleaners are falling behind.

Could you also paste your stores.xml file?

Thanks,

Brendan