Tune KyotoCabinet File Hash Database

[Luc Donnet]

Hi,

I use KyotoCabinet to backup nodes from osm data in a temporary file. I need good performance because osm data contains heavy data set.

I use the last ruby api to use KyotoCabinet in my ruby project : https://github.com/dryade/activeroad.

So I write a little ruby script to benchmark the use of KyotoCabinet :https://github.com/dryade/activeroad/blob/master/script/benchmark_import_kyotocabinet.rb.

The results are  : 

Import without options

Count: 6000000

Time: 149.853 sec.

Usage: 64.473 MB

Import with options apow=8, opts=l, bnum=2000000 and msiz=50000000

Count: 6000000

Time: 128.868 sec.

Usage: 47.691 MB

Import and update without options

Count: 6000000

Time: 298.644 sec.

Usage: 64.027 MB

Import and update with options apow=8, opts=l, bnum=2000000 and msiz=50000000

Count: 6000000

Time: 1006.279 sec.

Usage: 47.688 MB

I don't understand why performance felt when I use options like these : #apow=8#opts=l#bnum=2000000#msiz=50000000.

I think it could improve it but it doesn't work. Does anyone have already tune KyotoCabinet File Hash Database?

Best Regards 

Luc Donnet