Bursts of excessive IO utilization with Kyoto Tycoon?

[Dan Crosta]

We have a Kyoto Tycoon server set up with about 110M keys, about 20G on-disk size, and the following startup options:

ktserver -ls -port 3003 -plsv /usr/libexec/ktplugservmemc.so -plex 'port=3103#opts=f' -sid 1 -ulog /path/to/kyoto.ulog -rts /path/to/kyoto.rts -cmd /path/to/bin '/path/to/kyoto.kch#opts=l#bnum=20000000#msiz=30g#dfunit=8'

The machine has 48G physical RAM, about 40G of which is used for filesystem cache (about half of that is, I presume, the ~20G kch file that is mmaped into ktserver?). ktserver is under constant but relatively low volumes of constant write activity -- about 500 to 1000 keys per second, more or less random distribution over the keyspace.

Much of the time, the database works great -- lookups reliably take < 1ms. However, every 5-20 minutes (it is not regular, but usually within that window of time) we get an approximately 5 minute burst of heavy IO activity, during which time the disk subsytem is 100% utilized. These periods last 4-5 minutes, and during that time, the kernel "flush-N:M" threads, which are responsible for flushing dirty pages to disk, are using the majority of the IO subsystem. This causes slowdowns for readers from the database, as well as another (small) database also hosted on the same machine.

I'm wondering if anyone has run into a situation like this, or has any recommendations for ways to tune the Kyoto Tycoon settings to help mitigate this?

Thanks,
- Dan

[Stephen Hamer]

I've seen similar behavior and had good results tuning the background_bytes and background_dirty_bytes parameters of the linux VM subsystem (https://www.kernel.org/doc/Documentation/sysctl/vm.txt). Lowering the point where writebacks occur has evened out the write load significantly.

I found it helpful to use the atop utility to get an idea of the effect that different values was having on the writeback volume.

--
Stephen Hamer

--
You received this message because you are subscribed to the Google Groups "Tokyo Cabinet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tokyocabinet-us...@googlegroups.com.
To post to this group, send email to tokyocabi...@googlegroups.com.
Visit this group at http://groups.google.com/group/tokyocabinet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Dan Crosta]

Thanks, Stephen, I'll give that a try. I had been raising those settings, though in the light of a new day it does make sense to lower them. What values are you using, or do you think our use cases are sufficiently different that I should just experiment on my own?

Do you (or does anyone) know if the other database types have better IO properties, like the file tree db or the directory or forest dbs (I'm not even clear on whether the latter are options for KT or just for KC).

Thanks,

- Dan

[Stephen Hamer]

I have dirty_ratio set at 40 and dirty_background_bytes at 5000000. I have really, really slow IO on this system though so might not be appropriate for you. There wasn't a lot of thought that went into these values. It was largely trial and error to find values that seemed to reduce the IO storms.

I've only used the tree DB so I can't speak to the relative performance of the different types.

 

Stephen