Fail to back up the KyotoTycoon

[Yoshiro Ozeki]

Hello.

We are currently operating a KyotoTycoon.

I'm taking a hot backup on a regular basis.

But sometimes(about 80%) the data is broken.

When I ran the following command, that output WARN, and after that the recover process was started.

  $ kttimedmgr inform "/path/to/casket.kch.01400537662830000001#type=kch"

  kttimedmgr: [WARN]: /usr/local/include/kchashdb.h: 2717: reorganize_file: -: reorganizing the database

The recover process created /path/to/casket.kch.01400537662830000001.tmpkch, the original file was overwritten.

Information of specifications:

* Kyoto Tycoon on AWS(c3.2xlarge), Ubuntu12.04, instance storage(SSD)

* The file size is 15GB. The number of keys is about 170 million. The data has expiration time.

* Write process is only to master from the application

* "set" to the master is about 1000QPS

* "hot backup" runs on slave

Setting of slave:

  ktserver -port 1978 -tout 30 -th 256 -dmn -pid /path/to/ktserver.pid -log /path/to/ktserver.log -ls -ulog /path/to/ulog -ulim 500m -sid 2 -mhost xxx.xxx.xxx.xxx -mport 1978 -rts /path/to/rts -cmd /path/to/bin /path/to/casket.kch#bnum=80000000#msiz=8g#dfunit=8

Backup script:

  #! /bin/sh

  srcfile="$1"

  destfile="$1.$2"

  cp -f "$srcfile" "$destfile"

Backup command:

  ktremotemgr sync-cmd dbbackup-db 0

I don't understand why this problem is happened.

Could you tell me the same problems, or the good solutions.

Thank you.

[Yoshiro Ozeki]

Sorry, I forgot to write important information.

  $ ktserver --version
  Kyoto Tycoon 0.9.56 (2.19) on Linux (Kyoto Cabinet 1.2.76)

Thanks.

2014年5月22日木曜日 18時50分06秒 UTC+9 Yoshiro Ozeki: