Tokyo Tyrant wal file grows out of control and then db hangs for inserts

[Jerry Champlin]

During the last two days we have experienced a problem where tokyo
tyrant begins creating a tc write ahead log file. At some point there
is something that triggers this file to begin growing at about 500MB/
min. This goes along until the tyrant begins failing to insert new
records. When you stop and restart the database, it will read the
entire wal file into memory and then begin removing all records
indicated in the wal file. After a long time, it will complete and
most of the data is gone when the database finally opens. This is
similar to what happens when running tcbmgr -ln optimize against the
file.

Once this problem begins, tcrmgr copy fails with 9999 miscellaneous
error. I can however open the copied files and successfully operate
from them even though tcbmgr inform shows additional flags of
"fatal". On the copy tcbmgr -ln optimize seems to remove a few
records and then declares the file ok but tyrant will successfully
open the "repaired" file.

My objective is to not have the wal files grow out of control.
Possibly disabling this protection mechanism could be helpful as long
as the db is replicated and the app knows how to handle failover. Any
help will be greatly appreciated.

-Jerry

Misc Info

tc version 1.4.45 and tt version 1.1.40 on ubuntu 10.04 x64

/usr/bin/ttserver -log /var/log/tokyotyrant/ttserver.log -ld -pid /var/
log/tokyotyrant/ttserver.pid -ulog /data2/tokyotyrant_ulog /data/
tokyotyrant/ttserver.tcb#bnum=4000000#opts=lb

output from tcbmgr inform after last backup with no errors:
root@db2:/data2/tokyo_backups/20100826# tcbmgr inform ttserver.tcb
path: ttserver.tcb
database type: btree
additional flags: fatal
comparison function: lexical
max leaf member: 128
max node member: 256
leaf number: 210393
node number: 1048
bucket number: 4194301
alignment: 256
free block pool: 1024
inode number: 49274886
modified time: 2010-08-26T20:17:09Z
options: large bzip
record number: 719002
file size: 2086913280

It seems that with or without errors from tcrmgr copy the output from
tcbmgr is materially the same.

[andynnov]

Can it be the issue with large file support, did you compile TC with ./
configure --enable-off64?

thanks, Andrei