RFP: Bounding memory usage in TT/TC
Neal Richter
bound on memory usage of TT + TC. The reward is cash. Send me a
proposal. The patch needs to work, not crash TT/TC or corrupt data.
Here's a start: http://github.com/nealrichter/tokyotyrant_rsshack
I've attempted to do this myself and have not had the time to finish
or fully test it. I've asked Mikio for feedback/help finishing this
and he's been nearly silent on the request.
At the moment we (myself, Sam Tingleff and Mike Dierken) work around
this issue by continuing to play with various parameter TC settings
and restarting the daemon when the memory usage grows beyond a comfort
level.
I'm a C coder and have hacked the internals of BerkeleyDB in the past,
so can help review code, trade ideas, etc. We (as a team) don't have
the time to work on this at the moment.
If you are interested contact me! We've got a few other ideas for TT
enhancements as well...
Thanks - Neal Richter
Neal Richter
I think this will be easier and better to do with getrusage() than
Linux's libproc.
It's a POSIX function, thus would work with all Unix OSs.
http://linux.about.com/library/cmd/blcmdl2_getrusage.htm
http://www.osxfaq.com/man/2/getrusage.ws
http://www.manpages.info/freebsd/getrusage.2.html
PostgreSQL has a port that they keep up to date that supports:
/* This code works on:
* univel
* solaris_i386
* sco
* solaris_sparc
* svr4
* hpux 9.*
* win32
*/
http://doxygen.postgresql.org/getrusage_8c-source.html
I'll work on porting my rsshack patch to TT/TC over to this in the
next week.
Is anyone willing to help test or write code on this?
- Neal
Adrian Cuthbertson
>
> - Neal
Hi Neal,
I've only recently started with TC and have not yet used TT. I am
embedding TC in a Java (actually Clojure) application and have done a
certain amount of experimentation, but made no real heavy duty use of
it yet. I have been really impressed with the library and intend to
commit it to significant parts of my upcoming development. So, I have
an interest in its viability and in its robustness particularly.
I am an ex C programmer of (quite a few) years ago and about a year
ago dabbled with libevent and some C integration work, but I'm
probably quite rusty. However, I've been through the TC source and it
looks very well structured and not too daunting.
I do have a pretty heavy daily load but as I'm moving into TC usage, I
could provide some input into helping address some of the problematic
areas. Perhaps my experience with Clojure could be useful in setting
up some tests and/or a test framework of sorts - that could be
reasonably "low hanging fruit".
I will now go though the source again and your posts and references to
get more up to speed. I'll do some work on this over this weekend and
give some feedback after that.
- Regards, Adrian