Landed large commit/branch into default

2 views
Skip to first unread message

burtonator

unread,
Apr 20, 2012, 4:31:06 PM4/20/12
to peregrine...@googlegroups.com
I just landed a large branch into default.

1.  Fixes a major memory bug we had on running large jobs and leaking memory and not releasing it.  

2.  New fix also uses RLIMIT_MEMLOCK so that we are FORCED within a constrained window so if this happens again it will not lock up the kernel but instead the process will segfault once too much memory is used.

3.  The daemon now setuids itself for both security and so that memlock rlimits apply.

This means that ALL our resources are now constrained.

I also think it means that we can run larger jobs and release 0.6.0 with support for Cassandra... w00h00.

I'm going to start playing with it benchmarking larger jobs.

I also rewrote the unit test framework so that we actually startup daemons as procs and setuid them as opposed to running them as threads which lead to a BUNCH of stupid bugs since the tests were running differently than in production.

Kevin

Roland Gude

unread,
Apr 23, 2012, 4:05:11 AM4/23/12
to peregrine...@googlegroups.com
this sounds great. i will port the combiner to the new head asap and see if it helps me with my memory bug.
Reply all
Reply to author
Forward
0 new messages