I've been toying around with the idea of creating a "ruby optimized"
ruby fork
i.e.
1.8.6 REE with the original garbage collector (which is faster, is
that right?) and maybe with a few tweaks applied (like MBARI, faster
GCD, etc.)
The rationale behind this is that I actually want to build an
"optimized" version of ruby for windows, so if I base it off of REE
there might be some room for collaboration.
And then after that possibly 1.9 same thing (larger GC, tcmalloc).
Do you think that's something that REE would be happy to incorporate
or shall I do it on my own?
Just wondering. Don't want to steal any fire from REE :)
Thanks.
-=r
Speaking purely from the position of self-interest, what I would like to
see is that you work with the REE folks so that future REE releases are
more highly optimised. I think this is what they are already trying to
do (and, generally, succeeding).
That said, the REE GC bug hit us pretty hard and the recent forking
issue was a big deal as well. I spent some time trying to resolve the
GC bug but wasn't able to track it down. My point is that correctness
is still paramount, more important than performance.
rogerdpack wrote:
> I've been toying around with the idea of creating a "ruby optimized"
> ruby fork
> i.e.
> 1.8.6 REE with the original garbage collector (which is faster, is
> that right?) and maybe with a few tweaks applied (like MBARI, faster
> GCD, etc.)
> The rationale behind this is that I actually want to build an
> "optimized" version of ruby for windows, so if I base it off of REE
> there might be some room for collaboration.
> And then after that possibly 1.9 same thing (larger GC, tcmalloc).
> Do you think that's something that REE would be happy to incorporate
> or shall I do it on my own?
> Just wondering. Don't want to steal any fire from REE :)
> Thanks.
> -=r
> Speaking purely from the position of self-interest, what I would
> like to
> see is that you work with the REE folks so that future REE releases
> are
> more highly optimised. I think this is what they are already trying
> to
> do (and, generally, succeeding).
> That said, the REE GC bug hit us pretty hard and the recent forking
> issue was a big deal as well. I spent some time trying to resolve the
> GC bug but wasn't able to track it down. My point is that correctness
> is still paramount, more important than performance.
> rogerdpack wrote:
>> I've been toying around with the idea of creating a "ruby optimized"
>> ruby fork
>> i.e.
>> 1.8.6 REE with the original garbage collector (which is faster, is
>> that right?) and maybe with a few tweaks applied (like MBARI, faster
>> GCD, etc.)
>> The rationale behind this is that I actually want to build an
>> "optimized" version of ruby for windows, so if I base it off of REE
>> there might be some room for collaboration.
>> And then after that possibly 1.9 same thing (larger GC, tcmalloc).
>> Do you think that's something that REE would be happy to incorporate
>> or shall I do it on my own?
>> Just wondering. Don't want to steal any fire from REE :)
>> Thanks.
>> -=r
On Thu, May 21, 2009 at 3:28 PM, rogerdpack <rogerpack2...@gmail.com> wrote:
> I've been toying around with the idea of creating a "ruby optimized" > ruby fork > i.e.
> 1.8.6 REE with the original garbage collector (which is faster, is > that right?) and maybe with a few tweaks applied (like MBARI, faster > GCD, etc.)
MBARI integration is already planned. But it needs quite some work because MBARI's changes overlap with my own and that of others (e.g. the zero-stack-copy thread switching patch which I plan on integrating as well). I just haven't gotten the time to do it until now.
> The rationale behind this is that I actually want to build an > "optimized" version of ruby for windows, so if I base it off of REE > there might be some room for collaboration.
I thought MBARI is not Windows-compatible?
> And then after that possibly 1.9 same thing (larger GC, tcmalloc).
> Do you think that's something that REE would be happy to incorporate > or shall I do it on my own? > Just wondering. Don't want to steal any fire from REE :)
Sure. I wanted to do all these things myself but if you want to help, more power to you. :)
1.9 probably needs an entirely different fork because the codebase is different.
Regards, Hongli Lai -- Phusion | The Computer Science Company
Web: http://www.phusion.nl/ E-mail: i...@phusion.nl Chamber of commerce no: 08173483 (The Netherlands)
On Thu, May 21, 2009 at 5:05 PM, Christopher Thompson
<cthomp...@nexopia.com> wrote: > That said, the REE GC bug hit us pretty hard and the recent forking > issue was a big deal as well. I spent some time trying to resolve the > GC bug but wasn't able to track it down. My point is that correctness > is still paramount, more important than performance.
Sorry about the forking issue. The patch turns out to be not as stable as I thought it would be. In the future we'll incorporate new features as optional installer flags for a while before enabling them by default, in order to prevent things like this from happening again.
Regards, Hongli Lai -- Phusion | The Computer Science Company
Web: http://www.phusion.nl/ E-mail: i...@phusion.nl Chamber of commerce no: 08173483 (The Netherlands)
> MBARI integration is already planned. But it needs quite some work
> because MBARI's changes overlap with my own and that of others (e.g.
> the zero-stack-copy thread switching patch which I plan on
> integrating as well). I just haven't gotten the time to do it until
> now.
That would be cool. I wonder if we can get brent in on this...
> I thought MBARI is not Windows-compatible?
Works well on doze. I'm not sure, however, if tcmalloc works well on
windows. If not then maybe nedmalloc [?] not sure.
> Sure. I wanted to do all these things myself but if you want to help,
> more power to you. :)
Perhaps we could make some things optional, like...if you want the
"old school" GC you set some environment variable, would that be
acceptable? Windows users wouldn't benefit from the fork friendly GC
as much :)
> 1.9 probably needs an entirely different fork because the codebase is different.
Please elaborate--fork from 1.9 proper?
Thanks!
-=r
On Fri, May 22, 2009 at 5:23 AM, rogerdpack <rogerpack2...@gmail.com> wrote: > Perhaps we could make some things optional, like...if you want the > "old school" GC you set some environment variable, would that be > acceptable? Windows users wouldn't benefit from the fork friendly GC > as much :)
We already default to using the 'old school' GC. The copy-on-write friendly one must be explicitly enabled during runtime.
> Please elaborate--fork from 1.9 proper?
Yes, and rebasing a lot of changes against 1.9, instead of trying to merge 1.9 into the current codebase.
-- Phusion | The Computer Science Company
Web: http://www.phusion.nl/ E-mail: i...@phusion.nl Chamber of commerce no: 08173483 (The Netherlands)
> > That said, the REE GC bug hit us pretty hard and the recent forking
> > issue was a big deal as well. I spent some time trying to resolve the
> > GC bug but wasn't able to track it down. My point is that correctness
> > is still paramount, more important than performance.
> Sorry about the forking issue. The patch turns out to be not as stable
> as I thought it would be. In the future we'll incorporate new features
> as optional installer flags for a while before enabling them by
> default, in order to prevent things like this from happening again.
Don't let one incident resort you to such measures. Giving users
options is not always the best option. Take Gentoo as an example:
building packages with different USE flags and compiler optimizations
have only brought more complexities, bugs and harder debugging to the
table. It didn't prevent the bug reports either.
You have a clear vision of where REE is heading and should stick to
that. Combine it with a little more beta testing in the field and
issues like the SIGVTALRM will be easier to prevent.
rogerdpack wrote: >> MBARI integration is already planned. But it needs quite some work >> because MBARI's changes overlap with my own and that of others (e.g. >> the zero-stack-copy thread switching patch which I plan on >> integrating as well). I just haven't gotten the time to do it until >> now.
> That would be cool. I wonder if we can get brent in on this...
>> I thought MBARI is not Windows-compatible? > Works well on doze. I'm not sure, however, if tcmalloc works well on > windows. If not then maybe nedmalloc [?] not sure.
This was a while ago, you understand, but nedmalloc did not give us significantly reduced memory use compared to the default allocator (MRI Ruby). Similarly, it was not any faster. JEMalloc showed reduced memory use (about 25% less memory use) but was significantly slower than the default allocator.
It is entirely possible, given the similarities between my stats for nedmalloc compared to the default allocator, that I had not properly integrated it.
Jemalloc resulted in some weird issues, Ruby was never stable with it.
On 22 mei 2009, at 17:00, Christopher Thompson wrote:
>>> I thought MBARI is not Windows-compatible? >> Works well on doze. I'm not sure, however, if tcmalloc works well on >> windows. If not then maybe nedmalloc [?] not sure.
> This was a while ago, you understand, but nedmalloc did not give us > significantly reduced memory use compared to the default allocator > (MRI > Ruby). Similarly, it was not any faster. JEMalloc showed reduced > memory use (about 25% less memory use) but was significantly slower > than > the default allocator.
> You have a clear vision of where REE is heading and should stick to
> that. Combine it with a little more beta testing in the field and
> issues like the SIGVTALRM will be easier to prevent.
I agree it's best to plow forward--and also have RC candidates before
release, I suppose ;)
with regard to options, I'd argue that they'd be useful, at least for
the difference between windows and linux builds (and hopefully minor).
Cheers!
-=r
The MBARI and zero-copy context switching patches merger is coming
along nicely. We've spent half a week so far on trying to merge the
two and it felt like an uphill battle, but it seems to work now. The
work has been pushed to Github and it passes RubySpec, the Rails
tests, the EventMachine tests and the Phusion Passenger tests. Right
now it has only been tested on OS X, but it will be tested on Linux
soon. Continuations are still broken but we're working on it.
> The MBARI and zero-copy context switching patches merger is coming
> along nicely. We've spent half a week so far on trying to merge the
> two and it felt like an uphill battle, but it seems to work now. The
> work has been pushed to Github and it passes RubySpec, the Rails
> tests, the EventMachine tests and the Phusion Passenger tests. Right
> now it has only been tested on OS X, but it will be tested on Linux
> soon. Continuations are still broken but we're working on it.
On Tue, Jun 2, 2009 at 7:13 PM, Andrew Lippert <aclipp...@gmail.com> wrote:
> Nice work, Hongli. We are all looking forward to the fruits of your > labors. Any initial indications of statistical benefit over the > current release?
This usually depends on the application and the load, but we'll publish some comparison benchmarks.
-- Phusion | The Computer Science Company
Web: http://www.phusion.nl/ E-mail: i...@phusion.nl Chamber of commerce no: 08173483 (The Netherlands)
> We already default to using the 'old school' GC. The copy-on-write
> friendly one must be explicitly enabled during runtime.
Hmm. I thought I remembered having run some benchmarks once [a year
ago?] that showed REE [with the default 'old school' GC] was still a
few percent slower. Hence my thinking that for windows, using that
would be better.
Another recent thought I've had re: optimization is to take out the
event_hook functionality [or built two side by side versions, one
with, one without], since it's just used for debugging anyway, though
I suppose benchmarks are the only things that could tell us if that is
actually a useful optimization or not. I suppose I'd need to profile
and look at the core more to see if there's any low hanging
optimizations possible.
