Are Threads mapped to threads on the underlying operating system in 1.9 or still using green threads? If still using green threads, it is so because of a designing decision or it is a limitation of the current implementation?
By the way, except for eigenclass there is no official changelog of changes in 1.9 ? It would be great (for the news, the people getting excited / start deprecating code, etc)
Jordi wrote: > Are Threads mapped to threads on the underlying operating system in > 1.9 or still using green threads? > If still using green threads, it is so because of a designing decision > or it is a limitation of the current implementation?
> By the way, except for eigenclass there is no official changelog of > changes in 1.9 ? > It would be great (for the news, the people getting excited / start > deprecating code, etc)
> -- > Jordi
You mean "changelog" in the sense of differences between 1.8 and 1.9 syntax and semantics, right? There is a changelog downloaded every time you update the source from the repository of changes *within* 1.9.
In any event, given that we are approaching the final month before release, what I think would be a really great idea would be for everyone who has the free time and machines to throw everything they can think of at 1.9 -- your favorite one-liners, huge Rails projects, benchmarks, Windows extension builds, GCC 4.3, Fedora 8, cross-compiling for ARM, etc. I personally don't have a huge code base in Ruby 1.8, so it's not a big deal for me to do a lot of new code in 1.9.
On Nov 14, 2007 10:02 AM, Jordi <mumi...@gmail.com> wrote:
> Are Threads mapped to threads on the underlying operating system in > 1.9 or still using green threads?
Native Threads.
> If still using green threads, it is so because of a designing decision > or it is a limitation of the current implementation?
Guess someone from "core" will need to properly answer this, but switching from green to native threads is a huge redesign this will require better and careful coding of C extensions :-)
> By the way, except for eigenclass there is no official changelog of > changes in 1.9 ? > It would be great (for the news, the people getting excited / start > deprecating code, etc)
Guess no one, beside eigenclass took the task of parsing the whole changelog to summarize up-to-date the changes in the upcoming ruby, even that we are so close to feature-freeze :-O
-- Luis Lavena Multimedia systems - Leaders are made, they are not born. They are made by hard effort, which is the price which all of us must pay to achieve any goal that is worthwhile. Vince Lombardi
On Nov 14, 2007 11:01 PM, M. Edward (Ed) Borasky <zn...@cesmail.net> wrote:
> Jordi wrote: > > Are Threads mapped to threads on the underlying operating system in > > 1.9 or still using green threads? > > If still using green threads, it is so because of a designing decision > > or it is a limitation of the current implementation?
I don't mean the internals of all the changes, just:
- A new interpreted (previously YARV) 2.5x faster on averaged - threads now are natives
Something like that would be nice for a lot of that is getting interested in Ruby, news places, etc.
> > By the way, except for eigenclass there is no official changelog of > > changes in 1.9 ? > > It would be great (for the news, the people getting excited / start > > deprecating code, etc)
> > -- > > Jordi
> You mean "changelog" in the sense of differences between 1.8 and 1.9 > syntax and semantics, right? There is a changelog downloaded every time > you update the source from the repository of changes *within* 1.9.
> In any event, given that we are approaching the final month before > release, what I think would be a really great idea would be for everyone > who has the free time and machines to throw everything they can think of > at 1.9 -- your favorite one-liners, huge Rails projects, benchmarks, > Windows extension builds, GCC 4.3, Fedora 8, cross-compiling for ARM, > etc. I personally don't have a huge code base in Ruby 1.8, so it's not a > big deal for me to do a lot of new code in 1.9.
Jordi wrote: > Are Threads mapped to threads on the underlying operating system in > 1.9 or still using green threads? > If still using green threads, it is so because of a designing decision > or it is a limitation of the current implementation?
As others have responded, Ruby 1.9 uses native threads. But you still don't get true concurrency. The fear is that there are extensions that are not thread-safe yet. So even though native threads are used, only one is allowed to run at a time. This means that you won't see increased performance running a multi-threaded Ruby program on a multi-core CPU, for example.
> By the way, except for eigenclass there is no official changelog of > changes in 1.9 ? > It would be great (for the news, the people getting excited / start > deprecating code, etc)
I posted this list to my blog back in August. Probably a bit out of date now, but it hits the major points:
> -----Original Message----- > From: David Flanagan [mailto:da...@davidflanagan.com] > Sent: Wednesday, November 14, 2007 11:31 AM > To: ruby-c...@ruby-lang.org > Subject: Re: state of threads in 1.9
> Jordi wrote: > > Are Threads mapped to threads on the underlying operating system in > > 1.9 or still using green threads? > > If still using green threads, it is so because of a > designing decision > > or it is a limitation of the current implementation?
> As others have responded, Ruby 1.9 uses native threads. But > you still don't get true concurrency. The fear is that there > are extensions that are not thread-safe yet. So even though > native threads are used, only one is allowed to run at a > time. This means that you won't see increased performance > running a multi-threaded Ruby program on a multi-core CPU, > for example.
Can't we have both native *and* green threads? There are advantages and disadvantages to both, yes? I know at least one user who says he won't be able to use Ruby 1.9/2.0 because he's using it on very old hardware (486's) for Point of Sale software. The native threads will be too heavy.
Really, the only time that I've needed native thread support is for extending existing C libraries. I don't see the upside for general concurrency, since we're not getting multi-core support anyway.
Have I missed something?
Regards,
Dan
This communication is the property of Qwest and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.
> Really, the only time that I've needed native thread support is for > extending existing C libraries. I don't see the upside for general > concurrency, since we're not getting multi-core support anyway.
No multi-core support... ever? I had presumed native threads were a step in that direction. (No?)
What I've been looking forward to is embedding Ruby in a multi- threaded C++ app, and someday being able to have any C++ thread call directly into the Ruby interpreter.
Currently I've implemented a bridging mechanism in our C++ app to make this possible... but it's not nearly as convenient as direct calls would be.
> > Really, the only time that I've needed native thread support is for > > extending existing C libraries. I don't see the upside for general > > concurrency, since we're not getting multi-core support anyway.
> No multi-core support... ever? I had presumed native threads were > a step in that direction. (No?)
They didn't say that :-P
The thing is that some C extensions currently coded don't are thread-safe, so weird things could happen (including crashes and loss of profit) ;-)
From my POV, is a ongoing process to be fully thread-safe, when that goal is achieved, the lock around this issue will be removed.
-- Luis Lavena Multimedia systems - Leaders are made, they are not born. They are made by hard effort, which is the price which all of us must pay to achieve any goal that is worthwhile. Vince Lombardi
> From: "Berger, Daniel" <Daniel.Ber...@qwest.com> >> Really, the only time that I've needed native thread support is for >> extending existing C libraries. I don't see the upside for general >> concurrency, since we're not getting multi-core support anyway.
> No multi-core support... ever?
No multi-core support yet.
> I had presumed native threads were a step in that direction. (No?)
It is. We're going from one thread to many threads with a global lock. Fine-grained locking is both expensive and hard. It has taken FreeBSD years to move from a global lock to fine grained locking.
> What I've been looking forward to is embedding Ruby in a multi- > threaded C++ app, and someday being able to have any C++ thread > call directly into the Ruby interpreter.
If you've embedded ruby, you may be able to have multiple C++ threads running and one ruby thread running (out of N ruby threads).
-- Poor workers blame their tools. Good workers build better tools. The best workers get their tools to do the work for them. -- Syndicate Wars
Would not it be possible to release the global lock manually in C extensions ? I guess that implementation of blocking calls like Mutex#lock, #select and such must release the global lock before calling the blocking functions ? -- Sylvain
> On Nov 14, 2007, at 11:18 , Bill Kelly wrote: > > From: "Berger, Daniel" <Daniel.Ber...@qwest.com> > >> Really, the only time that I've needed native thread support is for > >> extending existing C libraries. I don't see the upside for general > >> concurrency, since we're not getting multi-core support anyway.
> > No multi-core support... ever?
> No multi-core support yet.
> > I had presumed native threads were a step in that direction. (No?)
> It is. We're going from one thread to many threads with a global > lock. Fine-grained locking is both expensive and hard. It has taken > FreeBSD years to move from a global lock to fine grained locking.
It is really bad news for me. My application is really heavy threaded and even with only 2 or 4 cores I wanted to use them (you may think that Ruby is not the right language for that anyway ...). I guess I can use C++ and Qt library (my option number two after ruby, suggestions accepted)
> > What I've been looking forward to is embedding Ruby in a multi- > > threaded C++ app, and someday being able to have any C++ thread > > call directly into the Ruby interpreter.
> If you've embedded ruby, you may be able to have multiple C++ threads > running and one ruby thread running (out of N ruby threads).
So, you mean having all the threads code but one done in C++ and only one in Ruby? I think PyPy achieved true threading. Anyway, with dual cores everywhere, Toshiba using Ruby over the Cell processor, etc. Multicore-systems will not go away and the pressure for Ruby to fully use will only increase.
> > > Really, the only time that I've needed native thread support is for > > > extending existing C libraries. I don't see the upside for general > > > concurrency, since we're not getting multi-core support anyway.
> > No multi-core support... ever? I had presumed native threads were > > a step in that direction. (No?)
> They didn't say that :-P
> The thing is that some C extensions currently coded don't are > thread-safe, so weird things could happen (including crashes and loss > of profit) ;-)
> From my POV, is a ongoing process to be fully thread-safe, when that > goal is achieved, the lock around this issue will be removed.
If the only fear is that extensions won't be thread-safe why not let the extension author declare that in the extension in some way. This can be checked at extension load time.
That way the interpreter could use concurrent threads by default, and handle not-yet thread-safe extensions. There are several possibilities how to do that (in order of decreasing flexibility and probably also implementation complexity):
- lock some global lock on entering the unsafe extension - start using the lock once the extension is loaded - require an option when the interpreter is started to switch on the lock and allow loading extensions not marked as thread safe
As 1.9 is supposed to be some sort of beta release even the last option should be acceptable for that.
Jordi wrote: > On Nov 15, 2007 11:51 PM, M. Edward (Ed) Borasky <zn...@cesmail.net> wrote: >> Jordi wrote: >>> Anyway, with dual cores everywhere, Toshiba using Ruby over the Cell >>> processor, etc. >> Do you have any detail/references on that? I'm very interested in the >> "Cell" processor.
> > I had presumed native threads were a step in that direction. (No?)
> It is. We're going from one thread to many threads with a global > lock. Fine-grained locking is both expensive and hard. It has taken > FreeBSD years to move from a global lock to fine grained locking.
But a high level language is far different from a kernel.
Maybe I just show my ignorance with this comment but... In Ruby methods are not called, messages are sent to objects. If shared variables are avoided, it should be easy to avoid the problems of concurrency.
On Fri, 16 Nov 2007 00:19:25 +0900, Jordi <mumi...@gmail.com> wrote: > Maybe I just show my ignorance with this comment but... > In Ruby methods are not called, messages are sent to objects. If > shared variables are avoided, it should be easy to avoid the problems > of concurrency.
Unfortunately, Ruby is rife with shared, mutable objects: classes and modules.
On Thu, 15 Nov 2007 22:38:05 +0900, Sylvain Joyeux <sylvain.joy...@m4x.org> wrote: > Would not it be possible to release the global lock manually in C > extensions ? > I guess that implementation of blocking calls like > Mutex#lock, #select and such must release the global lock before calling > the blocking functions ?
On Nov 15, 2007 9:58 PM, MenTaLguY <men...@rydia.net> wrote:
> On Thu, 15 Nov 2007 22:38:05 +0900, Sylvain Joyeux <sylvain.joy...@m4x.org> wrote: > > Would not it be possible to release the global lock manually in C > > extensions ?
Wasn't there a talk of possibility of allowing release of global lock in C extensions, allowing fine grained control to C extension authors, who want more than one thread running parallely?
> > I guess that implementation of blocking calls like > > Mutex#lock, #select and such must release the global lock before calling > > the blocking functions ?
> There is rb_thread_blocking_region()...
> -mental
-- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals.
> On Nov 15, 2007 7:33 AM, Eric Hodel <drbr...@segment7.net> wrote: >> On Nov 14, 2007, at 11:18 , Bill Kelly wrote: >>> From: "Berger, Daniel" <Daniel.Ber...@qwest.com> >>>> Really, the only time that I've needed native thread support is for >>>> extending existing C libraries. I don't see the upside for general >>>> concurrency, since we're not getting multi-core support anyway.
>>> No multi-core support... ever?
>> No multi-core support yet.
>>> I had presumed native threads were a step in that direction. (No?)
>> It is. We're going from one thread to many threads with a global >> lock. Fine-grained locking is both expensive and hard. It has taken >> FreeBSD years to move from a global lock to fine grained locking.
> It is really bad news for me. My application is really heavy threaded > and even with only 2 or 4 cores I wanted to use them (you may think > that Ruby is not the right language for that anyway ...). I guess I > can use C++ and Qt library (my option number two after ruby, > suggestions accepted)
If this is something you really want to have happen, the best way to make it reality is to start working on it yourself. Right now ruby is taking baby steps.
>>> What I've been looking forward to is embedding Ruby in a multi- >>> threaded C++ app, and someday being able to have any C++ thread >>> call directly into the Ruby interpreter.
>> If you've embedded ruby, you may be able to have multiple C++ threads >> running and one ruby thread running (out of N ruby threads).
> So, you mean having all the threads code but one done in C++ and only > one in Ruby? > I think PyPy achieved true threading. > Anyway, with dual cores everywhere, Toshiba using Ruby over the Cell > processor, etc. Multicore-systems will not go away and the pressure > for Ruby to fully use will only increase.
We know this (it has been the plan since 2005), but we don't want to achieve true threading at the expense of performance. Right now there are more important things to work on.
-- Poor workers blame their tools. Good workers build better tools. The best workers get their tools to do the work for them. -- Syndicate Wars
>> What I've been looking forward to is embedding Ruby in a multi- >> threaded C++ app, and someday being able to have any C++ thread >> call directly into the Ruby interpreter.
> If you've embedded ruby, you may be able to have multiple C++ threads > running and one ruby thread running (out of N ruby threads).
Hmm. In 1.8 we currently have multiple C++ threads each talking to their own Ruby green thread... But of course Ruby is running in a single native thread. The main issue is the Ruby threads end up polling for events from the C++ threads, where I'd prefer they could wait on a native mutex / condition variable.
This leads me to wonder if the following will be possible in 1.9:
Since the Ruby threads will be native, could I have a native Ruby thread--via a C extension--wait on a native mutex, without blocking all the rest of the native Ruby threads?
(Or would there be some unavoidable issue with a global interpreter lock such that waiting on a native mutex ends up effectively blocking all Ruby threads anyway?)
If each native Ruby thread could wait on a native mutex without blocking the other Ruby threads, that would be awesome!
func() is your function which you want to do without GL. YARV release GL and call func(). If func() returned, YARV acquire GL and continue ruby program.
ubf (unblocking function) is called when another thread interrupts this thread with Thread#kill, or main thread exits. You must set ubf as interrupt function for func(). You must write ubf or ruby will not exit until func() ends.
Jordi wrote: > It is really bad news for me. My application is really heavy threaded > and even with only 2 or 4 cores I wanted to use them (you may think > that Ruby is not the right language for that anyway ...). I guess I > can use C++ and Qt library (my option number two after ruby, > suggestions accepted)
You may wish to try JRuby, which allows concurrent native threading. In 1.1 we're trying to tidy up the core classes to be entirely thread-safe, but if you take care not to manipulate the same data structures in unsafe ways you can run Ruby code concurrently without many issues in JRuby.
> func() is your function which you want to do without GL. YARV > release GL and call func(). If func() returned, YARV acquire GL and > continue ruby program.
> ubf (unblocking function) is called when another thread interrupts > this thread with Thread#kill, or main thread exits. You must set > ubf as interrupt function for func(). You must write ubf or ruby > will not exit until func() ends.
Note that 1.9 thread interface is not tested, so we need more experience. This means your report is important, and interfaces may change if necessary :)
