I thought I'd talk about my new project here, since there is a good chance that someone might be interested in it. I'm planning to put Ruby on top of Smalltalk VMs. Ruby and Smalltalk are very similar under the covers, so Smalltalk VMs are a very good match for the language. This will give Ruby a much faster execution environment (perhaps 30X), VMs which are capable of incremental garbage collection, generational garbage collection that is so fast your progra still works even with an infinite loop allocating new objects (I do this as a lark sometimes), a wonderful debugger which will let programmers modify methods on the fly & continue execution, a "workspace" window where you can execute arbitrary code, a visual "inspect", a powerful "Refactoring Browser," an industrial strength OODB (Gemstone) with objects and methods you can define in Ruby, and a readily accessible meta-level which will allow Rubyists to readily modify their own language. (For example, you could then use Method wrappers to very quickly implement an Aspect-Oriented Ruby.)
My strategy for doing this involves writing a Ruby parser (or, rather, translating the existing one in JRuby to Ruby) then writing a Smalltalk Parser object to request parsing of Ruby code into an AST from a Ruby program outside of Smalltalk. We then reify the AST inside the image and use it to compile Ruby methods into bytecodes. (Multiple syntaxes can coexist in one Smalltalk image.) Once this is done, we can then compile the external Ruby parser and bring it into Smalltalk. Afterwards, we can use the Refactoring Browser Smalltalk parser plus a little runtime type inferencing to incrementally transform the image into pure Ruby.
We can do all of this without changing the structure of Ruby files & Modules or requiring Rubyists to do Smalltalk style image oriented development. And for those of you sharp enough to wonder: yes, we can handle Modules, Mixins, and fully qualified Method names without changing the Smalltalk VMs. (At least those that have Namespaces.)
If anyone is interested, please drop me a line.
--Peter
-- There's neither heaven nor hell, save what we grant ourselves. There's neither fairness nor justice, save what we grant each other.
> I thought I'd talk about my new project here, since there is a good > chance that someone might be interested in it. I'm planning to put > Ruby on top of Smalltalk VMs. Ruby and Smalltalk are very similar > under the covers, so Smalltalk VMs are a very good match for the > language. This will give Ruby a much faster execution environment > (perhaps 30X), VMs which are capable of incremental garbage collection, > generational garbage collection that is so fast your progra still works > even with an infinite loop allocating new objects (I do this as a lark > sometimes), a wonderful debugger which will let programmers modify > methods on the fly & continue execution, a "workspace" window where you > can execute arbitrary code, a visual "inspect", a powerful "Refactoring > Browser," an industrial strength OODB (Gemstone) with objects and > methods you can define in Ruby, and a readily accessible meta-level > which will allow Rubyists to readily modify their own language. (For > example, you could then use Method wrappers to very quickly implement > an Aspect-Oriented Ruby.)
> My strategy for doing this involves writing a Ruby parser (or, rather, > translating the existing one in JRuby to Ruby) then writing a Smalltalk > Parser object to request parsing of Ruby code into an AST from a Ruby > program outside of Smalltalk. We then reify the AST inside the image > and use it to compile Ruby methods into bytecodes. (Multiple syntaxes > can coexist in one Smalltalk image.) Once this is done, we can then > compile the external Ruby parser and bring it into Smalltalk. > Afterwards, we can use the Refactoring Browser Smalltalk parser plus a > little runtime type inferencing to incrementally transform the image > into pure Ruby.
> We can do all of this without changing the structure of Ruby files & > Modules or requiring Rubyists to do Smalltalk style image oriented > development. And for those of you sharp enough to wonder: yes, we can > handle Modules, Mixins, and fully qualified Method names without > changing the Smalltalk VMs. (At least those that have Namespaces.)
> If anyone is interested, please drop me a line.
> --Peter
> -- > There's neither heaven nor hell, save what we grant ourselves. > There's neither fairness nor justice, save what we grant each other.
-- R. Mark Volkmann Partner, Object Computing, Inc.
On Apr 7, 2005, at 10:47 AM, R. Mark Volkmann wrote:
> I think it would be really cool if you could do this on top of the > free Squeak > VM.
Yes, I was thinking along these lines. However, no real Namespaces (yet) in Squeak. We could finagle this by referencing classes with Module-name prefixes. (This has already been proposed.) When real Namespaces appear in Squeak, we could then properly place the classes in modules/namespaces and rename them programatically.
The VMs I am targeting first are commercial ones, but they have free versions/licenses. (VisualWorks and Gemstone) In addition, they are very attractive for implementing servers. (VisualWorks is damn fast, and Gemstone is a mature, rock-solid OODB.)
-- There's neither heaven nor hell, save what we grant ourselves. There's neither fairness nor justice, save what we grant each other.
> PS> I thought I'd talk about my new project here, since there is a good > PS> chance that someone might be interested in it. I'm planning to put
> Is this just brainstorming or have you already done something ?
Lothar,
I have my hands on a working copy of Racc, and the JRuby source. I've begun the port of the RubyLexer into Smalltalk, but I've decided to abandon that and use the bootstrap from outside the image technique instead. I've also started the port of DefaultRubyParser.y from the JRuby source. I have a simple Ruby script that uses Regexps to transform the DefaultRubyParser.ry into Ruby. (Regexps are not powerful enough to tackle the job generically -- that would take a Context Free Grammar -- but I am including enough specific context in the regexps that it will work in the specific cases needed.)
I have also worked out and demonstrated the manipulation of method dictionaries so that we can compile fully qualified Ruby method names for the VisualWorks VM. I have sketched out a design for handling Ruby Mixins and assignment-implicit accessors.
I have also recruited a helper for translating the AST node classes. (There is something to be said for little languages. It's a lot easier to build syntax driven tools for them!)
Next, I will be taking a very simplified Ruby subset (the calculator example from the "Dragon Book" which comes with Racc, and the ability to define methods in Ruby classes) and working out the Parser-Outside-The-Image scheme with that. Hopefully I'll have that by this weekend.
--Peter
-- There's neither heaven nor hell, save what we grant ourselves. There's neither fairness nor justice, save what we grant each other.
- Would this support programming both within or without an image, choice up to the user?
- From what I understand Smalltalk has a small VM written in C, with the rest written in Smalltalk itself. Would your version be the Ruby equivalent of this (VM with everything else in Ruby), or would there still be some Smalltalk behind the Ruby?
On Apr 7, 2005 12:19 PM, jc <james.cromw...@gmail.com> wrote:
> Hi Peter,
> - Would this support programming both within or without an image, > choice up to the user?
> - From what I understand Smalltalk has a small VM written in C, with > the rest written in Smalltalk itself. Would your version be the Ruby > equivalent of this (VM with everything else in Ruby), or would there > still be some Smalltalk behind the Ruby?
(Although this is about Ruby in Ruby, not Ruby in smalltalk)
> -jc
-- thanks, -pate ------------------------- We are often unable to tell people what they need to know, because they want to know something else, and would therefore only misunderstand what we said - the Raven (George MacDonald, Lilith)
Peter Suk wrote: > I thought I'd talk about my new project here, since there is a good > chance that someone might be interested in it. I'm planning to put > Ruby on top of Smalltalk VMs.
This does indeed sound promising. Having a self-hosted Ruby would have certain benefits as well.
> a wonderful debugger which will let programmers modify methods on the > fly & continue execution
That is already possible with ruby-breakpoint which spawns an IRB session at a specific place of your code. See http://ruby-breakpoint.rubyforge.org/ if you are interested in this sort of things.
> - Would this support programming both within or without an image, > choice up to the user?
Basically, you could treat the image as just a very powerful debugger/IDE. The image would spit out new versions of your module files, which you could then edit/diff to your liking. I have devised a very powerful algorithm to respect comments, based on diffs of token sequences.
> - From what I understand Smalltalk has a small VM written in C, with > the rest written in Smalltalk itself. Would your version be the Ruby > equivalent of this (VM with everything else in Ruby), or would there > still be some Smalltalk behind the Ruby?
Initially, there would be some Smalltalk behind the Ruby, but we'd fix it so you'd never see it. Eventually, we can develop a completely Ruby meta-layer.
-- There's neither heaven nor hell, save what we grant ourselves. There's neither fairness nor justice, save what we grant each other.
> On Apr 7, 2005 12:19 PM, jc <james.cromw...@gmail.com> wrote: >> Hi Peter,
>> - Would this support programming both within or without an image, >> choice up to the user?
>> - From what I understand Smalltalk has a small VM written in C, with >> the rest written in Smalltalk itself. Would your version be the Ruby >> equivalent of this (VM with everything else in Ruby), or would there >> still be some Smalltalk behind the Ruby?
> (Although this is about Ruby in Ruby, not Ruby in smalltalk)
I am aware of MetaRuby. This project is also about Ruby in Ruby. (Smalltalk is in Smalltalk. But the VM doesn't care which language/meta-language it is running!)
However, this meta-Ruby will run two orders of magnitude faster!
-- There's neither heaven nor hell, save what we grant ourselves. There's neither fairness nor justice, save what we grant each other.
>> I thought I'd talk about my new project here, since there is a good >> chance that someone might be interested in it. I'm planning to put >> Ruby on top of Smalltalk VMs.
> This does indeed sound promising. Having a self-hosted Ruby would have > certain benefits as well.
>> a wonderful debugger which will let programmers modify methods on the >> fly & continue execution
> That is already possible with ruby-breakpoint which spawns an IRB > session at a specific place of your code. See > http://ruby-breakpoint.rubyforge.org/ if you are interested in this > sort > of things.
> It sure can't hurt to have more options, though.
In the Smalltalk debuggers, you can spawn off as many debugger sessions as you want by highlighting arbitrary code in the debugger as well as any browser window, as well as visually "inspect" any object (with each one appearing in a window), with the context of every window sensitive to the class being displayed. The effect is basically an "irb-anywhere" which you can arbitrarily interleave with any number of visual inspects. That, and a self-hosted language, faster VM, more garbage collection options, etc...
-- There's neither heaven nor hell, save what we grant ourselves. There's neither fairness nor justice, save what we grant each other.
PS> to the class being displayed. The effect is basically an PS> "irb-anywhere" which you can arbitrarily interleave with any number of
So it can crash anywhere :-) SCNR.
Yes maybe its possible to transfer part of the language (all?) into an smalltalk VM. But you can never get all the binary extensions to work, an emulation layer would kill all your speed benefits if possible at all. So maybe you should simply announce a ruby branch called SmallRuby, since this would not have so much todo with the current ruby anymore.
But i'm courious whats your result in the next years. Is this your Ph.D thesis ?
-- Best regards, emailto: scholz at scriptolutions dot com Lothar Scholz http://www.ruby-ide.com CTO Scriptolutions Ruby, PHP, Python IDE 's
On Fri, 08 Apr 2005, Lothar Scholz defenestrated me:
> Yes maybe its possible to transfer part of the language (all?) into an > smalltalk VM. But you can never get all the binary extensions to work, > an emulation layer would kill all your speed benefits if possible > at all. So maybe you should simply announce a ruby branch called SmallRuby, > since this would not have so much todo with the current ruby anymore.
OTOH, if it is 30x faster and makes an alternate implementation of socket (which likely would not have any speed deficiency) and a few other extensions it may become an attractive branch to use for many people.
Seems like an idea worth pursuing to me...Proofs in the pudding though. It will be interesting to see what gains materialize...
-Tom
-- + http://www.tc.umn.edu/~enebo +---- mailto:en...@acm.org ----+ | Thomas E Enebo, Protagonist | "A word is worth a thousand | | | pictures" -Bruce Tognazzini |
> Yes maybe its possible to transfer part of the language (all?) into an > smalltalk VM. But you can never get all the binary extensions to work, > an emulation layer would kill all your speed benefits if possible > at all. So maybe you should simply announce a ruby branch called > SmallRuby,
Wow, could you possibly be more negative?
This kind of dismissive overlording behavior does not reflect the usual spirit of acceptance and togetherness found on this list, especially since so many projects of the same spirit are in progress at the same time (YARV, Rubidium, JRuby, MetaRuby and a few less-publicized ones).
This is a time for offering constructive criticism and helpful insight into the difficult problems ahead, not a time to say things are impossible!
> PS> to the class being displayed. The effect is basically an > PS> "irb-anywhere" which you can arbitrarily interleave with any > number of
> So it can crash anywhere :-) SCNR.
In general this sort of thing doesn't crash under Smalltalk. In fact, the debugger is just a normal application doing a few funky things with Processes. It's just a browser for the reified context stack. In fact, you can write your own feature-poor, but working debugger in Smalltalk and have it running in a matter or minutes.
> Yes maybe its possible to transfer part of the language (all?) into an > smalltalk VM.
Not what I'm trying to do. The Smalltalk VM has basically *none* of the *language* in it.
> But you can never get all the binary extensions to work,
Why not? There are plenty of binary extensions to the various Smalltalks. They are designed to make it easy to build in binary extensions.
> an emulation layer would kill all your speed benefits if possible > at all. So maybe you should simply announce a ruby branch called > SmallRuby, > since this would not have so much todo with the current ruby anymore.
You missed something in my previous posts, or do not have some pieces of background concerning how Smalltalk works. There is no emulation going on, except stepping during debugging, and that happens at the level of the bytecodes so all VM/Object semantics are preserved. Ruby will be running on top of a Just-In-Time compiler, translating the bytecodes into native machine instructions. The VisualWorks JIT has been in development since 1989, so is one of the most mature JIT interpreters around. Its speed rivals Hotspot in real-world situations where GC is significant. Object allocation will be native -- Objects will just be Objects to the VM/garbage collector, with the Ruby/Smalltalk implementation language being moot.
> But i'm courious whats your result in the next years. > Is this your Ph.D thesis ?
No. I want to make the world safe for Pure-OO. A way I can help Pure-OO development is to vastly increase the power of the Ruby community by giving it access to the great technology developed for Smalltalk.
Other things that putting Ruby on top of Smalltalk VMs could garner for the Ruby community:
The Smalltalk MT environment can produce native Windows DLLs that are indistinguishable from C++ DLLs. We would theoretically be able to do the same for Ruby when we can host Ruby on that VM. There could be a Gemstone Ruby -- with a name like that there *should* be -- giving the community access to a great OODB. There is also a fast Smalltalk VM for .NET which could be used for Ruby. If we can overcome the lack of real Namespaces in Squeak, then we'd have a Ruby VM that runs completely bit-identical on 32 platforms.
Also, I'd like to do automated syntax + type-inference Ruby <--> Smalltalk translations so that Ruby can benefit from the wealth of great stuff done in Smalltalk, and Smalltalkers can be invigorated by contact with a larger, very active community.
--Peter
-- There's neither heaven nor hell, save what we grant ourselves. There's neither fairness nor justice, save what we grant each other.
> So, what are the potential platforms for Ruby on a Smalltalk VM?
Squeak alone has been ported to 32 platforms, running in all of them bit-identically, including Acorn Risc, WinCE, Mac OSX, Mac System 9, Windows, Linux, various Unixes. There is a Smalltalk VM for PalmOS. (slow, though) The first target, VisualWorks, runs on 8 platforms, including Various Windows, WinCE, Mac 8, Mac 9, Mac OSX, HPUX, AIX, IRIX, Solaris-SPARC, Linux x86, Linux SPARC, Linux PPC.
--Peter
-- There's neither heaven nor hell, save what we grant ourselves. There's neither fairness nor justice, save what we grant each other.
> I thought I'd talk about my new project here, since there is a good > chance that someone might be interested in it. I'm planning to put > Ruby on top of Smalltalk VMs.
Hi Peter,
I'm definitely interested in this; please keep me posted. Actually, a couple of people are aware that I was looking into the possibility of doing this as a commercial product - since, from my benchmarking, we can get about a 20x to 30x speed increase by hosting Ruby on a commercial Smalltalk VM like VisualWorks, it seemed like it might be worthwhile. However, the interest wasn't great enough for me to pursue it, so I'm very glad that someone is.
A small note: you say "at least those that have namespaces", but there's no need to be concerned about that. Namespaces are an image-level thing, not a VM-level thing; any Smalltalk VM worth its salt can support namespaces just fine (and in fact constructs like pool variables are effectively namespaces already, they just have somewhat different conventions than Ruby's).
>> Yes maybe its possible to transfer part of the language (all?) into an >> smalltalk VM. But you can never get all the binary extensions to work, >> an emulation layer would kill all your speed benefits if possible >> at all. So maybe you should simply announce a ruby branch called SmallRuby, >> since this would not have so much todo with the current ruby anymore.
> OTOH, if it is 30x faster and makes an alternate implementation of socket
>> I thought I'd talk about my new project here, since there is a good >> chance that someone might be interested in it. I'm planning to put >> Ruby on top of Smalltalk VMs.
>Hi Peter,
>I'm definitely interested in this; please keep me posted. Actually, a >couple of people are aware that I was looking into the possibility of >doing this as a commercial product - since, from my benchmarking, we >can get about a 20x to 30x speed increase by hosting Ruby on a >commercial Smalltalk VM like VisualWorks, it seemed like it might be
This "20x to 30x speed increase" stuff is interesting. I would have thought it would be closer to something like 5x. Can anyone elaborate on how those increases are being acheived in the SmallTalk VMs? I'm wondering if it might also be worthwhile to incorporate some of these ideas into YARV - parhaps that would get us to high-speed Ruby even faster than trying to put Ruby on top of a SmallTalk VM?
On Apr 7, 2005 10:04 PM, Phil Tomson <pt...@aracnet.com> wrote:
> In article <20050407220145.GI23...@garnet.tc.umn.edu>, > Thomas E Enebo <en...@acm.org> wrote: > >On Fri, 08 Apr 2005, Lothar Scholz defenestrated me:
> >> Yes maybe its possible to transfer part of the language (all?) into an > >> smalltalk VM. But you can never get all the binary extensions to work, > >> an emulation layer would kill all your speed benefits if possible > >> at all. So maybe you should simply announce a ruby branch called SmallRuby, > >> since this would not have so much todo with the current ruby anymore.
> > OTOH, if it is 30x faster and makes an alternate implementation of socket
> I'm very skeptical about the 30x faster claim.
I might be skeptical, but if I voice my skepticism loud enough, I may discourage those who might try (and may succeed).
So yeah... I think that 30x faster is quite possible ;-)
> > OTOH, if it is 30x faster and makes an alternate implementation of socket
> I'm very skeptical about the 30x faster claim.
> Phil
For what I've seen, reusing something like the Mono native backend (available for major processor architectures) can yield even higher performance gains. A custom tailored VM in assembler can provide 100x and possibly more depending on the architecture.
30x actually looks very conservative to me, but may well be worth the effort to reuse an already existing codebase.
>On Apr 7, 2005 10:04 PM, Phil Tomson <pt...@aracnet.com> wrote: >> In article <20050407220145.GI23...@garnet.tc.umn.edu>, >> Thomas E Enebo <en...@acm.org> wrote: >> >On Fri, 08 Apr 2005, Lothar Scholz defenestrated me:
>> >> Yes maybe its possible to transfer part of the language (all?) into an >> >> smalltalk VM. But you can never get all the binary extensions to work, >> >> an emulation layer would kill all your speed benefits if possible >> >> at all. So maybe you should simply announce a ruby branch called >SmallRuby, >> >> since this would not have so much todo with the current ruby anymore.
>> > OTOH, if it is 30x faster and makes an alternate implementation of socket
>> I'm very skeptical about the 30x faster claim.
>I might be skeptical, but if I voice my skepticism loud enough, I may >discourage those who might try (and may succeed).
>So yeah... I think that 30x faster is quite possible ;-)
Please, prove my skepticism to be wrong. A Ruby that was 30x faster (on the same hardware) would be great. As I said in the other post responding to Avi, if it's possible to speed things up that much with a SmallTalk VM, maybe we should be studying said VM to figure out how to make it happen in YARV (since matz has recently said that YARV is the future VM for Ruby).
> In article <67a2229205040719147fec0...@mail.gmail.com>, > Bill Guindon <agori...@gmail.com> wrote: > >On Apr 7, 2005 10:04 PM, Phil Tomson <pt...@aracnet.com> wrote: > >> In article <20050407220145.GI23...@garnet.tc.umn.edu>, > >> Thomas E Enebo <en...@acm.org> wrote: > >> >On Fri, 08 Apr 2005, Lothar Scholz defenestrated me:
> >> >> Yes maybe its possible to transfer part of the language (all?) into an > >> >> smalltalk VM. But you can never get all the binary extensions to work, > >> >> an emulation layer would kill all your speed benefits if possible > >> >> at all. So maybe you should simply announce a ruby branch called > >SmallRuby, > >> >> since this would not have so much todo with the current ruby anymore.
> >> > OTOH, if it is 30x faster and makes an alternate implementation of socket
> >> I'm very skeptical about the 30x faster claim.
> >I might be skeptical, but if I voice my skepticism loud enough, I may > >discourage those who might try (and may succeed).
> >So yeah... I think that 30x faster is quite possible ;-)
> Please, prove my skepticism to be wrong.
I leave that job to Peter, my job is to encourage him to try.
> A Ruby that was 30x faster (on > the same hardware) would be great. As I said in the other post > responding to Avi, if it's possible to speed things up that much with a > SmallTalk VM, maybe we should be studying said VM to figure out how to > make it happen in YARV (since matz has recently said that YARV is the > future VM for Ruby).
Agreed, and I hope Peter considers that, but if he's hell bent on building one on top of a SmallTalk VM, then I say go right ahead, and if it works, we can dissect that later.
