lazy.rb is a library providing transparent lazy evaluation and futures for Ruby. It provides a bag of clever tricks to help you avoid doing expensive computations up front.
= Lazy Evaluation
Lazy evaluation simply refers to computations which are run on an as-needed basis. For example:
x = promise { 3 + 5 }
Means that the block -- 3 + 5 -- won't actually be evaluated until something tries to use x's value.
p x # => #<Lazy::Promise computation=#<Proc:...>>
# forces evaluation p x * 3 # => 24
p x # => 8
You can also force evaluation using demand:
x = promise { 3 + 5 }
p x # => #<Lazy::Promise computation=#<Proc:...>>
# forces evaluation p demand( x ) # => 8
p x # => 8
It's a bit silly for 3 + 5, but it's handy for more intensive calculations. You can unconditionally promise a computation, yet only pay for it if and when its result is actually used.
= Futures
Futures are blocks of code that are evaluated immediately, but in a background thread.
x = future { 3 + 5 }
p x # => #<Lazy::Future computation=#<Proc:...>>
# You could do other stuff here while # the computation ran in the background
# blocks until the background thread completes p x * 3 # => 24
p x # => 8
Again, silly for 3 + 5 perhaps, but I'm sure you can see how this might come in handy for more involved computations.
= Other stuff
lazy.rb also includes support for circular programming, where a computation is passed its own result:
p matryoshka.first.first.object_id # => -605506544
This works for both promises and futures, although it has the usual limitations: if a computation tries to call methods on its own result, it will diverge.
== What's new in 0.9.5?
- Optional support for multithreaded programs:
require 'lazy/threadsafe' and you can safely use lazy evaluation in multithreaded programs.
- Futures:
With thread support, it turned out that futures were really easy to implement atop promises -- just fire off a thread with the computation and return a promise to join the thread and grab its result. So I implemented that.
== What happened to lazy streams from 0.2?
I ditched the lazy streams API for now. It just wasn't working out.
== What next?
Except perhaps for lazy streams (which might end up becoming a separate library), I think we're nearly feature-complete. Ideas and suggestions are very welcome, though.
> If you suspected "the right thing" you'd probably be right:
> irb(main):001:0> require 'lazy' > irb(main):002:0> x = promise { 5 + 3 } > => #<Lazy::Promise computation=#<Proc:0x0035788c@(irb):2>> > irb(main):003:0> p 3 * x > 24 > => nil
It's worth noting that this only works so well because of the very nice coercion machinery Ruby has built around its numeric classes.
Ruby's NUM2INT (for example) will call .to_int on the promise when FIXNUM_P fails, so everything Just Works(tm).
For other cases (e.g. promises returning file handles), you may find you need to unwrap them explicitly with demand() to pass them to certain methods.
Of course this is an issue only when passing promises to methods implemented in C; lazy.rb does a very good job of faking out Ruby otherwise (the boolean issue notwithstanding).
Quoting James Edward Gray II <ja...@grayproductions.net>:
> > I ditched the lazy streams API for now. It just wasn't working > > out.
> Can you expand on what you mean by this? I'm working on a huge > article about infinite streams for my blog and lazy.rb 0.2 was a > big inspiration to me.
Oh, mainly it was an aesthetic thing. It _worked_ fine.
I'd been fighting with the streams API to make it more Ruby-esque and easier to use properly, but I finally punted on it to get this release out the door quickly.
In retrospect, I probably should have just kept the 0.2 API for now. I definitely want something better for 1.0, though.
Hah! Awesome mental, I was going to be implementing something like this for an Actor library that I'm going to start work on Real Soon (tm). And, of course, you did it way better than I would have.
One question: How hard would it be to modify the way that exceptions are handled to hold off raising the exception until the result was requested? I'm thinking specifically of some non-deterministic situations where you may request a value but never end up needing to use it.
On 2/20/06, men...@rydia.net <men...@rydia.net> wrote:
> Quoting James Edward Gray II <ja...@grayproductions.net>:
> > > I ditched the lazy streams API for now. It just wasn't working > > > out.
> > Can you expand on what you mean by this? I'm working on a huge > > article about infinite streams for my blog and lazy.rb 0.2 was a > > big inspiration to me.
> Oh, mainly it was an aesthetic thing. It _worked_ fine.
> I'd been fighting with the streams API to make it more Ruby-esque > and easier to use properly, but I finally punted on it to get this > release out the door quickly.
> In retrospect, I probably should have just kept the 0.2 API for now. > I definitely want something better for 1.0, though.
Jim Weirich pointed out that methods later added to Object or Kernel will no longer be "missing" in the Async (or Future) class. One work around is to trap "method_added" up the chain, for example:
But then I discovered that methods added by including a module weren't trapped by method_added -- so in the end, a KernellessObject (from evil.rb) was needed (same link as above) as Async's parent to keep the proxy clean.
regards, andrew
-- Andrew L. Johnson http://www.siaris.net/ What have you done to the cat? It looks half-dead. -- Schroedinger's wife
[root@poseidon tmp]# ll 합계 12 -rw-r--r-- 1 mkseo users 6656 2월 19 09:31 lazy-0.9.5.gem [root@poseidon tmp]# gem install lazy-0.9.5.gem Attempting local installation of 'lazy-0.9.5.gem' Successfully installed lazy, version 0.9.5 Installing RDoc documentation for lazy-0.9.5...
lazy.rb:60:22: Couldn't find DIVERGES. Assuming it's a module [root@poseidon tmp]# irb irb(main):001:0> require 'lazy' LoadError: no such file to load -- lazy from (irb):1:in `require' from (irb):1 irb(main):002:0> require 'lazy/future' LoadError: no such file to load -- lazy/future from (irb):2:in `require' from (irb):2 irb(main):003:0>
[root@poseidon tmp]# ls -l /usr/local/lib/ruby/gems/1.8/gems/ 합계 72 drwxr-xr-x 4 root root 4096 2월 12 07:32 actionmailer-1.1.5/ drwxr-xr-x 5 root root 4096 2월 12 07:32 actionpack-1.11.2/ drwxr-xr-x 5 root root 4096 2월 12 07:32 actionwebservice-1.0.0/ drwxr-xr-x 5 root root 4096 2월 12 07:31 activerecord-1.13.2/ drwxr-xr-x 3 root root 4096 2월 12 07:31 activesupport-1.2.5/ drwxr-xr-x 3 root root 4096 2월 22 00:42 lazy-0.9.5/ drwxr-xr-x 11 root root 4096 2월 12 07:32 rails-1.0.0/ drwxr-xr-x 6 root root 4096 2월 12 07:31 rake-0.7.0/ drwxr-xr-x 3 root root 4096 2월 12 07:30 sources-0.0.1/
[root@poseidon tmp]# ls -l /usr/local/lib/ruby/1.8/ | grep lazy [root@poseidon tmp]#
I've taken the liberty of posting installation problem to this post, because this is the first time for me to install local *.gem file. Can anybody tell me why this is happening? Isn't the lazy.rb file supposed to be installed in /usr/local/lib/ruby/1.8 ?
On Wed, 2006-02-22 at 00:58 +0900, Minkoo Seo wrote: > Still, no luck.
> [root@poseidon tmp]# ll > 합계 12 > -rw-r--r-- 1 mkseo users 6656 2월 19 09:31 lazy-0.9.5.gem > [root@poseidon tmp]# gem install lazy-0.9.5.gem > Attempting local installation of 'lazy-0.9.5.gem' > Successfully installed lazy, version 0.9.5 > Installing RDoc documentation for lazy-0.9.5...
> lazy.rb:60:22: Couldn't find DIVERGES. Assuming it's a module > [root@poseidon tmp]# irb > irb(main):001:0> require 'lazy' > LoadError: no such file to load -- lazy > from (irb):1:in `require' > from (irb):1 > irb(main):002:0> require 'lazy/future' > LoadError: no such file to load -- lazy/future > from (irb):2:in `require' > from (irb):2 > irb(main):003:0>
Haven't followed this thread all the way, but I'm assuming you must have already tried:
$ irb require 'rubygems' require 'lazy'
Or alternatively passing -rubygems as an option to IRB.
> [root@poseidon tmp]# ll > 합계 12 > -rw-r--r-- 1 mkseo users 6656 2월 19 09:31 lazy-0.9.5.gem > [root@poseidon tmp]# gem install lazy-0.9.5.gem > Attempting local installation of 'lazy-0.9.5.gem' > Successfully installed lazy, version 0.9.5 > Installing RDoc documentation for lazy-0.9.5...
> lazy.rb:60:22: Couldn't find DIVERGES. Assuming it's a module > [root@poseidon tmp]# irb > irb(main):001:0> require 'lazy' > LoadError: no such file to load -- lazy > from (irb):1:in `require' > from (irb):1 > irb(main):002:0> require 'lazy/future' > LoadError: no such file to load -- lazy/future > from (irb):2:in `require' > from (irb):2 > irb(main):003:0>
> [root@poseidon tmp]# ls -l /usr/local/lib/ruby/gems/1.8/gems/ > 합계 72 > drwxr-xr-x 4 root root 4096 2월 12 07:32 actionmailer-1.1.5/ > drwxr-xr-x 5 root root 4096 2월 12 07:32 actionpack-1.11.2/ > drwxr-xr-x 5 root root 4096 2월 12 07:32 actionwebservice-1.0.0/ > drwxr-xr-x 5 root root 4096 2월 12 07:31 activerecord-1.13.2/ > drwxr-xr-x 3 root root 4096 2월 12 07:31 activesupport-1.2.5/ > drwxr-xr-x 3 root root 4096 2월 22 00:42 lazy-0.9.5/ > drwxr-xr-x 11 root root 4096 2월 12 07:32 rails-1.0.0/ > drwxr-xr-x 6 root root 4096 2월 12 07:31 rake-0.7.0/ > drwxr-xr-x 3 root root 4096 2월 12 07:30 sources-0.0.1/
> I've taken the liberty of posting installation problem to this post, > because this is the first time for me to install local *.gem file. > Can anybody tell me why this is happening? Isn't the lazy.rb file > supposed to be installed in /usr/local/lib/ruby/1.8 ?
On Tue, 2006-02-21 at 16:00 +0900, Daniel Nugent wrote: > One question: How hard would it be to modify the way that exceptions > are handled to hold off raising the exception until the result was > requested? I'm thinking specifically of some non-deterministic > situations where you may request a value but never end up needing to > use it.
Hmm, if it doesn't already do that, it's a bug.
Can you give me a test case which demonstrates the problem?
> Jim Weirich pointed out that methods later added to Object or Kernel will > no longer be "missing" in the Async (or Future) class. One work around is > to trap "method_added" up the chain, for example:
> But then I discovered that methods added by including a module weren't > trapped by method_added -- so in the end, a KernellessObject (from evil.rb) > was needed (same link as above) as Async's parent to keep the proxy clean
On Tue, 2006-02-21 at 21:39 +0900, Jim Weirich wrote: > Its just that RDoc is complaining about something. I got the same error > on my system, but the RDoc looks ok, even with the error.
DIVERGES is a nodoc'd constant which is used internally. If anyone can find a way to avoid the RDoc warning, I'd really appreciate it...
>> Jim Weirich pointed out that methods later added to Object or >> Kernel will >> no longer be "missing" in the Async (or Future) class. One work >> around is >> to trap "method_added" up the chain, for example:
>> But then I discovered that methods added by including a module >> weren't >> trapped by method_added -- so in the end, a KernellessObject (from >> evil.rb) >> was needed (same link as above) as Async's parent to keep the >> proxy clean.
> Hmm, good catch. Thanks!
> I wonder if it's worth introducing a dependency on evil.rb?
Oh, I don't think so. It would cut off some users, like me. ;)
On Wed, 22 Feb 2006 03:35:03 +0900, James Edward Gray II
<ja...@grayproductions.net> wrote: > On Feb 21, 2006, at 12:29 PM, MenTaLguY wrote:
>> I wonder if it's worth introducing a dependency on evil.rb?
> Oh, I don't think so. It would cut off some users, like me. ;)
At one point I was going to try to extract a minimum subset of evil to just allow for KernellessObject, but never got to it. Would such a small_evil.rb be less of a dependency concern?
andrew
-- Andrew L. Johnson http://www.siaris.net/ Doing linear scans over an associative array is like trying to club someone to death with a loaded Uzi. -- Larry Wall
MenTaLguY wrote: > On Tue, 2006-02-21 at 17:33 +0900, Andrew Johnson wrote:
>> But then I discovered that methods added by including a module weren't >> trapped by method_added -- so in the end, a KernellessObject (from evil.rb) >> was needed (same link as above) as Async's parent to keep the proxy clean.
> Hmm, good catch. Thanks!
> I wonder if it's worth introducing a dependency on evil.rb?
The BlankSlate class in Builder handles this without resorting to the "evil" that lies in the heart of evil.rb. And the CVS head version of BlankSlate also handles the module hole Andrew mentioned earlier (I think ... I just now updated it).4
On Wed, 22 Feb 2006 06:49:11 +0900, Jim Weirich <j...@weirichhouse.org> wrote: > MenTaLguY wrote: [snip] >> I wonder if it's worth introducing a dependency on evil.rb?
> The BlankSlate class in Builder handles this without resorting to the > "evil" that lies in the heart of evil.rb. And the CVS head version of > BlankSlate also handles the module hole Andrew mentioned earlier (I > think ... I just now updated it).4
That does appear to plug it -- and I even recall looking at append_features back then and not seeing it. Thanks Jim!
andrew
-- Andrew L. Johnson http://www.siaris.net/ It's kinda hard trying to remember Perl syntax *and* Occam's razor at the same time :-) -- Graham Patterson
On Wed, 2006-02-22 at 04:01 +0900, Daniel Nugent wrote: > In that case, what I want (I'm so damn greedy :D) is to be able to > raise the exception immediately (in a case where you damned well know > that you're going to need the value and would prefer to fail early) > and I see that that's trivial to implement (just have a conditional > toss the exception up in the thread if some @raise_immediately flag is > set to true). > There's one other thing I was wondering about: What's the prudency of > adding a method to Future to allow the Future to go back to sleep > (release the lock, pass the thread)? In this case, I'm considering > waiting for a resource that some other future/thread might be using, > or waiting for a port to get some data. (mostly implementation details > for that library I mentioned, I'll deal with them, but I'm curious if > there's any issue I'm forgetting).
Hmm. Could you give me some examples of the behavior you have in mind?
On Wed, 2006-02-22 at 09:07 +0900, Daniel Nugent wrote: > What's the purpouse of the spinlock in __synchronize__? > To keep a Future's block from evaluating several times in parallel?
s/Future/Promise/, and you've got it.
It ensures that each promise only ever gets evaluated once, even when multiple threads demand its result at the same time.
Nothing to do with futures in particular; I only require threadsafe for futures because using futures guarantees that your program will have multiple threads.
MenTaLguY wrote: > On Wed, 2006-02-22 at 06:49 +0900, Jim Weirich wrote: >> MenTaLguY wrote: >> The BlankSlate class in Builder handles this without resorting to the >> "evil" that lies in the heart of evil.rb. And the CVS head version of >> BlankSlate also handles the module hole Andrew mentioned earlier (I >> think ... I just now updated it).4
> Hmm, that sounds more like it. Any chance of a separate BlankSlate gem?
This is a quick breakout of the BlankSlate class into its own gem. It now sits as a top level namespace (instead of being nested in the Builder module). It is built from the same source as Builder, so the builder gem still includes the class physically. (Shouldn't be a problem unless there start to be weird version mismatches).
There are still some documentation issues (e.g. the blankslate gem RDoc still refers to the builder README file), but give this guy a spin around the block before I make an official release.
irb(main):004:0> require 'blankslate' LoadError: No such file to load -- builder from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:18:in `require__' from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:18:in `require' from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:163:in `activate' from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:23:in `require' from (irb):4
Seems that the dependency on builder is still there.
