[ANN] lazy.rb 0.9.5 -- transparent futures!
MenTaLguY
thread safety and transparent futures!
Here's the web site, complete with gem and tarball downloads, plus a bit
of documentation:
http://moonbase.rydia.net/software/lazy.rb/
Bug reports would be very, very welcome.
== What is lazy.rb?
lazy.rb is a library providing transparent lazy evaluation and futures
for Ruby. It provides a bag of clever tricks to help you avoid doing
expensive computations up front.
= Lazy Evaluation
Lazy evaluation simply refers to computations which are run on an
as-needed basis. For example:
x = promise { 3 + 5 }
Means that the block -- 3 + 5 -- won't actually be evaluated until
something tries to use x's value.
p x # => #<Lazy::Promise computation=#<Proc:...>>
# forces evaluation
p x * 3 # => 24
p x # => 8
You can also force evaluation using demand:
x = promise { 3 + 5 }
p x # => #<Lazy::Promise computation=#<Proc:...>>
# forces evaluation
p demand( x ) # => 8
p x # => 8
It's a bit silly for 3 + 5, but it's handy for more intensive
calculations. You can unconditionally promise a computation, yet only
pay for it if and when its result is actually used.
= Futures
Futures are blocks of code that are evaluated immediately, but in a
background thread.
x = future { 3 + 5 }
p x # => #<Lazy::Future computation=#<Proc:...>>
# You could do other stuff here while
# the computation ran in the background
# blocks until the background thread completes
p x * 3 # => 24
p x # => 8
Again, silly for 3 + 5 perhaps, but I'm sure you can see how this might
come in handy for more involved computations.
= Other stuff
lazy.rb also includes support for circular programming, where a
computation is passed its own result:
matryoshka = demand( promise { |result| [result] } )
p matryoshka # => [[...]]
p matryoshka.object_id # => -605506544
p matryoshka.first.object_id # => -605506544
p matryoshka.first.first.object_id # => -605506544
This works for both promises and futures, although it has the usual
limitations: if a computation tries to call methods on its own result,
it will diverge.
== What's new in 0.9.5?
- Optional support for multithreaded programs:
require 'lazy/threadsafe' and you can safely use lazy evaluation in
multithreaded programs.
- Futures:
With thread support, it turned out that futures were really
easy to implement atop promises -- just fire off a thread with the
computation and return a promise to join the thread and grab its
result. So I implemented that.
== What happened to lazy streams from 0.2?
I ditched the lazy streams API for now. It just wasn't working out.
== What next?
Except perhaps for lazy streams (which might end up becoming a separate
library), I think we're nearly feature-complete. Ideas and suggestions
are very welcome, though.
-mental
itsm...@hotmail.com
> I'd like to announce a new version of lazy.rb -- this one offering
> thread safety and transparent futures!
Nice.
> # forces evaluation
> p x * 3 # => 24
What will it do for:
p 3 * x
Thanks for sharing.
matt.s...@gmail.com
irb(main):001:0> require 'lazy'
irb(main):002:0> x = promise { 5 + 3 }
=> #<Lazy::Promise computation=#<Proc:0x0035788c@(irb):2>>
irb(main):003:0> p 3 * x
24
=> nil
men...@rydia.net
It's worth noting that this only works so well because of the very
nice coercion machinery Ruby has built around its numeric classes.
Ruby's NUM2INT (for example) will call .to_int on the promise when
FIXNUM_P fails, so everything Just Works(tm).
For other cases (e.g. promises returning file handles), you may find
you need to unwrap them explicitly with demand() to pass them to
certain methods.
Of course this is an issue only when passing promises to methods
implemented in C; lazy.rb does a very good job of faking out Ruby
otherwise (the boolean issue notwithstanding).
-mental
men...@rydia.net
> > I ditched the lazy streams API for now. It just wasn't working
> > out.
>
> Can you expand on what you mean by this? I'm working on a huge
> article about infinite streams for my blog and lazy.rb 0.2 was a
> big inspiration to me.
Oh, mainly it was an aesthetic thing. It _worked_ fine.
I'd been fighting with the streams API to make it more Ruby-esque
and easier to use properly, but I finally punted on it to get this
release out the door quickly.
In retrospect, I probably should have just kept the 0.2 API for now.
I definitely want something better for 1.0, though.
-mental
Daniel Nugent
this for an Actor library that I'm going to start work on Real Soon
(tm). And, of course, you did it way better than I would have.
One question: How hard would it be to modify the way that exceptions
are handled to hold off raising the exception until the result was
requested? I'm thinking specifically of some non-deterministic
situations where you may request a value but never end up needing to
use it.
--
-Dan Nugent
Andrew Johnson
> --=-Nhoy1n60Sqvzvq/VUinT
> Content-Type: text/plain
> Content-Transfer-Encoding: quoted-printable
>
> I'd like to announce a new version of lazy.rb -- this one offering
> thread safety and transparent futures!
>
> Here's the web site, complete with gem and tarball downloads, plus a bit
> of documentation:
>
> http://moonbase.rydia.net/software/lazy.rb/
>
> Bug reports would be very, very welcome.
When I toyed with a proof-of-concept of Io's asynchronous messages in Ruby
http://www.siaris.net/index.cgi/Programming/LanguageBits/Ruby/Async.rdoc
Jim Weirich pointed out that methods later added to Object or Kernel will
no longer be "missing" in the Async (or Future) class. One work around is
to trap "method_added" up the chain, for example:
http://www.siaris.net/cgi-bin/siwiki.pl?FeedBack/AsyncMessages
But then I discovered that methods added by including a module weren't
trapped by method_added -- so in the end, a KernellessObject (from evil.rb)
was needed (same link as above) as Async's parent to keep the proxy clean.
regards,
andrew
--
Andrew L. Johnson http://www.siaris.net/
What have you done to the cat? It looks half-dead.
-- Schroedinger's wife
Minkoo Seo
[root@poseidon tmp]# ruby -v
ruby 1.8.4 (2005-12-24) [i686-linux]
[root@poseidon tmp]# ls -l lazy-0.9.5.gem
-rw-r--r-- 1 mkseo users 6656 2월 19 09:31 lazy-0.9.5.gem
[root@poseidon tmp]# gem install lazy-0.9.5.gem
Attempting local installation of 'lazy-0.9.5.gem'
Successfully installed lazy, version 0.9.5
Installing RDoc documentation for lazy-0.9.5...
lazy.rb:60:22: Couldn't find DIVERGES. Assuming it's a module
[root@poseidon tmp]#
Any idea?
Minkoo Seo
Jim Weirich
> Sorry for newbie question. I tried to install lazy only to fail:
>
> [root@poseidon tmp]# ruby -v
> ruby 1.8.4 (2005-12-24) [i686-linux]
> [root@poseidon tmp]# ls -l lazy-0.9.5.gem
> -rw-r--r-- 1 mkseo users 6656 2월 19 09:31 lazy-0.9.5.gem
> [root@poseidon tmp]# gem install lazy-0.9.5.gem
> Attempting local installation of 'lazy-0.9.5.gem'
> Successfully installed lazy, version 0.9.5
The installatiof of the lazy software is OK ...
> Installing RDoc documentation for lazy-0.9.5...
>
> lazy.rb:60:22: Couldn't find DIVERGES. Assuming it's a module
Its just that RDoc is complaining about something. I got the same error
on my system, but the RDoc looks ok, even with the error.
--
-- Jim Weirich
--
Posted via http://www.ruby-forum.com/.
Minkoo Seo
[root@poseidon tmp]# ll
합계 12
-rw-r--r-- 1 mkseo users 6656 2월 19 09:31 lazy-0.9.5.gem
[root@poseidon tmp]# gem install lazy-0.9.5.gem
Attempting local installation of 'lazy-0.9.5.gem'
Successfully installed lazy, version 0.9.5
Installing RDoc documentation for lazy-0.9.5...
lazy.rb:60:22: Couldn't find DIVERGES. Assuming it's a module
[root@poseidon tmp]# irb
irb(main):001:0> require 'lazy'
LoadError: no such file to load -- lazy
from (irb):1:in `require'
from (irb):1
irb(main):002:0> require 'lazy/future'
LoadError: no such file to load -- lazy/future
from (irb):2:in `require'
from (irb):2
irb(main):003:0>
[root@poseidon tmp]# ls -l /usr/local/lib/ruby/gems/1.8/gems/
합계 72
drwxr-xr-x 4 root root 4096 2월 12 07:32 actionmailer-1.1.5/
drwxr-xr-x 5 root root 4096 2월 12 07:32 actionpack-1.11.2/
drwxr-xr-x 5 root root 4096 2월 12 07:32 actionwebservice-1.0.0/
drwxr-xr-x 5 root root 4096 2월 12 07:31 activerecord-1.13.2/
drwxr-xr-x 3 root root 4096 2월 12 07:31 activesupport-1.2.5/
drwxr-xr-x 3 root root 4096 2월 22 00:42 lazy-0.9.5/
drwxr-xr-x 11 root root 4096 2월 12 07:32 rails-1.0.0/
drwxr-xr-x 6 root root 4096 2월 12 07:31 rake-0.7.0/
drwxr-xr-x 3 root root 4096 2월 12 07:30 sources-0.0.1/
[root@poseidon tmp]# ls -l /usr/local/lib/ruby/1.8/ | grep lazy
[root@poseidon tmp]#
I've taken the liberty of posting installation problem to this post,
because this is the first time for me to install local *.gem file.
Can anybody tell me why this is happening? Isn't the lazy.rb file
supposed to be installed in /usr/local/lib/ruby/1.8 ?
- Minkoo Seo
Ross Bamford
> Still, no luck.
>
> [root@poseidon tmp]# ll
> 합계 12
> -rw-r--r-- 1 mkseo users 6656 2월 19 09:31 lazy-0.9.5.gem
> [root@poseidon tmp]# gem install lazy-0.9.5.gem
> Attempting local installation of 'lazy-0.9.5.gem'
> Successfully installed lazy, version 0.9.5
> Installing RDoc documentation for lazy-0.9.5...
>
> lazy.rb:60:22: Couldn't find DIVERGES. Assuming it's a module
> [root@poseidon tmp]# irb
> irb(main):001:0> require 'lazy'
> LoadError: no such file to load -- lazy
> from (irb):1:in `require'
> from (irb):1
> irb(main):002:0> require 'lazy/future'
> LoadError: no such file to load -- lazy/future
> from (irb):2:in `require'
> from (irb):2
> irb(main):003:0>
Haven't followed this thread all the way, but I'm assuming you must have
already tried:
$ irb
require 'rubygems'
require 'lazy'
Or alternatively passing -rubygems as an option to IRB.
--
Ross Bamford - ro...@roscopeco.REMOVE.co.uk
Logan Capaldo
require 'rubygems'
require 'lazy'
MenTaLguY
> One question: How hard would it be to modify the way that exceptions
> are handled to hold off raising the exception until the result was
> requested? I'm thinking specifically of some non-deterministic
> situations where you may request a value but never end up needing to
> use it.
Hmm, if it doesn't already do that, it's a bug.
Can you give me a test case which demonstrates the problem?
-mental
MenTaLguY
MenTaLguY
> Its just that RDoc is complaining about something. I got the same error
> on my system, but the RDoc looks ok, even with the error.
DIVERGES is a nodoc'd constant which is used internally. If anyone can
find a way to avoid the RDoc warning, I'd really appreciate it...
-mental
James Edward Gray II
>> proxy clean.
>
> Hmm, good catch. Thanks!
>
> I wonder if it's worth introducing a dependency on evil.rb?
Oh, I don't think so. It would cut off some users, like me. ;)
James Edward Gray II
Andrew Johnson
At one point I was going to try to extract a minimum subset of evil
to just allow for KernellessObject, but never got to it. Would such
a small_evil.rb be less of a dependency concern?
andrew
--
Andrew L. Johnson http://www.siaris.net/
Doing linear scans over an associative array is like
trying to club someone to death with a loaded Uzi.
-- Larry Wall
Jim Weirich
>
> Hmm, good catch. Thanks!
>
The BlankSlate class in Builder handles this without resorting to the
"evil" that lies in the heart of evil.rb. And the CVS head version of
BlankSlate also handles the module hole Andrew mentioned earlier (I
think ... I just now updated it).4
Andrew Johnson
> MenTaLguY wrote:
[snip]
>> I wonder if it's worth introducing a dependency on evil.rb?
>
> The BlankSlate class in Builder handles this without resorting to the
> "evil" that lies in the heart of evil.rb. And the CVS head version of
> BlankSlate also handles the module hole Andrew mentioned earlier (I
> think ... I just now updated it).4
That does appear to plug it -- and I even recall looking at
append_features back then and not seeing it. Thanks Jim!
andrew
--
Andrew L. Johnson http://www.siaris.net/
It's kinda hard trying to remember Perl syntax *and* Occam's
razor at the same time :-)
-- Graham Patterson
MenTaLguY
> In that case, what I want (I'm so damn greedy :D) is to be able to
> raise the exception immediately (in a case where you damned well know
> that you're going to need the value and would prefer to fail early)
> and I see that that's trivial to implement (just have a conditional
> toss the exception up in the thread if some @raise_immediately flag is
> set to true).
> There's one other thing I was wondering about: What's the prudency of
> adding a method to Future to allow the Future to go back to sleep
> (release the lock, pass the thread)? In this case, I'm considering
> waiting for a resource that some other future/thread might be using,
> or waiting for a port to get some data. (mostly implementation details
> for that library I mentioned, I'll deal with them, but I'm curious if
> there's any issue I'm forgetting).
Hmm. Could you give me some examples of the behavior you have in mind?
-mental
gabriele renzi
>> Hmm, good catch. Thanks!
>>
>> I wonder if it's worth introducing a dependency on evil.rb?
maybe just making it optional.. you could even make great use of
Object#become :)
MenTaLguY
> What's the purpouse of the spinlock in __synchronize__?
> To keep a Future's block from evaluating several times in parallel?
s/Future/Promise/, and you've got it.
It ensures that each promise only ever gets evaluated once, even when
multiple threads demand its result at the same time.
Nothing to do with futures in particular; I only require threadsafe for
futures because using futures guarantees that your program will have
multiple threads.
-mental
Jim Weirich
> On Wed, 2006-02-22 at 06:49 +0900, Jim Weirich wrote:
>> MenTaLguY wrote:
>> The BlankSlate class in Builder handles this without resorting to the
>> "evil" that lies in the heart of evil.rb. And the CVS head version of
>> BlankSlate also handles the module hole Andrew mentioned earlier (I
>> think ... I just now updated it).4
>
Ask and ye shall receive:
gem install blankslate --source http://onestepback.org/betagems
This is a quick breakout of the BlankSlate class into its own gem. It
now sits as a top level namespace (instead of being nested in the
Builder module). It is built from the same source as Builder, so the
builder gem still includes the class physically. (Shouldn't be a
problem unless there start to be weird version mismatches).
There are still some documentation issues (e.g. the blankslate gem RDoc
still refers to the builder README file), but give this guy a spin
around the block before I make an official release.
Guillaume Marcais
> gem install blankslate --source http://onestepback.org/betagems
irb(main):004:0> require 'blankslate'
LoadError: No such file to load -- builder
from
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:18:in
`require__'
from
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:18:in `require'
from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:163:in `activate'
from
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:23:in `require'
from (irb):4
Seems that the dependency on builder is still there.
Guillaume.
Jim Weirich
> Le 21 f�vr. 06, � 21:55, Jim Weirich a �crit :
>
>> gem install blankslate --source http://onestepback.org/betagems
>
> irb(main):004:0> require 'blankslate'
> LoadError: No such file to load -- builder
Dang! Autorequire bit me again.
Ok, I think I fixed it. Give it another try.
Daniel Sheppard
table_name = "CUSTOMERS"
x = promise { table_name }
puts x
=> "CUSTOMERS"
though you have to then make sure that you leave that table_name
variable alone.
table_name = "CUSTOMERS"
x = promise { table_name }
table_name = "MONKEYS"
puts x
=> "MONKEYS"
________________________________
From: Shashank Date [mailto:shank...@yahoo.com]
Sent: Thursday, 23 February 2006 9:15 AM
To: ruby-talk ML
Cc: men...@rydia.net
Subject: Re: [ANN] lazy.rb 0.9.5 -- transparent futures!
Hi,
I have a feature request (along with a quick-hack solution).
Please ignore if this is already done or sounds crazy. :-)
Problem: All the promises I make, cannot have any arguments ! !
So something like this is what I have in mind:
x = promise("CUSTOMERS") { |table_name|
long_running_query(table_name)}
Admitedly, in this one-liner it does not make much sense. But if
the block had many many lines of code and referred to the block variable
at multiple places, suddenly it would be mean more to me.
I have attached my hackish solution without really thinking all
the way thru.
Thanks,
-- Shashank
MenTaLguY <men...@rydia.net> wrote:
I'd like to announce a new version of lazy.rb -- this one
offering
thread safety and transparent futures!
Here's the web site, complete with gem and tarball downloads,
plus a bit
of documentation:
http://moonbase.rydia.net/software/lazy.rb/
Bug reports would be very, very welcome.
________________________________
Yahoo! Mail
Use Photomail
<http://pa.yahoo.com/*http://us.rd.yahoo.com/evt=38867/*http://photomail
mail.yahoo.com> to share photos without annoying attachments.
Daniel Nugent
x = promise {table_name.dup}
table_name = "MONKEYS"
puts x
Would avoid the problem. Though it's good to point out that you
likely want to avoid changing the state of any variable that's
captured by a promise closure (unless of course you plan on exploiting
just that).
> .mail.yahoo.com> to share photos without annoying attachments.
>
>
>
--
-Dan Nugent
Shashank Date
Daniel Nugent <nug...@gmail.com> wrote: table_name = "CUSTOMERS"
x = promise {table_name.dup}
table_name = "MONKEYS"
puts x
=>"CUSTOMERS"
Would avoid the problem. Though it's good to point out that you
likely want to avoid changing the state of any variable that's
captured by a promise closure (unless of course you plan on exploiting
just that).
All this being said, is there any merit in the idea of passing block parameters? I was toying with it briefly and decided to give it a shot on a whim. Will try and come up with a more convincing answer.
Thanks for all your inputs.
-- shanko
---------------------------------
Brings words and photos together (easily) with
PhotoMail - it's free and works with Yahoo! Mail.
Daniel Nugent
their references to the outside. You get exactly the same effect as
capturing the variables in a closure and in fact, the given examples
are kinda contrived, it would make more sense to assign CUSTOMERS as a
local variable inside of the promise block, because then it can't be
affected outside of the closure.
However, currently, there IS a block parameter (that of the promise
itself, something to do with circular programming, which sounds
intriguing, but I have hella no idea what it means [PLEASE SEND
LINKS]), but it doesn't seem like you'd use it in most cases.
If you wanted something like, I dunno, currying a promise, you could
just wrap it in a lambda.
--
-Dan Nugent
MenTaLguY
> table_name = "CUSTOMERS"
> x = promise {table_name.dup}
> table_name = "MONKEYS"
> puts x
> =>"CUSTOMERS"
>
> Would avoid the problem.
Actually, no. String#dup wouldn't get called until the promise was
demanded, by which time table_name refers to a different string.
-mental
MenTaLguY
> However, currently, there IS a block parameter (that of the promise
> itself, something to do with circular programming, which sounds
> intriguing, but I have hella no idea what it means [PLEASE SEND
> LINKS]), but it doesn't seem like you'd use it in most cases.
Here are a couple trivial examples...
Building a circular data structure (a linked list in this case):
def build_circular_list( *stuff )
promise { |result|
if stuff.empty?
nil
else
list = result
stuff.reverse.each { |thing|
list = [ thing, list ]
}
list
end
}
end
Padding lines of text to a maximum width in a single pass:
def pad_lines( lines )
promise { |result|
max_width = 0
padded_lines = []
lines.each { |line|
max_width = line.length if line.length > max_width
padded_lines.push promise { "%*s" % [ result[1], line ] }
}
[ padded_lines, max_width ]
}[0]
end
-mental
Jim Weirich
> On Wed, 2006-02-22 at 15:27 +0900, Jim Weirich wrote:
>
>> Ok, I think I fixed it. Give it another try.
>
> Invalid object explicitly tagged !ruby/Object: "requir"
Ahhh ... Interesting.
I've tracked this down to a corrupt compressed gem index on my beta
server. My home box wasn't getting it because it uses a unreleased
version of RubyGems that does incremental downloads of the individual
gem specs rather than grabbing the massive gem index all the time.
This might also explain an unrelated issue I was having with testing
RubyGems. I think this bug has made my day.
Now all I have to do is fix it.
Thanks.
Daniel Nugent
both table_name and name are pointing to "CUSTOMERS" and then only
table_name is pointing to "MONKEYS"
Yes, see, the thing that's important to consider is that I'm dumb.
*tries out Mental's examples, figures out how they work*
..
Woah, that's some powerful ju-ju. Maybe I should go and futz with
GhostWriter again anyhow...
On 2/23/06, Jim Weirich <j...@weirichhouse.org> wrote:
> Daniel Nugent wrote:
> > table_name = "CUSTOMERS"
> > x = promise {table_name.dup}
> > table_name = "MONKEYS"
> > puts x
> > =>"CUSTOMERS"
> >
> > Would avoid the problem. Though it's good to point out that you
> > likely want to avoid changing the state of any variable that's
> > captured by a promise closure (unless of course you plan on exploiting
> > just that).
>
> Hmmm ... I just tried it and got "MONKEYS" for the result. It makes
> sense becase by the time table_name is dup'ed in the promise, it has
> already been modified.
>
> This would do it:
>
> def table_promise(name)
> promise { name }
> end
>
> table_name = "CUSTOMERS"
> x = table_promise(table_name)
> table_name = "MONKEYS"
> puts x # => "CUSTOMERS"
>
> --
> -- Jim Weirich
>
>
>
> --
> Posted via http://www.ruby-forum.com/.
>
>
--
-Dan Nugent
Daniel Nugent
the Future initializer that accepts a block to be run inside the
rescue clause of the Future's thread?
The situation I am specifically thinking of is this:
I have a number of objects that will want the value resulting from the
Future. I know all of these objects WILL need to access that value.
Because the computation in the Future is very expensive, I would like
to raise exceptions on those objects as soon as the thread is caught
so that they can begin handling the error as soon as possible.
So, at the time of the exception, I can lock down the current
subscribers list and then fire off the exception to each of them.
Something like this:
future(lambda{|exception| lock.synchronize{subscribers.each{|s|
s.raise(exception)}}}){holy_crap_long_execution}
On 2/23/06, Jim Weirich <j...@weirichhouse.org> wrote:
> Yep, it was a bug in the new RubyGems server-side indexing software.
> Should be fixed now. Give it another try...
>
> gem install blankslate -s http://onestepback.org/betagems
>
> --
> -- Jim Weirich
>
>
>
> --
> Posted via http://www.ruby-forum.com/.
>
>
--
-Dan Nugent