Serializing/storing closures
Jacob Rothstein
This is a totally naïve question coming from someone who doesn't
understand the inner workings of node/v8. Is it possible to serialize
a closure, including bound scope? For now, I'm not particularly
concerned with how compact the serialized representation is. Is it
possible?
For an example use case, I'd like to be able to take a function with
bound vars, run it through this magic serialization engine, throw it
into a queue, and have another worker on another machine deserialize
and execute it. Sound interesting?
Happy nodeing,
–Jacob
inimino
> For an example use case, I'd like to be able to take a function with
> bound vars,
Free variables?
> run it through this magic serialization engine, throw it
> into a queue, and have another worker on another machine deserialize
> and execute it. Sound interesting?
The reason to use a closure, as opposed to just a self-contained
JavaScript function, is that the closure can modify the state of
variables in the enclosing scope. Once you serialize it, that
presumably won't be the case anymore, so what's the benefit?
Jacob Rothstein
function along with it's execution context.
var outside = 'world'
, fn = function (argument) { return [argument, outside].join (' ') }
fs.writeFile ('serialized.fn', magicSerializer.serialize (fn))
---
and somewhere else with no shared memory,
fs.readFile ('serialized.fn', function (err, data) {
magicSerializer.deserialize (data) ('hello') //=> 'hello world'
})
Is that more clear?
–Jacob
> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>
Marak Squires
Scott Taylor
Just use .toString(), +, and eval().
Marak Squires
Ryan Gahl
Ben Noordhuis
> For an example use case, I'd like to be able to take a function with
> bound vars, run it through this magic serialization engine, throw it
> into a queue, and have another worker on another machine deserialize
> and execute it. Sound interesting?
Interesting question, yes. After giving it a few minutes thought, I
don't think there is a reliable way to do this. In the example below,
what would you expect to be actually serialized? I pained my mind but
couldn't come up with a good answer.
var foo = 'bar', bar = 'baz';
function x() { return window[foo]; } // should return 'baz'
Louis Santillan
> To get this working, in a generic sense, my guess is that it would be
> approximately a ridiculous amount of effort, and I can't imagine in my
> wildest dreams what benefits you'd receive vs marshalling relevant data to a
> well-defined service instead.
FYI, SpiderMonkey has this
(http://mxr.mozilla.org/mozilla-central/source/js/src/jsxdrapi.h).
-L
Dean Landolt
On Jun 8, 2010, at 4:48 PM, Marak Squires wrote:Just use .toString(), +, and eval().That won't save the evaluation context. Is there a native way to get it in node?
Mikeal Rogers
Trying to serialize the free variables is a leaky abstraction because
you won't be able to tell when they change, and even if you could
notify the clients of the change you'll hit aweful side effects when
clients are competing for setting the same state in the free space.
Bad, bad, bad idea. Don't do this.
-Mikeal
Jacob Rothstein
magically distributes execution across workers/machines without any
restrictions on how the code is written? I think that would be
ridiculously powerful, and perfect for node. In some circumstances
this could be better than forking, since it could cross machine
boundaries.
--
I understand that you wouldn't be able to track changes or make
changes back *into* the original context, but that's ok; it would be a
something like a snapshot of state. In fact, the intent would be to
execute the function with the free variable state at the time of
serialization, even if values have since changed.
I understand there would be things difficult/impossible to serialize,
like file handles & sockets. I must admit I expected those issues to
be what people brought up, but a few limitations like that don't seem
so horrible.
Is this really impossible, or is it just difficult/intimidating?
Thanks for all the feedback,
–Jacob
Timothy Caswell
It may even be possible to copy around file descriptors across processes (within the same machine of course).
Correa Diego
Dean Landolt
I find this a facinating idea, despite its spaghetti guideline.I guess, the fact of context serialization must be deterministically driven across the network for reliable re-execution in the context given.
Sami Samhuri
Erlang functionality. I'd love to have something like this, going to
have to experiment with json-ext.
-s
cwolves
dependencies into an object and serialize the object??
-Mark
Jacob Rothstein
using a redis-backed task queue that works exactly like that. One
type of process serializes and enqueues the name of the function to
call and any arguments and the other side loads up the file that
contains the function and apply()s it with the provided arguments.
Worker machines have a copy of the same code as the task-creation
machines.
This works great, but I think node could have an even slicker form of
distributed computing.
I believe serializing execution context would allow you to do this
with anonymous functions. You could wrap nearly any block of code in
your app with "distribute (function () { ... })" and it could be sent
to another process or machine. The worker machines wouldn't need to
know anything about your app, because they'd get anything they needed
along with the function.
Locally, forking/worker interfaces could take advantage of this and
instead of forking a particular predefined static file, they could
fork an anonymous function, much like ruby's Kernel.fork. Imagine
"fork (function() { ... })" just magically executing that code in a
subprocess.
–Jacob
jbaron
like Rhino has it right now.
Then you would be able to start, stop and serialize JavaScript
execution.
When I was using Rhino, there were a few people using it exactly as
you propose.
However the few times this topic came up on the v8 mailing list, the
answer was always no (
I guess this would also be very difficult to implement if you have a
lot of the optimizations like V8 does).
Peter
Bradley Meck
After a little digging, yes you can get them but I am not doing
something right as it is exiting silently, however, it requires v8-
debug.h. http://gist.github.com/434441
