Feedback on Async/flow/promise framework using Harmony Proxies

[Sean Hess]

Thanks to the node-proxy module, I was able to cook up a crazy idea
for a flow-control framework, where the promises are created
implicitly and you don't have to change any of your existing code. I
know everyone and their dog is making one of these, but AFAIK nobody
has tried to use proxies to make "magic" promises yet.

It's inspired in part by my recent work with Haskell, where the
language clearly separates IO code from pure functions. The idea is
that instead of doing the async code, create a series of instructions
that represent it, that can be passed continuations by the framework
instead of the developer.

I'd like general and syntactical feedback (the syntax of the wrapper
function is pretty arbitrary), as well as name suggestions :)

The following code works, and does not require an extra compilation
step or anything.

https://gist.github.com/1372013

It works by sending in proxy-promise versions of the modules you send
it, and the args, so whenever you call anything inside the block, it's
just creating a promise to call it, which is executed when the real
function is called.

Thoughts?

[Brandon Benvie]

Gozala's  https://github.com/Gozala/meta-promise does this though it needs to be updated. Proxies are fertile ground that is barely being touched so far but that'll probably change soon. It's just something that needs some clear, useful examples out in the wild before the awareness is there.

I'm working on various lower level interfaces for prroxies and planning to build out some cool demos based on them. There's some really cool tidbits that can be done with just a bit of code using the base API, but overall it's really "heavy" and requires library assistance to be useful for day to day problems.

Just a few days of work into it but already shows how powerful proxies can be.

https://github.com/Benvie/meta-objects 

[Sean Hess]

Hmm, yeah, similar tech, but very different approach.

I think though, the final syntax of this idea is pretty novel, and has
several advantages. I'll post a simpler example here to make it easy
to see:

var mc = require('theframework').makeCallback
var fs = require('fs')

var copyFile = mc(fs, function(fs, source, dest) {
var contents = fs.readFile(source)
fs.writeFile(dest, contents)
return true // = cb(err, true)
})

copyFile("source.txt", "dest.txt", function(err, success) {
console.log(err, success)
})

1. You don't have to do anything to adapt existing modules - just pass
them in. That "fs" there is the normal fs module. You don't have to
wrap it with any promise code.

2. It looks like the normal async code, just called async style.

3. It creates a normal, cb-style function. Any of your non-promise
code can call it.




On Nov 16, 7:08 pm, Brandon Benvie <brandon.ben...@gmail.com> wrote:
> Gozala's  https://github.com/Gozala/meta-promisedoes this though it needs

[Dh]

I find this idea very interesting.

Donald

[Adam Crabtree]

I built a Proxy-based flow-control library about a year ago if you're interested:

http://github.com/CrabDude/sexy

Cheers,

Adam Crabtree

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--
Better a little with righteousness
       than much gain with injustice.
Proverbs 16:8

[crypticswarm]

Sean,

Is the code of the framework up?

Seems like a neat idea, but most likely has the following limitations.

People would need to bring any function used into a separate the environment that uses or produces these proxy promises. (what you did with 'fs' in the example.)

So in the example you posted below you have parameter called fs that is presumably a proxy that proxies to the fs module, contents is probably also a proxy.  As long as you have something like this:

contents.whatever(...)
fs.readFile(...)

everything would work out, but if you pass one of these to a function that doesn't isn't a proxy then you will run into problems.

console.log(contents)  // probably prints [object Object]
var x = 'hello' + contents // probably sets x to 'hello[object Object]'

to me that seems like it could be easy to forget to put the usage examples like the above in the 'success' callback causing all sorts of headaches due to weird values.

[Sean Hess]

Just pushed it here - https://github.com/seanhess/node-async-framework-temp

You're right, you have to make sure to use wrapped versions of things
inside the block. I'm not sure I like it better, but an alternative to
passing fs in to the block that creates the function is to convert it
outside (using the name "as" as written in the source I pushed):

var as = require('as')

var fs = require('fs')

var fsc = as.convert(fs)

var copyFile = as(function(source, dest) {
var contents = fsc.readFile(source)
fsc.writeFile(dest, contents)
return true
})

If you forget, and just use the normal version of an asynchronous
functions, you'll usually get a "cb is not defined" error when it
tries to callback. I'd like to work on improving error messages as
much as possible, but it may be something you have to get used to.

console.log(contents) would show "{ readFile(source) }" (The dump of
the promise). However, if you happened to pass console.log or console
into the block, it would work as expected:

var copyFile = as(fs, console.log, function(fs, log, source, dest) {
var contents = fsc.readFile(source)
log(contents)
})

It's impossible to get '+' and other non-functional operators to work.
The good news is that I think this actually encourages good
separation. You have to define a function to do any heavy
manipulation. So, your async code is all about pulling resources
together and manipulating state, and you're forced to write pure
functions to do any fancy processing.

I got used to the limitations pretty quickly when coding the tests and
examples, but I'd definitely make them very clear in the
documentation.

@Adam - cool, I hadn't looked closely enough at $xy. It's pretty
similar, aside from a few design decisions. I'd love to dig in to your
source later.

[Brandon Benvie]

Well the idea behind harmony proxies is that you anything you 're doing with what you described can be mimicked transparently using Proxies, and you can even wrap over existing variables and functions if you're clever. You can even jury rig a sync call pretty easily by creating a function proxy that wraps some existing async function and kills time in between by running a sync fs.read call call to a self served socket.

[Douglas Martin]

I actually wrote something like this for comb, comb.executeInOrder (Could be named better).  Its in a beta stage right now but works pretty well.

https://github.com/doug-martin/comb/blob/v.0.0.5/lib/promise.js

[Jann Horn]

Can't you just add a custom toString property to promises?

> Is the code of th...

[Mariusz Nowak]

I really like the idea, I think this would be extremely useful when we'll have Harmony

I'm just not sure about such function wrappers, I think it should just work on promise object, the way Gozala suggested it in his meta-promise (https://github.com/Gozala/meta-promise )

[Brandon Benvie]

Well in terms of when Proxy and WeakMap are available normally, that's an unknown because ES6 isn't going to be finalized until 2013 and I wouldn't expect V8 to bring it out from behind the flag for at least 3-6 more months if not longer. But it's pretty trivial for usage in Node to make sure you have it right now. At most basic, you can just write this at the top of your module's index

if (typeof Proxy == 'undefined' || typeof WeakMap == 'undefined') {

  throw "This module requires node to be run with flags '--harmony_proxies --harmony_weakmaps'";

}

Or if you're clever you replace the error throw with running a new node process that reruns the same file but with flags and swaps over to that one's stdin so it happens transparently.

But basically there's very little preventing anyone using Node 0.6+ from using these tools right now, and I intend to try and use them to make some cool stuff.

[Sean Hess]

The node-proxy module works really well in node 4. You just npm
install it and go. I imagine it should be updatable to node 6.

I had some crazy ideas to help you if you forget to use an async
version. I could recommend that you proxy a module on import, and if
you're not in a promise-creation block, I can quickly resolve to the
real version. I'll give it a shot and see if it feels good.

@Mariusz, I'm not sure I understand. How would you change the syntax I
posted above?

[Brandon Benvie]

I don't recommend using node-proxy. It's re-implementing something as a Node module that now exists in core V8. It was nice before it existed in V8 but now it's a poor idea.

[Brandon Benvie]

To follow up, I found a comment on V8's tracker

Awesome! When would app developers be able to use proxies in V8 without flag?

We have no immediate plans for that. The feature has to be field-tested first. Since it touches so many corners of the implementation, there is lots of opportunities for fatal bugs. So we want to be confident that everything works as expected before turning it on for everybody.

[Mariusz Nowak]

On Saturday, November 19, 2011 12:44:51 AM UTC+1, Sean Hess wrote:

@Mariusz, I'm not sure I understand. How would you change the syntax I
posted above?

@Sean

Instead of fromPromises wrapper, I would rather use following concept: https://github.com/medikoo/deferred#asynchronous-functions-as-promises

this allows you 'convert' asynchronous functions into promises directly on the go, so you can write code this way:

a2p(fs.readdir, path).map(...)

You can make it even cleaner by configuring needed functions upfront:

var readdir = ba2p(fs.readdir)

// later in code:

readdir(path).map(...)

[Sean Hess]

@Brandon - Interesting. Is the node-proxy module not safe? I feel like
"npm install node-proxy" is less of a big deal than remembering to run
node with a flag, and I'm scared of those fatal bugs.

@Mariusz - Thanks! I need to digest this and the other frameworks, try
a few things and see how it ends up.

[Brandon Benvie]

node-proxy is a custom third party implementation of the logic for something that alters a JavaScript engine's internals at a very fundamental level. At that level it can and does have large consequences in terms of both performance and security. Since it's currently implemented (and continually improved, read the V8 release notes and issue tracker) in core V8 itself it's just an all around not useful and potentially harmful thing to do.