Re: [Q] Compilation of Sync Like Code

[Kris Kowal]

I’ve had it on the back of my mind (and have some notes squirreled
away in a textfile somewhere) about using Esprima to compile
JavaScript to JavaScript and producing a source map on the side. The
target file would use prototype chains for closures, have a flat name
space, and would transform generators into state machines.

But at this point it is a thought experiment. If it were possible to
complete the project to the satisfaction of a sizable group of users
before it became obsolete, it would bridge the gap and let us use the
existing Q.async method in engines apart from Firefox. I’m curious
whether it would perform on par with or better than the original text.

Kris Kowal

[Forbes Lindesay]

I've started to impliment it as QJS (https://github.com/ForbesLindesay/QJS) at the moment it provides a sudo function called `await` that converts a Q promise into the value it gets resolved to.  It works pretty well (even handling try catch blocks.  At the moment I'm in the process of transforming it to use `yield` as the function name rather than await.  At the moment it also automatically wraps your function with `async` and I want to change it so users have to do that explicitly.  One noteable advantage it has is that we can safely resolve promises in the same turn of the event loop using a `fastWhen` method.

I'd love contributions/help with this.  Especially if someone could make stack trqces look nicer.

[Mark Miller]

On Tue, Sep 4, 2012 at 11:02 AM, Forbes Lindesay <for...@lindesay.co.uk> wrote:

I've started to impliment it as QJS (https://github.com/ForbesLindesay/QJS) at the moment it provides a sudo function called `await` that converts a Q promise into the value it gets resolved to.  It works pretty well (even handling try catch blocks.  At the moment I'm in the process of transforming it to use `yield` as the function name rather than await.  At the moment it also automatically wraps your function with `async` and I want to change it so users have to do that explicitly.  One noteable advantage it has is that we can safely resolve promises in the same turn of the event loop using a `fastWhen` method.

What is the semantics of fastWhen? What "safety" do you mean? From the name and brief description above, I worry that it has the same problems as the old Q.asap.

 

I'd love contributions/help with this.  Especially if someone could make stack trqces look nicer.

On Tuesday, September 4, 2012 7:46:05 AM UTC+1, Kris Kowal wrote:

I’ve had it on the back of my mind (and have some notes squirreled
away in a textfile somewhere) about using Esprima to compile
JavaScript to JavaScript and producing a source map on the side.  The
target file would use prototype chains for closures, have a flat name
space, and would transform generators into state machines.

But at this point it is a thought experiment. If it were possible to
complete the project to the satisfaction of a sizable group of users
before it became obsolete, it would bridge the gap and let us use the
existing Q.async method in engines apart from Firefox. I’m curious
whether it would perform on par with or better than the original text.

Kris Kowal

--
Text by me above is hereby placed in the public domain

  Cheers,
  --MarkM

[Forbes Lindesay]

fastWhen looks like:

function fastWhen(promise, callback) {
  if (Q.isFulfilled(promise)) {
    return callback(null,  Q.nearer(promise));
  } else  {
    return promise.then(function (value) {
      return callback(null, value);
    }, function (err) {
      return callback(err);
    });
  }
}

The two key problems with doing that in the main library are that:

 a) A function could sometimes be called sooner than expected because it could get called syncronously:

function go() {

  var res = getAsyncValue().then(function (foo) {

    return foo + bar;

  });

  var bar = 'bar';

  return res;

}

If you call that with then implemented using then it is safe, because bar will always be declared before the then callback is called.  With fastWhen it could get called too soon, so that would be a bad time to use fastWhen.

Using QJS you might try and write:

var go = async(function () {

  var res = yield(getAsyncValue()) + bar;

  var bar = 'bar';

  return res;

});

This won't work with fastWhen or normal When because bar won't have been declared before you attempt to use it, but that's obvious and expected from reading the code.  You'd need to refactor it to:

var go = async(function () {

  var bar = 'bar';

  var res = yield(getAsyncValue()) + bar;

  return res;

});

Which works fine with fastWhen.

b) It could cause stack overflow if you keep yielding on lots of already resolved promises:

This is a little trickier to reason about, fastWhen, as implemented above won't work properly, because it will add a layer or two to the stack each time you yield on an already resolved promises.  This is only a problem when you have a lot of iterations though, and it can actually be fixed by more subtly including the behaviour of fastWhen in the implementation of Async such that it doesn't add any layers to the stack when a promise is already resolved.

If there are other problems with asap/fastWhen then let me know, as I don't want to be sacrificing safety for performance.

[John J Barton]

On Fri, Aug 10, 2012 at 6:51 PM, Forbes Lindesay <for...@lindesay.co.uk> wrote:
> I wanted to have a go at producing something to compile es.next style yield
> type code into something that would run now, because I can't stand waiting
> for that all important yield keyword. I ended up using await rather than
> yield, and then automatically inserting the Q.async equivalent whenever
> await was used in a function. I'd be really interested to hear feedback
> from anyone who may have attempted to do something similar for Q. My
> libraries at https://github.com/ForbesLindesay/QJS and can be installed
> from npm as qjs. If I could get stack traces to be prettier it really
> wouldn't be too bad. My thinking is that I could create a very simple
> implementation of it to produce yield based code when that starts being
> supported.

You might also look at how traceur implements await and yield:
http://code.google.com/p/traceur-compiler/
jjb

[Forbes Lindesay]

I have looked at how they implement them a fair bit for inspiration.  They break the flow of functions by turning them into switch statements with a series of steps.  That's the essential point that makes it viable to attempt this, since it lets you deal with each type of statement on a more or less case by case (no pun intended) basis.  Working from the most nested nodes up is the other brainwave that makes all this work (I thank falafel by substack for that inspiration)

Unfortunately the traceur yield implementation is out of date, so not useful for async and await is also out of date, so relying on it is a bad idea.  I chose not to fully implement yield, but rather to implement something that was close enough to make resolving promises achievable.

[Mark Miller]

On Wed, Sep 5, 2012 at 11:14 AM, Forbes Lindesay <for...@lindesay.co.uk> wrote:

fastWhen looks like:

function fastWhen(promise, callback) {
  if (Q.isFulfilled(promise)) {

    return callback(null,  Q.nearer(promise));
  } else  {
    return promise.then(function (value) {
      return callback(null, value);

    }, function (err) {
      return callback(err);
    });
  }
}

That's what I was afraid of. This is exactly equivalent to Q.asap, and has all the same problems. I think it is a bad idea. It cannot be prevented of course, since the primitives you use above are available, but it should be discouraged. It leads to unreliable programs. For a quick summary of the issue, see Section 3 and Table 1 at <http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/37626.pdf>.

fastWhen (or equivalently, Q.asap) by sometimes running synchronously and sometimes asynchronously, has neither of the advantages and both of the disadvantages listed there. In addition, programs will often be tested only in one of these two cases, leading to unexpected bugs when the other case occurs in the wild.

 

The two key problems with doing that in the main library are that:

 a) A function could sometimes be called sooner than expected because it could get called syncronously:

function go() {

  var res = getAsyncValue().then(function (foo) {

    return foo + bar;

  });

  var bar = 'bar';

  return res;

}

If you call that with then implemented using then it is safe, because bar will always be declared before the then callback is called.  With fastWhen it could get called too soon, so that would be a bad time to use fastWhen.

Exactly. The unexpected "too soon" disrupts assumptions of both the caller and the callee.

 

Using QJS you might try and write:

var go = async(function () {

  var res = yield(getAsyncValue()) + bar;

  var bar = 'bar';

  return res;

});

This won't work with fastWhen or normal When because bar won't have been declared before you attempt to use it, but that's obvious and expected from reading the code.  You'd need to refactor it to:

var go = async(function () {

  var bar = 'bar';

  var res = yield(getAsyncValue()) + bar;

  return res;

});

Which works fine with fastWhen.

The issue is the interleaving of the side effects after the yield point with sensitivity to side effects in the turn which would normally have completed before the yield returns. 

 

b) It could cause stack overflow if you keep yielding on lots of already resolved promises:

This is a little trickier to reason about, fastWhen, as implemented above won't work properly, because it will add a layer or two to the stack each time you yield on an already resolved promises.  This is only a problem when you have a lot of iterations though, and it can actually be fixed by more subtly including the behaviour of fastWhen in the implementation of Async such that it doesn't add any layers to the stack when a promise is already resolved.

If there are other problems with asap/fastWhen then let me know, as I don't want to be sacrificing safety for performance.

These problems with asap/fastWhen are adequate to classify it as sacrificing safety. I am glad you agree that we shouldn't sacrifice safety.

For more on plan interference hazards from such interleaving, see Part III of <http://erights.org/talks/thesis/markm-thesis.pdf>, especially chapters 13 and 14.