Do you use promise progress events?

[Brian Cavalier]

All,

Do you use promise progress events (e.g. when.js's deferred.progress(..))?  They are one area of the Promises/A spec that is very open.  In fact, implementing them at all is entirely optional, and when.js's current implementation (<= v1.5.1) is pretty minimal.  For example, only progress handlers that are registered directly with a deferred's promise (not a promise further down a promise chain) will receive progress events issued via deferred.progress(...).

So, we'd love to hear if you're using them, be it with when.js or some other promise impl, and if so, how are you using them?  Could you provide a simple use case description, or better yet, some code snippets?

Also, here are some changes/improvements to progress events we are considering for the next release:

1. Should progress events propagate through a promise chain to downstream promises?

2. If so, will the upstream progress events actually have meaning to downstream progress event handlers?  For example, what if multiple promises in a chain are issuing progress events in different formats (e.g. string vs. object) ... does that mean every progress handler needs to inspect the progress event before trying to use it?

3. Should progress handlers be allowed to transform progress events by returning a value, in the same way that resolution handlers can?  This may solve the problem in #2.

4. If a progress handler throws an exception, should that stop propagation of the progress event?  If NOT, then what value should be propagated to the next promise in the chain?

5. Anything else we should be thinking about?

Let us know what you think so we can incorporate your feedback into the next release!

p.s. We're experimenting with answers to most of these in the dev branch, but we definitely want to hear feedback!  In dev, progress events will propagate, can be transformed, and exceptions stop downstream propagation, so feel free to take a look and try it out.

[Luke Smith]

1. Should progress events propagate through a promise chain to downstream promises?

No. Progress pertains to the execution time of the work necessary to resolve a promise.  Downstream promises are bound to the termination state of the promise, be it resolved or rejected. Cascading progress notifications downstream breaks the notion of discrete, decoupled chunks of work.

2. If so, will the upstream progress events actually have meaning to downstream progress event handlers?  For example, what if multiple promises in a chain are issuing progress events in different formats (e.g. string vs. object) ... does that mean every progress handler needs to inspect the progress event before trying to use it?

It should be the onus of the broadcaster of the progress events to use a consistent format for ease of consumption. Cascading progress events would violate that.

3. Should progress handlers be allowed to transform progress events by returning a value, in the same way that resolution handlers can?  This may solve the problem in #2.

Am I understanding the question correctly that if a promise has two resolution subs, the first can change the values passed to the second by returning an alternate value? Promise values should be immutable, right? (object property changes allowed by pass by ref, not withstanding)

I'm also curious about real life implementations using progress handlers.

[Mudi Ugbowanko]

Real life use case could be simply uploading a file:

1. Select file

2. Initialise promise

3. Start upload file via xhr/iframe

4. return promise to calling code

5. update progress and let the calling code update its GUI (or whatever) till the upload completes

Rather simple use case though.

[Stepan Riha]

In our web application, we use a UI progress handler to display progress notifications that are emitted by asynchronous tasks.  Some progress notifications are emitted by the application internals (e.g. "downloading file", etc.) and some by 3rd party application extensions (when they perform lengthy operations).

1. Should progress events propagate through a promise chain to downstream promises?

Yes. Our progress handler always sits at the very end of the promise chain.  Not propagating progress makes the feature either useless or terribly cumbersome (in that we would have to explicitly chain progress).

2. If so, will the upstream progress events actually have meaning to downstream progress event handlers?  For example, what if multiple promises in a chain are issuing progress events in different formats (e.g. string vs. object) ... does that mean every progress handler needs to inspect the progress event before trying to use it?

In our case, we use progress for a single purpose so the emitters and consumer agree on the format (we use strings) and there is no need for inspection.

3. Should progress handlers be allowed to transform progress events by returning a value, in the same way that resolution handlers can?  This may solve the problem in #2.

I think it makes sense to allow progress handlers to transform the progress events, which would parallel how promise values and errors are propagated.  I agree that this could be used to deal with #2, esp. if the progress handler has a way to suppress propagation (e.g. returning undefined or throwing an error).

4. If a progress handler throws an exception, should that stop propagation of the progress event?  If NOT, then what value should be propagated to the next promise in the chain?

I think throwing an exception might be a valid way to signal that propagation should be stopped.  If NOT, either the original progress notification or the exception error should be propagated.  I don't have strong feelings about this either way since our progress handler is always at the end of the promise chain.

5. Anything else we should be thinking about?

We would find it very convenient if promises maintained a progress notification state s.t. a newly subscribed progress consumer would immediately be notified with the most recent progress event iff a progress event had previously been issued on the promise.  This way a function returning a promise can easily issue a progress notification before anyone has had the chance to subscribe to it.  Otherwise you need to use setTimeout/nextTick before issuing to notification to allow the caller to subscribe.

Here's a more concrete example: https://github.com/cujojs/when/issues/58#issuecomment-9304644

[Brian Cavalier]

Thanks for jumping in, Luke.

For #1, I agree that one view of a promise chain is multiple steps to a final goal, and that the completion of each step (the resolution of each intermediate promise) can be considered as "progress" in and of itself.  It seems like there could also be situations where finer grained progress that propagates to the end of the chain could be useful.  One example would be an operation that fetches a resource, grabs a url out of that resource and uses it to fetch a second resource, e.g.:

return fetchFirstThing().then(extractUrlAndFetchSecondThing)

If the fetches are streaming and support progress updates, it might be useful for the caller to display some sort of overall progress.  That'd mean that progress events from the fetchFirstThing promise would need to propagate thru to the second promise (the one that's actually returned).

Of course, therein lies the problem of #2 ... those progress events from the fetchFirstThing promise will probably indicate progress from 0-100%.  So either they need to be transformed to events of 0-50% (or whatever), or the caller needs to be coded in a way that understands there are 2 sequential operations, each of which will produce progress events that range from 0-100%.  The latter seems like overly tight coupling to me.

Another way I could see to restructure that code would be something like:

var d = when.defer();

fetchFirstThing().then(fetchSecondThing, null, d.progress).then(d.resolve, d.reject, d.progress);

return d.promise;

Although you'd probably still want to wrap d.progress in a function that transforms the progress events.  Hmmm, now that I've written that code, it doesn't look as awful as I thought it would :)  Thoughts?

3. Should progress handlers be allowed to transform progress events by returning a value, in the same way that resolution handlers can?  This may solve the problem in #2.

Am I understanding the question correctly that if a promise has two resolution subs, the first can change the values passed to the second by returning an alternate value? Promise values should be immutable, right? (object property changes allowed by pass by ref, not withstanding)

Sorry for the poorly worded question.  Yes, promise values are immutable.  If a promise has two resolution handlers, both will always see the same value--neither is allowed to affect the other.  However, each handler can return a value that will be seen by the next promise in the respective chain.  So, I'm wondering if that transform-for-the-next-promise-in-the-chain behavior should be allowed for progress handlers, so that progress events can be transformed as they flow down a promise chain.

 

[Brian Cavalier]

Thanks for the info, Stepan.  I'd be interested to hear what you think about the example I gave in response to Luke--the one that uses an extra defer() to aggregate the progress from two (or more) promises.  Would that kind of thing work for your situations?

I'm still not sold on your last point about promises "caching" the latest progress event.  Progress events are essentially an event emitter pattern, and I don't know of any other event emitter that does that kind of caching.  It seems like it'd be possible to manage that kind of thing via a custom caching event emitter, which you'd hook up to you promise progress events.

That said, I certainly would love to hear what other think about it.

[Brian Cavalier]

Thanks, Mudi.  That does seem like a good (even if simple) use case.  Any thoughts on the questions from my original post?

[Stepan Riha]

On Friday, October 12, 2012 12:31:21 PM UTC-5, Brian Cavalier wrote:

Thanks for the info, Stepan.  I'd be interested to hear what you think about the example I gave in response to Luke--the one that uses an extra defer() to aggregate the progress from two (or more) promises.  Would that kind of thing work for your situations?

No, this only works for trivial situations where the progress emitter and consumer are within the scope of the same function.

We actually use this patter in some places for emitting the progress events (e.g. the downloadFiles function in this example), but the consumer is usually several levels removed down the promise chain.  In our case the intermediate promise handlers usually have no interest in the progress callbacks, so having to add intermediate deferreds just to propagate the progress would introduce a lot of complexity.

 

I'm still not sold on your last point about promises "caching" the latest progress event.  Progress events are essentially an event emitter pattern, and I don't know of any other event emitter that does that kind of caching.  It seems like it'd be possible to manage that kind of thing via a custom caching event emitter, which you'd hook up to you promise progress events.

Since we have a viable workaround, this isn't terribly important for us.  However, one could view the progress as a state pattern rather than a simple emitter pattern.

[Brian Cavalier]

On Friday, October 12, 2012 5:06:03 PM UTC-4, Stepan Riha wrote:

No, this only works for trivial situations where the progress emitter and consumer are within the scope of the same function.

Right.  For deep chaining across many function call levels, it becomes tedious to manually preserve the progress chain.

 

Since we have a viable workaround, this isn't terribly important for us.  However, one could view the progress as a state pattern rather than a simple emitter pattern.

Good point.  Typically, promises are described as having 3 possible states: pending, fulfilled, or rejected.  So maybe it's reasonable to think of the "current progress point" as sub-state along the path from pending to fulfilled/rejected.

[Brian Cavalier]

After reading the replies so far, and pondering a bit more, my current thinking is that there are 2 reasonable options:

1. Promises simply don't propagate progress to the next promise in the chain automatically.  Devs have to use some other mechanism (e.g. another event emitter, or whatever) to propagate progress.

2. Promises propagate progress automatically, and progress handlers are allowed to return a value that will be forwarded as the progress value to the next promise (similar to fulfill/reject value forwarding).  An exception in a progress handler should stop progress propagation, but*not* change the state of the associated or next promise (i.e. will not cause either to be rejected).  IMHO, there must also be some simple & reasonable way for a progress handler to explicitly stop progress propagation, such as returning undefined, which would be compatible with existing code that uses progress handlers on promises that don't allow propagation (but is also obviously different from how fulfill/reject handlers behave).

To me, allowing progress propagation *without* allowing progress handlers to return a new progress value or stop propagation seems like a recipe for lots of head scratching when someone's progress handlers start breaking because they receive a value they don't understand from a promise that is far upstream.

Thoughts?

[Domenic Denicola]

On Monday, October 15, 2012 7:46:54 AM UTC-4, Brian Cavalier wrote:

After reading the replies so far, and pondering a bit more, my current thinking is that there are 2 reasonable options:

1. Promises simply don't propagate progress to the next promise in the chain automatically.  Devs have to use some other mechanism (e.g. another event emitter, or whatever) to propagate progress.

2. Promises propagate progress automatically, and progress handlers are allowed to return a value that will be forwarded as the progress value to the next promise (similar to fulfill/reject value forwarding).  An exception in a progress handler should stop progress propagation, but*not* change the state of the associated or next promise (i.e. will not cause either to be rejected).  IMHO, there must also be some simple & reasonable way for a progress handler to explicitly stop progress propagation, such as returning undefined, which would be compatible with existing code that uses progress handlers on promises that don't allow propagation (but is also obviously different from how fulfill/reject handlers behave).

To me, allowing progress propagation *without* allowing progress handlers to return a new progress value or stop propagation seems like a recipe for lots of head scratching when someone's progress handlers start breaking because they receive a value they don't understand from a promise that is far upstream.

Thoughts?

Q has dealt with this recently. We're converging on something more like 2, mainly because my real-world use case parallels Stepan Riha's ("Our progress handler always sits at the very end of the promise chain.  Not propagating progress makes the feature either useless or terribly cumbersome"). My current thinking is that returning undefined propagates a progress event with undefined. We don't handle thrown exceptions yet, although obviously we should, hmm.

Related Q thread:

https://github.com/kriskowal/q/pull/114

See especially the question of what the progress looks like for C in:

var A = fast();
var B = slow();
var C = A.then(function () {
    return B;
});

[Luke Smith]

I think it's important to note that resolve and reject relate to flow control, and can be explained with analogies to synchronous code operation. Progress can't (so far as I can tell), except perhaps as code coverage instrumentation or debugger step hooks/breakpoint operations.  So I see it less as a requirement of a promise system and more as sugar; it's solving a different problem, and I find it an odd fit for inclusion in the then() signature.  Useful, yes, but necessary as part of the core spec, or better an extension to the core spec?

[Luke Smith]

On Monday, October 15, 2012 1:56:16 PM UTC-7, Luke Smith wrote:

I think it's important to note that resolve and reject relate to flow control, and can be explained with analogies to synchronous code operation. Progress can't (so far as I can tell), except perhaps as code coverage instrumentation or debugger step hooks/breakpoint operations.  So I see it less as a requirement of a promise system and more as sugar; it's solving a different problem, and I find it an odd fit for inclusion in the then() signature.  Useful, yes, but necessary as part of the core spec, or better an extension to the core spec?

All of which may be moot, and distract from the conversation.

I admit, I do like the look of progress handler at the end of the chain.

[Brian Cavalier]

Thanks Domenic.  Reading the Q discussions was very interesting--even more to think about :)  I'm leaning toward #2 also.

On Monday, October 15, 2012 4:01:31 PM UTC-4, Domenic Denicola wrote:

Q has dealt with this recently. We're converging on something more like 2, mainly because my real-world use case parallels Stepan Riha's ("Our progress handler always sits at the very end of the promise chain.  Not propagating progress makes the feature either useless or terribly cumbersome"). My current thinking is that returning undefined propagates a progress event with undefined. We don't handle thrown exceptions yet, although obviously we should, hmm.

Playing devil's advocate for a sec, I'm curious why you're thinking that returning undefined should propagate undefined rather than stopping propagation.

 

[Brian Cavalier]

Ok all, after pondering a bit more last night, here's my current thinking on what might be best.  Seems like this enables "progress handlers at the end of a promise chain", and keeps the special casing to a minimum.

1. Progress events propagate

Yay

2. When there is a progress handler, its return value, even if undefined (keep it simple, no special cases), will propagate to the next promise.  When there is no progress handler, the current progress value is propagated.  This is identical to how fulfill handlers work.

It seems like the only situation where this creates a WTF is when you forget to return a value from a progress handler.  But since the same rule applies to a fulfill handler, it seems like the consistency wins over special casing of undefined.

3. If a progress handler throws, the caught value will be propagated as the progress value.

My thinking here is that a library should never squelch exceptions, so somehow that exception must be made observable.  A progress handler should not be able to cause a promise to reject by throwing, so that's out.  Putting console.error inside when.js is not a realistic solution (although having when/debug log it *is* an option).  The only realistic places I can see to send the exception are: 1) to the next progress handler, 2) to the VM by rethrowing it in a setTimeout/nextTick, so as to make it uncatchable.  2 just seems way too heavy handed to me, given that there's no way for when.js to know anything about the exception.

What am I missing? Unless someone really hates that approach, I'm going to implement it in dev ASAP :)

On Thursday, October 11, 2012 12:01:45 PM UTC-4, Brian Cavalier wrote:

[Brian Cavalier]

This is in dev now.  Feel free to give it a go and shoot holes in it :)  Seriously, if you have any feedback, especially on the exception handling, I'd love to hear.

Thanks for all the discussion so far!

[Luke Smith]

So progress handlers will not support returning promises, like resolve and reject, because they relate to a different, dynamic, timeline. The returned promise would be passed to the next progress handler in the promise chain. Progress handlers are therefore inherently unprotected from race conditions (which is kinda the point of promises, right?).

After reading the Q thread and thinking on it more, I'm returning to my original position.  Promises solve the problem of flow control for a series of operations that may be sync or async. Progress tracking is a separate problem. Treating progress as an almost-sibling to resolve and reject conflates two timelines and complicates the simplicity of the story of promises by introducing exceptions (resolve and reject work like this, but progress works like this).  I think I'd rather limit the scope of progress to the currently executing promise, then compose broader promise generating functions that insert progress notifications in the inner promise chain.

But it's not spec'd, and my academic opinions haven't dealt with the inconveniences of actual use, so if you want to go with progress propagation, your decisions seem sound.

[Domenic Denicola]

(Wow, Google Groups totally lied when it said it would notify me about posts here.)

I tend to agree with Luke that it's kind of unfortunate progress got included in the original spec (even as underspecified as its behavior is). It's just convenient enough to be tempting, but just out of place enough to cause problems.

I don't feel strongly about special-casing undefined; it might be reasonable to stop propagation that way. The simplest approach is just to propagate it though, as you say, so I have no problem leaving Q like that.

Exceptions... grr. Really not sure here.

To play devil's advocate back at you, why shouldn't they be able to reject the promise? I only bring that up because all the other solutions are pretty bad too.

Rethrowing in nextTick is dangerous, given that e.g. it crashes you in Node or Windows 8 apps (although in the browser it's no problem).

Swallowing exceptions leads to hidden bugs, even though you could argue "it's just a progress handler, it can't be that important." Relatedly, maybe throwing should stop progress propagation, but then, what do you do with the error itself---hide it?

Passing it on to the next progress handler is not great; you end up with Node-style `if (err)` except it's more like `if (progress instanceof Error)` or even `if (!isExpectedTypeAndFormat(progress))`. If you don't have that code, you probably end up misusing an error object as progress data, causing a new exception in that progress handler, etc.

And what about if the last handler in the chain throws an exception? Are you currently swallowing it?

On Tuesday, October 16, 2012 9:52:23 AM UTC-4, Brian Cavalier wrote:

[Bob Myers]

I'd suggest, for no particularly good reason other than it seems useful, that promises be accepted as progress values.

The progress update is posted (ie the progress handler is called) when the promise fulfills.

The progress handler may also return a promise; if it does, downstream promises are given progress alerts when that promise fulfills.

Progress propagation stops when the progress handler throws, but  if the progress handler was a promise its error handler can be invoked to deal with that.

I've also given the initial progress_update an "org" argument which is passed all the way down, to help downstream guys figure out what progress is being reported to them.

Pseudo code:

Promise.prototype.progress_update=function(v,org){

    var promise=this;

    Promise.when(v).then(function(vv){

        for (var then in promise.thens) {

            try{

                then.progress_update(typeof then.progressHandler==="function" ? then.progressHandler(vv) : vv, org;);

            }

            catch (e) { /* cancel propagation of progress updates */ }

        }

    });

It's a simple extension to the notion of progress updates which could yield a lot of useful flexibility.

This is based on the notions that

1. Yes, progress reports propagate downstream (but with a second org argument that can provide additional information about the original progress report)

2. Yes, if the progress handler is a function then its return value is what is passed along, otherwise the value is passed along as is

3. Yes, the progress handler throwing terminates propagation. Errors are handled via the error handler on the progress update promise

On Saturday, October 20, 2012 4:30:21 PM UTC+5:30, Domenic Denicola wrote:

I tend to agree with Luke that it's kind of unfortunate progress got included in the original spec (even as underspecified as its behavior is). It's just convenient enough to be tempting, but just out of place enough to cause problems.

I don't feel strongly about special-casing undefined; it might be reasonable to stop propagation that way. The simplest approach is just to propagate it though, as you say, so I have no problem leaving Q like that.

[Brian Cavalier]

Hey Bob, thanks for the thoughts.  A few comments inline.

On Saturday, June 29, 2013 6:08:01 PM UTC-4, Bob Myers wrote:

I'd suggest, for no particularly good reason other than it seems useful, that promises be accepted as progress values.

The progress update is posted (ie the progress handler is called) when the promise fulfills.

The progress handler may also return a promise; if it does, downstream promises are given progress alerts when that promise fulfills.

Progress propagation stops when the progress handler throws, but  if the progress handler was a promise its error handler can be invoked to deal with that.

Actually, this is a problem.  Promises/A+ states that promise handlers may be called at most once.  So, if we call a rejection handler in response to a progress error, I can see only 3 choices:

1. call the associated rejection handler (assuming you meant the one that was passed to the same then() as the progress handler which triggered the progress error): this means we can never call this rejection handler again, which is bad since we need to call it if the promise rejects.  Calling a rejection handler before a promise becomes rejected will also be disallowed by the upcoming Promises/A+ v1.1, which leads to ...
   
2. transition the promise to the rejected state *and also* call the rejection handler: this leads to some tricky situations, but more importantly grants too much authority to a party who otherwise would not be able to reject the promise in question
   
3. don't call the rejection handler: unfortunately, this puts us back at square one

 It's a tricky situation for sure.

Allowing promises for progress event data also leads to some tricky situations.  For example, should a progress update be able to postpone the resolution of a promise indefinitely?  For example:

var d = when.defer();

consumer1(d.promise);

consumer2(d.promise);

function consumer1(promise) {

   // consumer1 has too much authority, can prevent the promise

   // from ever resolving

   promise.notify(promiseThatIsPendingForever);

}

function consumer2(promise) {

   // peer consumers of the same promise should never be able to

   // interfere with one another via the promise

   // consumer1 has prevented consumer2 from ever doing the important work

   // even though consumer1 should not have that authority

   promise.then(doImportantWork);

}

The sole responsibility for d.promise's state must belong to the owner of d.resolver, not with consumers of d.promise.

There are also weird (but probably less hazardous) cases like:

var d = when.defer();

d.notify(promiseThatFulfillsInTenSeconds);

d.resolve(promiseThatFulfillsInOneSecond);

When does d.promise fulfill?

Part of the problem is that progress events deal with the time before a promise resolves, and fulfillment/rejection handlers deal with the time afterward.

Any thoughts on those issues?  I'm def interested in figuring out a good solution