futures and error handling

[Massimiliano Tomassoli]

Hi,
I'm reading another article about futures: Futures and Error Handling.
Have a look at this example which appear in the above-mentioned article:

  void main() {
    funcThatThrows()
      .then((_) => print("Won't reach here..."))   // Future completes with an error.
      .whenComplete(() => print("... or here...")) // Future completes with the same error.
      .then((_) => print("... nor here."))         // Future completes with the same error.
      .catchError(handleError)                     // Error is handled here.
  }

I'm not a native speaker but it seems to me that the strings in this example suggest that "... or here..." won't be printed, which is false.

[Alex Tatumizer]

One of the reasons why I'm advocating Circuit API is that thinking in terms of Future API is almost impossible for me.

Things are so complicated there that tutorials themselves are full of bugs.

Error processing, which is quite a brain-teaser even for normal program, becomes a nightmare.

Cited example from tutorial shows how confusing things are even in the simplest scenario

void main() {

  funcThatThrows()

    .then((_) => print("Won't reach here..."))   // Future completes with an error.

    .whenComplete(() => print("... or here...")) // Future completes with the same error.

    .then((_) => print("... nor here."))         // Future completes with the same error.

    .catchError(handleError) ;                    // Error is handled here.

}

First off, if funcThatThrows really throws, none of the handlers will be invoked - program will simply crash with error. Try it out:

funcThatThrows() {

  throw "thrown by funcThatThrows";

}

There's no one behind this function to catch error - it propagates to main event loop.

Tutorial uses this functionThatThrows everywhere, which makes all examples just misleading.

What is meant there is not "funcThatThrows", but "funcThatReturnsFutureThatCompletesWithError":

funcThatReturnsFutureThatCompletesWithError() {

  return new Future.sync(()=>42).then((v)=>throw "hello from then handler");

}

Having fixed this, lets' execute that example. For convenience, here's the full text of program:

import 'dart:async';

void main() {

  funcThatReturnsFutureThatCompletesWithError()

    .then((_) => print("Won't reach here..."))   // Future completes with an error.

    .whenComplete(() => print("... or here...")) // Future completes with the same error.

    .then((_) => print("... nor here."))         // Future completes with the same error.

    .catchError(handleError) ;                    // Error is handled here.

}

handleError(e) {

  print("handling error $e");

}

funcThatReturnsFutureThatCompletesWithError() {

  return new Future.sync(()=>42).then((v)=>throw "hello from then handler");

}

When you run it, it prints

... or here...

handling error hello from then handler

Naturally, it prints ".. or here", though one line above it pledges to NOT reach here.

And if you think things are a bit confusing, you aint seen nothing yet. Try parallel operations, timeouts, futures in a loop...

And multiply this by a 1000 to get the estimate of complexity of real project, full of database calls, connection retries, error recovery and whatnot.

Guys, please give Circuit a try. It's Turing-complete, easy to understand mechanism.  

Too bad it didn't draw any attention: but I console myself with the fact that number 0 took 5000 years to be recognized as valid concept.

https://github.com/tatumizer/circuit-dart

I'd like to use this as opportunity to challenge dart team for discussion: Raw Futures vs Circuit. How about that?

 

[Anders Holmgren]

I'd like to see you blog w a real world(ish) problem where you implement first as normal futures and then as circuit. Followed by a discussion of the pros and cons of each.

[Massimiliano Tomassoli]

On Saturday, April 19, 2014 6:56:02 PM UTC+2, Alex Tatumizer wrote:

Guys, please give Circuit a try. It's Turing-complete, easy to understand mechanism.  

Too bad it didn't draw any attention: but I console myself with the fact that number 0 took 5000 years to be recognized as valid concept.

https://github.com/tatumizer/circuit-dart

I'd like to use this as opportunity to challenge dart team for discussion: Raw Futures vs Circuit. How about that?

I'm new both to Dart and to asynchronous programming, so this is just my very humble opinion.
Your Circuits looks interesting, but I don't think they're a big improvement over raw futures. For instance, let's look at a couple of examples taken from your documentation.

Here's the first one:

  Future<int> add(a, b) => new Future(() => a + b);

  Future<int> add4(a, b, c, d) {
    return new Circuit([
        (@out sum1) => sum1 << add(a, b),
        (@out sum2) => sum2 << add(c, d),
        (sum1, sum2, @out result) => result << add(sum1, sum2)
     ]);
  }

And here's how I'd do that without circuits:

  Future<int> add(a, b) => new Future(() => a + b);

  Future<int> add4(a, b, c, d) {
    var sum1 = add(a, b);
    var sum2 = add(c, d);
    return Future_call(add, sum1, sum2);
  }

Here I use a function called Future_call which waits for both sum1 and sum2 to complete and then passes the values to add. I'm not sure such a function can be implemented efficiently in Dart. I've just read that Dart doesn't support varargs, etc... IMHO that's a big limitation because it makes difficult to write functions that modify other functions.

Here's the second example:

  sumProgression(n) {
    int acc=0, k=1;
    return new Circuit([
        (@out s) { s << add(acc, k++); },
        (s, @out next, @out result) { acc=s; (k <= n ? next : result) << acc;}
     ]);
  }

which isn't as intuitive as you think. I prefer my version without circuits:

  sumProgression(n) {
    if (n <= 0) return new Future.value(0);
    else return sumProgression(n - 1).then((sum) => add(sum, n));
  }

The only drawback is that n can't be too big. More efficient:

  sumProgression(n) {
    recur(acc, i) {
      if (i > n) return acc;
      else return new Future(() => recur(Future_call(add, acc, i), i + 1));
    }
    return recur(new Future.value(0), 1);
  }

[Alex Tatumizer]

You don't need Future_call. There's a function already in async library: static Future<List> wait(Iterable<Future> futures, {bool eagerError: false})

I'm aware of it. It doesn't play well with error processing.

>The only drawback is that n can't be too big. 

The only? Well.

Second version (probably) suffers from the same problem, but I admit I don't understand it. Are you sure it's simpler than Circuit version?

To avoid out-of-memory condition, you have no choice but create Completer and wire everything by hand.  With Cirucit, this is not necessary..

I admit the syntax of Circuit iteration, and the idea of "next" latch, are not ideal (to say the least). It can be improved. I know several ways to improve it, that's why I want to discuss.

I'm not aware of other flaws with respect to readability. Actually, I WANT people to criticize the concept to make it better.

Anything that can be said about advantages of Futures over Circuit can equally apply to goto vs for/while/do/continue/break. Goto looks better in all respects, until you really start using it and discover a big mess. 

Please don't hesitate to send further comments about Circuit - what you like and what you don't. I appreciate any feedback.

Thanks,

Alex

[Massimiliano Tomassoli]

On Sunday, April 20, 2014 6:24:35 AM UTC+2, Alex Tatumizer wrote:

You don't need Future_call. There's a function already in async library: static Future<List> wait(Iterable<Future> futures, {bool eagerError: false})

You don't need Future_call but it's very handy:

  Future_call(add, a, b)

instead of

  Future.wait([a, b]).then((args) => add(args[0], args[1]))

Of course, Future_call will use Future.wait internally. Future_call would also support non-future arguments. For instance, in one of my examples I use
 
  Future_call(add, acc, i)

where acc is a future but i isn't.
 

>The only drawback is that n can't be too big. 

The only? Well.

Second version (probably) suffers from the same problem, but I admit I don't understand it. Are you sure it's simpler than Circuit version?

The second version shouldn't suffer from the same problem. Here is the code again:

  sumProgression(n) {
    recur(acc, i) {
      if (i > n) return acc;
      else return new Future(() => recur(Future_call(add, acc, i), i + 1));
    }
    return recur(new Future.value(0), 1);
  }

Let's say we call sumProgression(3). In what follows, each line is equivalent to the previous one:

  sumProgression(3)
  recur(new Future.value(0), 1)
  new Future(() => recur(Future_call(add, new Future.value(0), 1), 2))

Then the event loop executes

  recur(Future_call(add, new Future.value(0), 1), 2))

which is equivalent (at least conceptually) to

  recur(new Future.value(1), 2))

The consumption of memory is constant as you can see.

[Alex Tatumizer]

I think you started building your own concept. At some point, you will want to explain it to someone, then you will need more WORDS.

These words are: circuit, gate and latch (or something equivalent). My bet is: you will need all 3.

If you search npm repository, you will find hundreds of libraries attempting to do something about async programming (which is out-of-box horrible in nodejs - they don't even have futures).

A couple of years ago I  searched for the use of words "circuit", "latch" and "gate" with respect to javascript programming (npm, github etc).. These words were not in use.

Today, if you do the same, you will find quite a few. Some libraries use "circuit", some -"gate", some -"latch", The meaning of these words might be completely different (people just liked the word), but sometimes it's the same. To the best of my knowledge, no library uses combination of 2 words. (Probably, we will see examples next year). In fact, we need all 3. This will take another couple of years (or decades). The point is that meme spreads, and this process always takes time.  

The reason why you need more words is simple: programming language is human language, not solely a mathematical formalism. Occam doesn't apply here, or else we would be programming in Turing Machine with 3 commands. 

With respect to iterative computation, I think more elegant resolution would be to introduce asyncFold for streams. Currently, we have asyncMap and asyncExpand (BTW, they were added just recently). Probably, we need asyncFold, then sum of progression will become 1-liner. I have to investigate, will post this 1 line later.

 

[Alex Tatumizer]

Here's a 1-liner:

new Stream.fromIterable(new Iterable.generate(n+1)).asyncFold(0, add);

I opened an issue for asyncFold, please star: https://code.google.com/p/dart/issues/detail?id=18331

Now I can safely remove everything related to iterations from Circuit - it will simplify things quite a bit. 

Thanks for complaint! :-) In fact, I had some unresolved issues with iterative circuit which I didn't know how to handle(e.g. what to do with zombies from prior iterations) - now these problems are eliminated altogether.

@Anders, sorry I overlooked your comment: 

>> I'd like to see you blog w a real world(ish) problem where you implement first as normal futures and then as circuit

This won't work, One reason is that attention span of typical reader these days is 15 sec. Real-worldish problem in a blog, with 2 implementations? Even description of a problem will by far exceed the limit.

Another reason is that many of advantages of Circuit cannot be seen from text itself. E.g. Circuit is easy to debug, test, analyze performance, explain to yourself and others, etc.

(E.g., testing, due to spies, is a breeze: without modifying anything in your program, you get events, to which you can apply "expect"...).

One doesn't need full-blown implementation of unknown problem to appreciate these features. They are not visible in implementation per se, but come due to the fact you use the concept.

(There's nothing equivalent in raw futures). These characteristics are hard to measure quantitatively. (BTW, same was true for "goto" and a lot of other programming devices).

Having said that, if you have example of a problem that can be defined in simple terms, please let me know. Most real-world examples I'm aware of require the use of databases - that's where async programming can be very efficient, and that's exactly what typical server-side program is doing most of the time - but these examples, however practical, would be quite boring to read...

 

[Massimiliano Tomassoli]

On Sunday, April 20, 2014 11:03:58 PM UTC+2, Alex Tatumizer wrote:

Here's a 1-liner:

new Stream.fromIterable(new Iterable.generate(n+1)).asyncFold(0, add);

Nice.
 

I opened an issue for asyncFold, please star: https://code.google.com/p/dart/issues/detail?id=18331

I don't know why I didn't think of it before, but we don't need recursion:

  sumProgression(n) {
    Future acc = new Future.value(0);
    for (int i = 1; i <= n; i++)
      acc = acc.then((x) => add(x, i));
    return acc;
  }

[Alex Tatumizer]

This program is equivalent to recursion memory-wise. You are creating a linked list of N futures still. None of these futures has any chance to execute until the whole chain is built, and you return to caller, and caller returns to its caller etc.

 

[Alex Tatumizer]

BTW, it's a good exercise to think how it can be done in O(1) memory. Hint: create a Completer.

(Maybe there's a way to do it without completer, but I'm not aware of any. Stream with asyncFold will create such completer behind the scenes) 

[Massimiliano Tomassoli]

On Monday, April 21, 2014 3:28:19 PM UTC+2, Alex Tatumizer wrote:

BTW, it's a good exercise to think how it can be done in O(1) memory. Hint: create a Completer.

(Maybe there's a way to do it without completer, but I'm not aware of any. Stream with asyncFold will create such completer behind the scenes) 

As I've already shown, my second version is O(1).

[Massimiliano Tomassoli]

On Monday, April 21, 2014 3:01:47 PM UTC+2, Alex Tatumizer wrote:

This program is equivalent to recursion memory-wise. You are creating a linked list of N futures still. None of these futures has any chance to execute until the whole chain is built, and you return to caller, and caller returns to its caller etc.

My point was that one doesn't need recursion.

[Alex Tatumizer]

Sorry, but you said:
> Here I use a function called Future_call which waits for both sum1 and sum2 to complete and then passes the values > to add. I'm not sure such a function can be implemented efficiently in Dart.

You proof will be complete when you manage to implement it, and then actually run the whole thing  :-)

[Massimiliano Tomassoli]

But as you said, Future_call isn't needed. It's just handy.
I did test my two versions with a big n. As expected, the first one throws (out of memory), whereas the second one works just fine.

[Alex Tatumizer]

OK, understood :-)