Synchronous, but non blocking, versions of the Vert.x APIs

[Tim Fox]

As you all know one of the key features of Vert.x is it's non blocking (as in not blocking *operating system* threads) which means it can scale well.

In order to do non blocking we end up with asynchronous APIs (either callback/promise/rx) as Java (and the JVM) doesn't have any concept of continuations http://en.wikipedia.org/wiki/Continuation

Over the years various attempts have been made to do continuation like things in Java programs either by byte code engineering or transforming programs into continuation passing style http://en.wikipedia.org/wiki/Continuation-passing_style but nothing has really worked well.... until...

.. my attention was drawn today to this project: http://www.matthiasmann.de/content/view/24/26/ (which is used by quasar http://blog.paralleluniverse.co/2013/05/02/quasar-pulsar/ another interesting project)-  this claims to provide continuations, in Java, with little overhead.

If this is true (I haven't verified the claims), then that means we could (using Vert.x codegen) generate synchronous versions of any Vert.x API,

E.g. instead of

client.getConnection(res -> {
  if (res.succeeded()) {
     Connection conn = res.result();
     conn.executeQuery(query, res2 -> {
        if (res2.suceeded()) {
            ResultSet rs = res2.result();
        } else {
           // handle exception
        }
     });
  } else {
    // handle exception
  }
}

You could write:

Connection conn = client.getConnection();
ResultSet rs = conn.executeQuery(query);

and none of the above would actually block an OS thread.

Anyway I thought this might be an interesting idea for a little sub project....

[Stephane Bastian]

Great !

If it works the way it's advertised, it's great news and very interesting!

Stéphane

[Simon Gemmell]

Yes please - async code looks hideous in Java, even when using CompletableFuture rather than insane nesting.

[Nicolaas Frederick Huysamen]

Apart from Vert.x already being really awesome, this would be a serious game-changer in my opinion. I think this would increase adoption quite substantially, as not everyone is comfortable with frameworks like Rx.

[Simon Gemmell]

While looking up await/async for C++ I came across this which lists a couple of implementations in java:

http://stackoverflow.com/a/25624909/1284610

Posted here for ease:

Java libraries implementing Fibers

• JetLang
• Kilim
• Quasar (mentioned by tim)

[Nicolas Le Bas]

Just being able to write integration tests in this manner would make the
transition to vertx so much easier.

Game changer indeed.

On 15-05-25 02:33 AM, Nicolaas Frederick Huysamen wrote:
> Apart from Vert.x already being really awesome, this would be a serious
> game-changer in my opinion. I think this would increase adoption quite
> substantially, as not everyone is comfortable with frameworks like Rx.
>
> On Friday, 22 May 2015 15:36:17 UTC+2, Tim Fox wrote:
>
> As you all know one of the key features of Vert.x is it's non

> blocking (as in not blocking **operating system** threads) which

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to vertx+un...@googlegroups.com
> <mailto:vertx+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Tim Fox]

I needed a break from the 3.0 work so I spent a few hours hacking on this and I've got a simple prototype working:

https://github.com/purplefox/vertx-sync

[Joern Bernhardt]

I don't really get how this works - but I'd love to know. ;)

Is this supposed to work with "everything" that's async right now? What happens, when an excpetion occurs, let's say during reading a file asynchronously. Will it throw an exception just like it would if it were truly sync?

What happens with handlers that fire multiple times? Like incoming event bus handlers or server requests.

Thanks,

Joern

[Tim Fox]

On 30/05/15 18:01, Joern Bernhardt wrote:

I don't really get how this works - but I'd love to know. ;)

Is this supposed to work with "everything" that's async right now?

The initial idea is to convert async methods that take a Handler<AsyncResult<>> like this:

https://github.com/purplefox/vertx-sync/blob/master/src/test/java/io/vertx/ext/sync/test/AsyncInterface.java

Using codegen into the sync equivalent like this:

https://github.com/purplefox/vertx-sync/blob/master/src/test/java/io/vertx/ext/sync/test/SyncInterface.java


Then you can call them in a direct style like this:

https://github.com/purplefox/vertx-sync/blob/master/src/test/java/io/vertx/ext/sync/test/MyVerticle.java#L18

Even though the sync call might take 10 minutes to return a value, it doesn't actually block an OS thread.

What happens, when an excpetion occurs, let's say during reading a file asynchronously. Will it throw an exception just like it would if it were truly sync?

Yes

What happens with handlers that fire multiple times? Like incoming event bus handlers or server requests.

This is specifically for Handler<AsyncResult<>> which is inherently single shot (for now)

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[Joern Bernhardt]

Ok, so after reading a bit more about it (at the Quasar docs here http://docs.paralleluniverse.co/quasar/), I got another question:

Would it be possible to write a JDBC "wrapper" interface around actual drivers and have them async, too? I understand that I use this @Suspendable (or throws SuspendException) to mark my own methods and get their calls changed in the bytecode. But what about other APIs - is there a chance to get those async now?

Thanks,

Joern

[pron]

Hi. 

I work on Quasar.

There's a growing number of Quasar-integrated third-party libraries here. To integrate your own (as Tim did with vert.x itself) follow the guidelines and examples here.

JDBC is integrated (as well as jOOQ and JDBI), but because there isn't an async JDBC API, a separate thread pool is used to run (and block on) the actual JDBC operations. This shouldn't really matter, though. When you use the JDBC-quasar integration, the fiber making the call (and the sync verticle multiplexes fibers onto a single thread) blocks without blocking the underlying kernel thread. 

Please note that while JDBC itself is integrated, that doesn't mean any library built on top of it -- namely Hibernate or other JPA ORMs -- is. Currently, direct JDBC, jOOQ and JDBI are supported. We are working with Oracle to introduce a (very minor) change to the JVM in Java 9 that would make it so that if a low-level API (like JDBC) is integrated with Quasar, then anything built on top of it (like Hibernate) would work, too (and you won't need to mark your blocking methods with @Suspendable/throws SuspendExecution either.

Ron

[Tim Fox]

Don't confuse making something async with non blocking.

Our current JDBC client already wraps the inherently synchronous JDBC interface with an async wrapper. However that doesn't mean things aren't non blocking internally -we still use a thread pool (the worker pool) to perform to call the blocking JDBC driver.

So you still have your scalability bottleneck - it's just hidden away now behind an async API. Ideally everything would be non blocking (including the JDBC driver-  e.g. Mongo async client already is) but it's _very_ hard to take some blocking code written by someone else and magically make it non blocking.

But is that possible at all? I have spent some time thinking about it, and if it were possible it would be the holy grail as it means we could take any of those blocking libraries available in the Java ecosystem and use them effectively in a scalable way.

So how would our magical tool do this? I think we'd have to make a list of all the things in the JDK that can potentially block a thread.. this would include many IO operations - e.g. socket read/write and various other things. I guess we'd then have to intercept calls to such methods and inject code that replaced the operation with a non blocking equivalent (e.g. a non blocking socket write) and do some Quasar-like shenanigans so the fiber could "return" to that point in the call stack when the operation was complete. Or something like that ;)

[pron]

What you're suggesting might well be possible (in fact, it won't even require any work) with Quasar and Java 9. 

The question is how worthwhile it will be, especially considering JDBC. Blocking threads isn't bad: the OS is very, very good at waking blocked threads very quickly (although, only threads blocked on IO; not those waiting to be woken up by other threads). This is why blocking IO has lower latency than async IO. The problem with blocking threads is one of throughput and scale: the OS is not good at scheduling lots and lots of threads or threads that block very often. So it really shouldn't bother you -- neither should you try to optimize -- a pool of, say, 10 threads that blocks on JDBC. Your database is likelier to hit its limits long before blocking those few threads would have any negative impact on your server. You should be worried about blocking if you're using an in-memory database/cache or anything else that can operate at very high throughput. 

There's nothing we can do about latency (async IO makes it a little worse, in fact); what we're trying to increase is throughput (while maintaining reasonable latency), so the question of how important is not blocking kernel threads, depends on the throughput potentially offered by the API. JDBC usually offers relatively low throughput, so blocking kernel threads is probably good enough to make async not worthwhile.

Ron

[Tim Fox]

Hi Ron,

On 31/05/15 07:03, pron wrote:

Hi. 

I work on Quasar.

There's a growing number of Quasar-integrated third-party libraries here. To integrate your own (as Tim did with vert.x itself) follow the guidelines and examples here.

JDBC is integrated (as well as jOOQ and JDBI), but because there isn't an async JDBC API, a separate thread pool is used to run (and block on) the actual JDBC operations. This shouldn't really matter, though. When you use the JDBC-quasar integration, the fiber making the call (and the sync verticle multiplexes fibers onto a single thread) blocks without blocking the underlying kernel thread.

We already use an approach of wrapping blocking drivers (such as JDBC drivers) with an aysnc API (and have done for some time). E.g. here's our JDBC client http://vert-x3.github.io/docs/vertx-jdbc-client/java/

Wrapping blocking stuff with an async API means the user can use it without worrying about blocking OS threads but it doesn't really solve the problem of why we want non blocking in the first place - and that's scalablity - if, somewhere, you have a thread pool that does blocking operations then that's probably going to be your bottleneck. That's why we try and steer users to our truly non blocking database solutions - e.g. the latest async Mongo client http://vert-x3.github.io/docs/vertx-mongo-client/java/, or our mysql-postgresql driver http://vert-x3.github.io/docs/vertx-mysql-postgresql-client/java/-  these use non blocking I/O down to the OS calls - i.e. no thread pools.

To me, the holy grail would be (as mentioned in my previous post) if we could take an existing blocking driver (e.g. the Oracle JDBC driver) and magically convert it to be non blocking (and I don't mean just wrapping it with an async API and using a thread pool). I leave the implementation as a challenge for the reader ;)

Please note that while JDBC itself is integrated, that doesn't mean any library built on top of it -- namely Hibernate or other JPA ORMs -- is. Currently, direct JDBC, jOOQ and JDBI are supported. We are working with Oracle to introduce a (very minor) change to the JVM in Java 9 that would make it so that if a low-level API (like JDBC) is integrated with Quasar, then anything built on top of it (like Hibernate) would work, too (and you won't need to mark your blocking methods with @Suspendable/throws SuspendExecution either.

This is an interesting point. Having to mark the full fiber call stack as Suspendable is certainly a drawback right now. In Vert.x we support multiple languages not just Java and it would be great if we could use Quasar to do what I've done in the vertx-sync prototype but with other languages.

Right now that doesn't seem possible as the call stack would need to go through dynamic code that's implemented in other libraries, e.g. for JavaScript it looks something like this:

Java code (run fiber) -> call Nashorn to run JS script -> lots of dynamic wizardry -> call back into Java code -> call the suspendable Java API.

Right now the above won't work because of the requirement to annotate the whole call stack as Suspendable but it would be awesome if one day it was possible :)

[Tim Fox]

On 31/05/15 09:02, pron wrote:

What you're suggesting might well be possible (in fact, it won't even require any work) with Quasar and Java 9. 

The question is how worthwhile it will be, especially considering JDBC. Blocking threads isn't bad: the OS is very, very good at waking blocked threads very quickly (although, only threads blocked on IO; not those waiting to be woken up by other threads). This is why blocking IO has lower latency than async IO. The problem with blocking threads is one of throughput and scale: the OS is not good at scheduling lots and lots of threads or threads that block very often. So it really shouldn't bother you -- neither should you try to optimize -- a pool of, say, 10 threads that blocks on JDBC. Your database is likelier to hit its limits long before blocking those few threads would have any negative impact on your server. You should be worried about blocking if you're using an in-memory database/cache or anything else that can operate at very high throughput. 

There's nothing we can do about latency (async IO makes it a little worse, in fact); what we're trying to increase is throughput (while maintaining reasonable latency), so the question of how important is not blocking kernel threads, depends on the throughput potentially offered by the API. JDBC usually offers relatively low throughput, so blocking kernel threads is probably good enough to make async not worthwhile.

This really depends on what is actually limiting your throughput, there are a few possibilities:

1. Your database server is already working at full capacity. If this is the case then there's nothing you can do at the client side to increase throughput.

2. Design limitation in the driver - e.g. the database driver has a poor wire protocol - for example it doesn't support pipe-lining, so every request needs a network round trip, and your network is slow. This means your throughput can be limited by the latency of the network

3. You are limited by the size of the thread pool on the client. In this case the server is not at capacity and the driver is not design crippled. What's limiting your throughput is the fact you're limited to a max of X threads in your thread pool, because increasing higher goes into law of diminishing returns because of context switching overhead. Let's say each query takes 1 second to execute on average that means you have a maximum throughput of 200 requests per second even though your server and driver can be pushed much higher.

Non blocking db drivers solve number 3 but not 2 an 1. So the question is how often is the real problem 3 rather than 2 or 1? I'd say 3 is pretty common - most database servers won't let a single query max out the entire server - they're often big multi-core servers. Also it's worth bearing in mind that some applications will need to talk to more than one server, i.e. have more than one thread pool and its the total number of threads in all pools that we don't want to push above X.

[Tim Fox]

On 31/05/15 09:02, pron wrote:

What you're suggesting might well be possible (in fact, it won't even require any work) with Quasar and Java 9.

I am intrigued! Can you perhaps share some insight on how this would work and what's new in Java 9 that might enable this?

[pron]

Gladly -- but only once Oracle fully approves the change...

[Tim Fox]

On 31/05/15 09:31, pron wrote:

Gladly -- but only once Oracle fully approves the change...

I can't wait wait :)

When I was thinking of this before I thought one possibility, at the JDK level, would be for the JDK to let the underlying threading implementation be pluggable, so you could basically plugin your own green threads implementation (e.g. Quasar). But then you have to deal with all the system calls too (socket read/write etc) which would be a big job.

But I'm guessing from your previous replies that the change involves some kind of first class support for continuations (or at least easier manipulation of the stack) in the JDK.. ;)

[Simon Gemmell]

I had a quick look at this, it looks promising! Doesn't even look like too much has to change?!

As an example, in my tests I have this (hard to read) code which promisify's the continuations to be Java CompletableFuture's. Given the current state of play, what would actually need to happen to put something like this through the async/sync adapter?

private CompletableFuture<JsonArray> doProjectSearch(JsonObject postBody, TestContext context) {
   HttpClientRequest postReq = deployment.httpClient().post(deployment.httpPort(), deployment.httpHost(), "/api/2/projects-search");
   CompletableFuture<HttpClientResponse> postReqCF = Promisify.promisify(postReq);
   postReq.end(postBody.encode());
   Async async1 = context.async();
   return postReqCF.thenCompose((HttpClientResponse resp) ->
      Promisify.readHttpResponseBodyAsFuture(resp).thenApply((Buffer body) -> {
         JsonArray data = new JsonArray(body.toString());
         async1.complete();
         return data;
      })
   );
}

[Simon Gemmell]

The promisify functions look like this:

public static CompletableFuture<HttpClientResponse> promisify(HttpClientRequest request) {
   CompletableFuture<HttpClientResponse> cf = new CompletableFuture<>();
   request.handler(cf::complete);
   request.exceptionHandler(cf::completeExceptionally);
   return cf;
}

public static CompletableFuture<Buffer> readHttpResponseBodyAsFuture(HttpClientResponse response) {
   CompletableFuture<Buffer> cf = new CompletableFuture<>();
   response.bodyHandler(cf::complete);
   response.exceptionHandler(cf::completeExceptionally);
   return cf;
}

[Julien Viet]

The initial version of vertx-rx was actually adding extra methods that returned CompletableFuture<T> object for Handler<AsyncResult<T>>.

-- 
Julien Viet
www.julienviet.com

[pron]

All CompletableFutures can be automatically made fiber-blocking by Quasar, out of the box. Any function foo returning CompletableFuture<T> can be called as:

T result = AsyncCompletionStage.get(foo());

Which blocks the current fiber (but no kernel threads!), until the CompltableFuture completes. Any abnormal termination of the future will result in a simple thrown exception that can be caught as usual.

[Tim Fox]

Cool.

After we get Vertx 3.0 out, one thing I would like us to look at is in making a version of the Vert.x APIs that use CompletableFuture instead of callbacks - this would not only be useful for those users who prefer CompletableFuture but also make it easier to "fiberify" using Quasar as you mention below :)

Then we have a lot of choice for the user wanting to use Vert.x

1. "classic" callback based API
2. CompletableFuture based API
3. RxJava style API
4. Direct style API using Quasar fibers.

[Matthew Tyson]

This thread is old, but I am new to vert.x and running headlong into the maws of this particular issue.

With the naivette of a new adopter, the "holy grail" appears to be a way to make Java return async such that a normal call:

Objecy myResult = anObject.fantasticalMethod("foo");

Is actually async -- at least, customizable as such.

At least, for my needs, that would be beautiful.