Comments on development and runtime model

[Nils Henrik Lorentzen]

Hi,

I have been looking through the specifications and trying to grasp better the scope of the project. What strikes me is that there seems to be a lot of important mechanisms (fault tolerance, async messaging, streams processing) that is discussed and specified but seems lacking a bit on a coherent programming model to fit these mechanisms into (see Erlang for the extreme side of strict programming models).

My developer’s intuition tells me that one might over time end up in a situation where the executing container lacks the overall knowledge to schedule efficiently and there will be difficult to debug performance issues due to eg. same bean used both in sync and async context.

Stepping back a bit, there are a number of use cases one might want to handle in a container. The container ought to be given the best possible information about how to optimally allocate resources for and schedule these.

 

 - controller logic

    - in synchronous flow, eg. once typical controller in a web MVC framework that hits the DB, does some logic and returns a new view.

Network bound if hits DB.

    - in asynchronous invocation (message passing), typically state machines in order to reason about reactive invocations. Normally execute quickly.

 - long running batch jobs.

    - batch DB CRUD operations, blocking, network bound

    - data processing, polling data from queue or otherwise

 - stream/message queue processing, reactive, might be CPU bound for data processing

Probably lots more such higher level patterns.

How a container would optimally schedule this vary a lot depending on whether is CPU, local I/O (eg. persistent queue journal) or network bound

- CPU bound - number of threads depend on number of CPU cores, to avoid unnecessary context switching.

- local I/O bound - one thread executes at a time to avoid thrashing disk access (SSDs may fare better).

- network I/O - any number of threads almost since they will be blocking for I/O most of the time.

From this I’d say one would want to give the container as many hints as possible about the behaviour of managed entity, even up to the container refusing to perform an operation (like blocking I/O) unless the component has explicitly asked for it.

This could be done mainly declaratively, similar to the @Stereotypes in the CDI spec (?).

@MessageDriven - message driven bean, stateless, asynchonously invoked, can do synchronous calls

@Actor - stateful message processor, asynchronously invoked, useful for event processing logic (events from multiple sources that must be synchronised via state machines)

@Stateless - existing stateless session bean, synchronously invoked.

@Stateful - existing stateful bean, not many use cases for these as state normally resides persistently in DB

@MVCController

@DAO

@GET, @POST for REST services

@Batchjob - like blockingly reading from a queue and writing to DB

@Polling - at intervals, call some service (like an external FTP server) to check for new data.

@LongPolling - mostly used from client side I guess and not appropriate patter to use in a container.

This gives the container hints of what the role of the bean is within the system. In addition could be properties that would tell the container how to schedule

@Blocking(contraint=NETWORK, roundtrips=4) 

roundtrips would be a hint that for one single invocation of a method, it will do four network roundtrip calls. Is just for illustration this,

probably something a container might figure out anyway by keeping stats for any incoming bean call leading to outbound calls to network, and then adapt scheduling dynamically at runtime.

@CPUIntensive - could tell container to try to instrument execution time and adjust scheduling, eg. just pass to one of the per CPU-core bound threads to optimize for CPU cache locality.

Specifying these high-level patterns also helps the developer think clear about architecture, and might improve readability for other developers.

Intuition tells me one would want to avoid beans that are both CPU intensitive and spend long time blocking in the same call chain, rather do these in separate specialised beans with execution scheduled independently.

Good case for message passing/reactive programming.

For reactive programming,  an advantage here is of course that it gives container more leeway on how to schedule. Message passing or CSPs

allows the container to put tasks on queues and schedule when appropriate while a fully synchronous stack requires a lot of threads since most stuff will block towards some external DB or something of the sort.

However for there to be a large advantage to it I believe it has to be reactive all the way to something that takes time executing, eg. make network call async, simple if-this-do-that logic execute so fast anyways that whether reactive or not does not matter for performance.

All network interaction, even DB calls can be made asynchronous all down to the network wire. Client side Ajax calls already do that (via REST), they call Ajax with a continuation function and the browser can schedule REST HTTP calls at will, even in parallell. Below, the web browser could do all 10 calls to the server in parallel, utilising only one thread by doing asynchronous I/O for HTTP.

for (var i = 0; i < 10; ++ i) {
sendAsyncQueryForSomeInforFromDB(url, i, param2, param3, function() {
console.log(‘got HTTP response ’ + i + ‘ from server side’);
})
}

Entirely doable from Java too (easy with lambdas) but very few existing blocking APIs allows one to do this, a google search gives me this though

https://blogs.oracle.com/java/jdbc-next:-a-new-asynchronous-api-for-connecting-to-a-database

JPA does not support this, it is entirely synchronous though one could add continuation APIs in the future.

II suppose in theory, if all I/O is done async , one can get full CPU utilisation with only as many threads as there are cores, and avoid a lot of thread context switching, since threads would never block for I/O and hence be doing something, async processing is however more challenging programming wise, whether using continuations or state machines.

Oh well, just comments for further thought this.

Regards,

Nils Henrik Lorentzen

[Nils Henrik Lorentzen]

Less verbose and more highlevel way to express the same:

Imagine an application server dashboard that shows a live visual diagram of ones running applications after CDI wiring, showing the wiring connections and statistics on number of invocations among those wirings per time and total.

The container should have enough information from declarative annotations to mark groups of beans as reactive, other groups as synchronous.

Implies a few things:

 - cannot mix reactive and blocking programming in the same bean, confusing to developers and to the container.

 - purely async reactive beans should preferably never call blocking methods, preferably all the way to the network wire (so like Ajax calls in a browser). For beans declared as such, the container should be able to detect and throw exceptions for any blocking method calls, since it injects dependencies with proxies. If injected interfaces for I/O access are either blocking or non blocking, the container might even be able to detect this at wiring time (a purely async reactive bean asks for an interface that exposes synchronous I/O methods).

- an purely async reactive bean might then of course delegate to a non-async bean that can do blocking I/O by posting on a non-blocking message queue but unless the async bean does CPU intensitive opertions or models state machine event processing logic or perhaps async workflow/eventbus patterns, there is not much use of doing that. The main advantage comes from also doing I/O asynchronously.

Purely async reactive beans may then be scheduled by the container from a a fixed threadpool, while anything that requires blocking for I/O anywhere in their execution call chain needs many threads and dynamic threadpools, causing more overhead.

One could also have reactive beans that call blocking methods, these would then be showned as not purely async on the diagram and would also be scheduled from a dynamic threadpool.

Slight digression, this is sort of the difference between Apache web server and Nginx. Early apache spawned one OS process per inbound connection, scaled horribly (somewhat improved by using threads instead of OS processes). Nginx has one thread that uses NIO (or rather the Linux system call equivalent epoll() ) to manage connections, then passes on to worker processes. Since event driven, the number of processes equals the number of cores, minimizing context switching and increasing throughput.

"How Does NGINX Work? NGINX is built to offer low memory usage and high concurrency. Rather than creating new processes for each web request, NGINX uses an asynchronous, event-driven approach where requests are handled in a single thread."

[Emily Jiang]

Hi Nils,

Thanks for sharing your thoughts on reactive and async! I believe we are trying to define a stack for truly reactive and non-blocking. At the moment, we are trying to finalise Reactive Messaging. Once it is done, we would like to assess other specs to see how best to integrate with Reactive Messaging.

You also mentioned blocking IO for database management. At the moment, the project R2DBC is trying to come up with a solution to do non-blocking IO against Relational DBs. I am afraid only after well-known DBs have adopted R2DBC, we can then work with JPA to create reactive JPA perhaps.

Thanks

Emily

[Nils Henrik Lorentzen]

Thanks, I'll follow the progress on those DB interfaces.

The people at jnosql.org for NoSQL storage abstraction seem to have async interfaces so that's a good start since one often will use NoSQL with micro services methinks. If having to choose a relational DB, that means model might be too highly integrated to split into micro services  in the first place (if goal of microservice is complete isolation for whole stack including persistent storage).

What one could be looking for is writing an application following a pattern like

function onHTTPRequest(request, resonse) {

    database.query("select xyz from ..", function(result)) {

          respons.write(convertToJSON(result);

    }

}

More digression: Now I see the project Loom promises lightweight threads (fibers) that, if I understand correctly, will for blocking I/O calls save the whole call stack (like with context switching or user space setjmp()/longjmp()), schedule that work async behind the scenes, free up the thread to other stuff in the mean time and resume the fiber at place of suspend when the operation is done.

A nice trick to allow for sync looking coding model while saving on number of theads, even so I don't see quite why ones run of the mill server REST or web invocation is complex enough to not handle it async like above instead. In my opinion the main issue is not that calling a continuation function is that difficult to write or maintain, it is more rather that I/O drivers has been implemented with sync interfaces by default probably because of Java's historical focus on ease to spawn and synchronise threads, and one thought "just spawn a thread for async" - which no longer holds for scalability.

Instead they should have been implemented async instead, and then easy to do async interface on top. Even the Loom proposal talks about how easy it is to adapt an async interface to a sync one, but then goes on to say that one should drop async driver/library interfaces when fibers arrive, then removing the option for people to do async/reactive (!) 

In addition to making concurrent applications simpler and/or more scalable, this will make life easier for library authors, as there will no longer be a need to provide both synchronous and asynchronous APIs for a different simplicity/performance tradeoff.

The answer is, if they do not want to maintain both, for IMO I/O library authors to always write only asynchronous APIs, and sync can easily be made on top. Remember that even request/response protocols are really asynchronous in what happens on the network wire and asynchronous API most closely matches that. The whole world is in fact async by nature, one does not give another person a task and then commence to go blocking by looking into thin air waiting till that person is done :)

Fibers are nice but one should not enforce it as the model to use, by using it as an argument to not write async I/O driver interfaces. If drivers are written async they might perhaps even have means for integration with a container NIO loop to reuse container main NIO select() loop thread), then one can just have one generic library piece of code that easily wraps it into a blocking call, or blocking-appearing in the case of fibers. Sort of what future.get() does already.

Regards,

Nils Henrik Lorentzen

[Nils Henrik Lorentzen]

More monologue :) Perhaps duplicate posting since hit send shortkey while typing

I might have to retract that on project Loom accidentially sabotaging async drivers/network interfaces.

Considering that Loom will have to store the call stack of fibers in I/O operations and restore fibers

 upon I/O completion, Loom might quite depend on calling async I/O to multiplex between multiple waiting fibers.

From Loom wiki:

The implementation of the networking APIs in the java.net and java.nio.channels  packages have as been updated so that fibers doing blocking I/O operations park, rather than block in a system call, when a socket is not ready for I/O. When a socket is not ready for I/O it is registered with a background multiplexer thread. The fiber is then unpacked when the socket is ready for I/O. These same blocking I/O are also updated to support cancellation. If a fiber is cancelled while in a blocking I/O operation then it will abort with an IOException.

Thus Loom could be more of a guarantor for async APIs to external systems than making them fall out of style. Will as many have mentioned replace async higher level interfaces though.

Should hold true for JDBC too, if is to be useful for server apps.

Example:

Fiber fiber1 = new Fiber1(() -> new URL("http://foo.com").openConnection());
Fiber fiber2 = new Fiber1(() -> new URL("http://bar.com").openConnection());

fiber1.start();
fiber2.start();



Fibers are like threads but not quite. They are running on the context of a thread but once in a blocking I/O operation, they would have to store the whole stack (and CPU registers?) so that the thread they run on can continue doing something else, while the I/O completes. The completion of I/O would I suppose need to happen with async I/O, otherwise there would not be much gain, one cannot spawn a new thread to complete the I/O.

So openConnection implementation would be something like this below perhaps, correct me if wrong



URLConnection openConnection() {

  final Fiber fiber = getCurrentFiber();
  final URLConnection result;

  // Called from within fiber?
  if (fiber != null) {

     // simple value holder to store value
     final Value<URLConnection> connection = new Value<>();

     // Schedule connect operation asynchronously
     scheduleOpenConnectionAsynchronously(urlConnection -> {
         // store result so can retrieve from calling scope
         connection.value = urlConnection

         // resume fiber from below yield()
         fiber.resume();
     });


    // saves the current call stack and adds to Fiber scheduler
    fiber.yield();
    
    // here after resume(), scheduler has restored stack and registers.
    // note that all this happens on the same thread, which is what allows for better performance
    result = connection.value;
  }
  else {
     result = alreadyExistingBlockingImplementation();
  }

  return result;
}



[clement escoffier]

Hello,

Thanks for this great write-up! 

We have been discussing these topics in the reactive hangout (happening every Tuesday at 11:00AM Paris time zone). You are more than welcome to join and work with us there. Let us know if the time does not work for you. 

As Emily said, we are moving towards the first version of Reactive Messaging, that would provide the base of message-driven / reactive model. What you described is something we had been covering a little bit in the current spec. Typically, @Blocking (not part of the spec) is something I want to experiment with in the SmallRye implementation (I like the constraint idea). 

About Loom, you are right, Loom does not remove the need to Async, it just eases user code as they would be able to write “imperative” code (synchronous) and execute it in an asynchronous-manner. But for this, it relies on non-blocking IO. I heavily recommend the talk from Martin Thomson about interaction protocols. I have the chance to see it last week at JBCN, and his take on Loom is the best metaphor ever. 

Clement

-- 
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/b711a145-607c-467c-9b58-e6bc2193eba0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nils Henrik Lorentzen]

Hi,

Not sure I have that much more to contribute, more just my general thoughts. I can chime in at that time if I come up with more.

There are indeed use cases that are reactive as such, telco is an example where events for the same call may happen simultaneously. For these, project loom would not necessarily solve the reactiveness of it, or at least not very elegantly, sent an email to the loom-dev list about that since they would think mostly of that imperative case.

Even though admittedly I haven’t written a single line of Erlang code myself, I’d recommend looking at its runtime model and also its error handling. The lightweight processes (actors) there are much like operating system processes but with only software isolation. I guess Akka does similar.

However errors propagate upwards like in OS processes and clean up properly. Not sure how applicable, could give a different perspective though.   

The constraint thing should work, use it for parallelization of my own (simple) implementation of maven build system where it builds a dependency graph and schedules according to constraint, have not been able to test it thoroughly yet though. Maven have all of these constraints (compile, local file assembly, download dependencies).

Regards,

Nils Henrik Lorentzen

mandag 3. juni 2019 15.54.49 UTC+2 skrev clement escoffier følgende:

Hello,

Thanks for this great write-up! 

We have been discussing these topics in the reactive hangout (happening every Tuesday at 11:00AM Paris time zone). You are more than welcome to join and work with us there. Let us know if the time does not work for you. 

As Emily said, we are moving towards the first version of Reactive Messaging, that would provide the base of message-driven / reactive model. What you described is something we had been covering a little bit in the current spec. Typically, @Blocking (not part of the spec) is something I want to experiment with in the SmallRye implementation (I like the constraint idea). 

About Loom, you are right, Loom does not remove the need to Async, it just eases user code as they would be able to write “imperative” code (synchronous) and execute it in an asynchronous-manner. But for this, it relies on non-blocking IO. I heavily recommend the talk from Martin Thomson about interaction protocols. I have the chance to see it last week at JBCN, and his take on Loom is the best metaphor ever. 

Clement

To unsubscribe from this group and stop receiving emails from it, send an email to microp...@googlegroups.com.