Towards MicroProfile 'Reactive Systems'
130 views
Skip to first unread message
Gordon Hutchison
Feb 5, 2019, 10:59:37 AM2/5/19
to Eclipse MicroProfile
Towards MicroProfile 'Reactive Systems'
If one considers the core message of the reactive manifesto
as expressed, for example in https://www.reactivemanifesto.org/images/reactive-traits.svg
We want to build systems that scale up and down dependant on demand in order to maintain
responsiveness and we want to be able to do this by having more servers not bigger ones,
sometimes termed scaling 'out' rather than up. Dividing up any service provision into members
of a cluster also limits the impact of any failure compared to having one large server engine.
It also provides a relatively easy way to dynamically size the service by adding or removing
cluster members.
To get the benefits of a clustered environment of micro-services we need to have weak affinity
between any particular bit of workload and a particular service instance. This means that
the originator of a request and the processor of it should be loosely coupled - both in 'space'
(not knowing which service instance is the final destination) and in 'time' (in order to be able
to tolerate the delays caused by service fail-over, dynamic provisioning and so on).
Ideally, as we now treat individual server failures, dynamic provisioning events, and so on as 'normal',
we don't want to 'block', hogging server resources, while waiting for any remote service
to respond. We prefer to have the requests handled asynchronously. This also helps constrain
the stutter of scaling and failure events from spreading across the whole system.
So we want inter-service requests be able to be routed, queued, recovered and re-delivered if needed
after failure. Traditionally this has been done by expressing inter-service requests as
messages, buffering them in queues and handling them asynchronously.
The reactive manifesto correctly identifies having inter-service events as messages as being just a
'means' to the 'forms' of elastic and resilient systems which remain responsive.
Customers are increasingly working with architectural patterns that build on top of this world view,
'Event Sourcing', 'CQRS' and so on, we might refer to the messages as events or the queues as streams
but principles endure.
However, we now see concerns (scaling, fault-tolerance etc.) that do not really belonging
to the business logic layer impacting the programming model application developers use.
But what alternative is there?
Istio?
So, in Istio, in the absence of using asynchronous messages:
1) We want some means to route a service request [check].
2) We want some means to re-route and re-deliver a service request in response to initial failure [check].
3) We want some means to monitor cross server performance and choke/back-pressure the workload ingress to a service if needed [MAYBE?].
4) We want to have non blocking asynchronous outgoing requests, to free up resources/threads [NOT SO MUCH]
5) We want to be able to use that freed up thread to do another piece of work in the interim and handle the response on a different thread if desired [NOT SO MUCH]
So what interest is this to the MicroProfile community?
Well, to enable 3 above, between a client-server pair, short of choking the work going into the upstream client
(which just moves the problem one hop upstream), the only way to flow-control requests coming out of the upstream client
and into a server instance is to send those request to a different server instance.
Istio CAN do the equivalent of cross client fan-in limiting for a single service instance
(cf Rsocket lease) as it 'sees' all the users of a service and how stressed it is.
Ultimately though, we are trying to deal with the situation where work-coming-in > work-that-can-be-done
so any scale-out will eventually run out of runway. If the problem is 'bursty' traffic we can buffer and smooth
(use messages/event queues to smooth perhaps). But if the stress is persistent, using the egress sidecar to 'block'
requests will surely result in timeouts at the application process which ultimately causes wasted work and
overall just decreases the efficiency of the system - sometimes creating a nasty 'tipping-point'.
A clean approach is to have some form of asynchronous outbound 'reactive' inter-server mechanism
that can pass back-pressure transitively up-stream.
So ultimately, to avoid using Kafka equivalents in 'too many' places,
inter-server asynch request sending/receiving with 'back pressure'.
I do not know if that means an addition to an existing spec
like a JAX-RS
```
public interface RxMpPublisherBuilderBosh extends RxInvoker<PublisherBuilder>
```
or if we need something added to the MicroProfile type safe rest work?
Gordon
(PS. I have not addressed 5. above on purpose. The ability to run work on any thread
(and thus perhaps a particular thread to reduce the need for locks by data-thread affinity)
is an interesting problem but I think that is probably left to individual implementations
for now, or perhaps a future MP Concurrency discussion, or at least, a different thread.)
If one considers the core message of the reactive manifesto
as expressed, for example in https://www.reactivemanifesto.org/images/reactive-traits.svg
We want to build systems that scale up and down dependant on demand in order to maintain
responsiveness and we want to be able to do this by having more servers not bigger ones,
sometimes termed scaling 'out' rather than up. Dividing up any service provision into members
of a cluster also limits the impact of any failure compared to having one large server engine.
It also provides a relatively easy way to dynamically size the service by adding or removing
cluster members.
To get the benefits of a clustered environment of micro-services we need to have weak affinity
between any particular bit of workload and a particular service instance. This means that
the originator of a request and the processor of it should be loosely coupled - both in 'space'
(not knowing which service instance is the final destination) and in 'time' (in order to be able
to tolerate the delays caused by service fail-over, dynamic provisioning and so on).
Ideally, as we now treat individual server failures, dynamic provisioning events, and so on as 'normal',
we don't want to 'block', hogging server resources, while waiting for any remote service
to respond. We prefer to have the requests handled asynchronously. This also helps constrain
the stutter of scaling and failure events from spreading across the whole system.
So we want inter-service requests be able to be routed, queued, recovered and re-delivered if needed
after failure. Traditionally this has been done by expressing inter-service requests as
messages, buffering them in queues and handling them asynchronously.
The reactive manifesto correctly identifies having inter-service events as messages as being just a
'means' to the 'forms' of elastic and resilient systems which remain responsive.
Customers are increasingly working with architectural patterns that build on top of this world view,
'Event Sourcing', 'CQRS' and so on, we might refer to the messages as events or the queues as streams
but principles endure.
However, we now see concerns (scaling, fault-tolerance etc.) that do not really belonging
to the business logic layer impacting the programming model application developers use.
But what alternative is there?
Istio?
So, in Istio, in the absence of using asynchronous messages:
1) We want some means to route a service request [check].
2) We want some means to re-route and re-deliver a service request in response to initial failure [check].
3) We want some means to monitor cross server performance and choke/back-pressure the workload ingress to a service if needed [MAYBE?].
4) We want to have non blocking asynchronous outgoing requests, to free up resources/threads [NOT SO MUCH]
5) We want to be able to use that freed up thread to do another piece of work in the interim and handle the response on a different thread if desired [NOT SO MUCH]
So what interest is this to the MicroProfile community?
Well, to enable 3 above, between a client-server pair, short of choking the work going into the upstream client
(which just moves the problem one hop upstream), the only way to flow-control requests coming out of the upstream client
and into a server instance is to send those request to a different server instance.
Istio CAN do the equivalent of cross client fan-in limiting for a single service instance
(cf Rsocket lease) as it 'sees' all the users of a service and how stressed it is.
Ultimately though, we are trying to deal with the situation where work-coming-in > work-that-can-be-done
so any scale-out will eventually run out of runway. If the problem is 'bursty' traffic we can buffer and smooth
(use messages/event queues to smooth perhaps). But if the stress is persistent, using the egress sidecar to 'block'
requests will surely result in timeouts at the application process which ultimately causes wasted work and
overall just decreases the efficiency of the system - sometimes creating a nasty 'tipping-point'.
A clean approach is to have some form of asynchronous outbound 'reactive' inter-server mechanism
that can pass back-pressure transitively up-stream.
So ultimately, to avoid using Kafka equivalents in 'too many' places,
we need some, perhaps MicroProfile, spec material to cover:
I do not know if that means an addition to an existing spec
like a JAX-RS
```
public interface RxMpPublisherBuilderBosh extends RxInvoker<PublisherBuilder>
```
or if we need something added to the MicroProfile type safe rest work?
or something else?
(PS. I have not addressed 5. above on purpose. The ability to run work on any thread
(and thus perhaps a particular thread to reduce the need for locks by data-thread affinity)
is an interesting problem but I think that is probably left to individual implementations
for now, or perhaps a future MP Concurrency discussion, or at least, a different thread.)
Kevin Sutter
Feb 5, 2019, 4:14:31 PM2/5/19
to Eclipse MicroProfile
Gordon,
That sounds like a blog post... There are some good ideas and you are looking for a discussion... Maybe we would get some more outside contributions if it was a blog post. You might have to re-word portions of it, but for the most part, it would make a good blog. Just an idea...
-- Kevin
Ken Finnigan
Feb 7, 2019, 9:25:01 AM2/7/19
to MicroProfile
I'm currently experimenting with developing a Reactive REST Client in SmallRye, with the goal of feeding necessary spec changes to MicroProfile.
I believe JAX-RS has some functionality for being Reactive, and I know RESTEasy has taken that further to support Publisher and Single from Rx.
But in terms of true reactive system across the entirety of a stack, with the likes of Istio that surely requires more than just MicroProfile?
It would also require the Envoy proxy sidecar to be reactive as well. Otherwise a client can be reactive to the sidecar, but then the sidecar is blocking to the server. At least that's my current understanding.
Ken
--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/875a40b0-4534-4853-8f6d-ae4df179ab85%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Gordon Hutchison
Feb 8, 2019, 6:09:46 AM2/8/19
to Eclipse MicroProfile
Hi Ken,
One can see RESTEasy's
public interface FlowableRxInvoker extends RxInvoker<Flowable<?>>
and its Impl etc.
I think the first work in an MP impl would be to 'lift' the equivalent to make a SubscriberBuilder.
Off the top of my head I can't think of any spec work in just doing that.
Regarding the back pressure through the envoy, the RESTEasy code uses SSE
(javax.ws.rs.sse.InboundSseEvent, see ::eventSourceToObservable etc.)
(There is something interesting here too https://dzone.com/articles/micronaut-mastery-consuming-server-sent-events-sse )
I could not find any data (and I have no original expertise or experience to add :-) ) on
the combination of SSE through Envoy.
Gordon
John Clingan
Feb 8, 2019, 2:03:39 PM2/8/19
to Eclipse MicroProfile
We've been in "thinking big" recently (Platform spec, growing pains, MP backwards compatibility, etc). We've also been moving forward on a lot of async/reactive discussions and work over the last 6 months - reactive streams specs, async relational database access, Reza's Axon/CQRS/Event Sourcing presentations, this discussion (including Ken's Reactive MP Rest Client R&D), and probably more.
It's getting to the point where we need a formal reactive/async strategy as opposed a collection of point specs/solutions/discussions.
Thoughts?
Ondro Mihályi
Feb 9, 2019, 3:57:19 AM2/9/19
to Eclipse MicroProfile
I agree with John, we need to think about reactive support in a bigger picture than just in individual specs.
Reactive programming is optional for many developers and therefore shouldn't be required for all specs to support it.
What about creating a separate "reactive" spec for a "reactive profile", which would specify how other specs work together and some extwnsions and cross-concern functionality on top of it.
I have these areas in mind to cover:
* list of specs that contain support for reactive programming and are designed to work well together (e.g.REST client, JAX-RS, reactive operators, Fault tolerance, future messaging, ...)
* how the specs work well together (e.g. all allow using reactive strewams and MP reactive operators)
* extensions to existing specs where it's possible and doesn't make much sense to add the requirements to existing specs
** e.g. specify additional default JAX-RS invoker provider for MP reactive operators (we can't change JAX-RS spec now)
** maybe support for SubscriberBuilder in MP REST client methods can go here if we don't want to put it into the spec itself, to make the REST client spec easier to implement alone (the same for Fault Tolerance interceptors). This idea assumes that individual specs teams don't have enough reactive experts and it's hard for them to specify reactive concepts properly within their spec.
--Ondro
John Clingan
Feb 10, 2019, 1:51:52 PM2/10/19
to Eclipse MicroProfile
+1 on not being required.
We have yet to agree on "profiles" or "suites", but we are discussing it in this thread.
Mark Little
Feb 11, 2019, 5:47:06 AM2/11/19
to microp...@googlegroups.com
+1
--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/97df32d4-dfd5-414f-beee-5398927ce5c3%40googlegroups.com.
Ken Finnigan
Feb 11, 2019, 3:26:50 PM2/11/19
to MicroProfile
While I certainly agree that not every developer will want to use reactive in all, or maybe any, of there microservices.
From an implementation perspective we need to have reactive below the current imperative programming model, otherwise we're layering a non-blocking technology on top of a blocking one.
It's going to require a coordinated effort across MP specifications to support reactive at the core.
Ken
--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/c3dce6cb-b631-43cd-ab16-62cf434d1099%40googlegroups.com.
James Roper
Feb 12, 2019, 7:05:01 PM2/12/19
to MicroProfile
Hi all,
My opinion is that we need to step back a bit and create a high level plan around reactive. Supporting reactive programming is good, but I believe the most value is achieved in reactive systems.
In a monolithic architecture, people rarely talk about state, they just talk about data, because state and data are the same thing, the database offers all of your data as a single, strongly consistent state that you can interact with without needing to think about what else may be happening elsewhere in the system, and it does this by using ACID transactions.
In a cloud native architecture, we need to talk about state, because state and data are not the same thing. Each service has its own state, and that state is a projection in space and time of data - a different projection to what every other service sees. Some services will see a projection that is further in time than other services, because it takes time for operations to propagate around the system - and let me make something clear, this isn't a choice where you can choose between strong or eventual consistency, this is a fact, you cannot atomically update all your services at once in a cloud native world, there is no strong consistency in a cloud native environment, you *must* deal with it.
In a cloud native architecture, we need to talk about state, because state and data are not the same thing. Each service has its own state, and that state is a projection in space and time of data - a different projection to what every other service sees. Some services will see a projection that is further in time than other services, because it takes time for operations to propagate around the system - and let me make something clear, this isn't a choice where you can choose between strong or eventual consistency, this is a fact, you cannot atomically update all your services at once in a cloud native world, there is no strong consistency in a cloud native environment, you *must* deal with it.
So how do we deal with this? You could try manually, with recovery actions and rollbacks etc, but the reality is, in even the simplest of business processes, there are tens if not hundreds of permutations of failure and state discrepancy scenarios. Programming it manually is an uphill, error prone and ultimately frivolous exercise, that's why ACID transactions existed for monoliths, so that developers didn't have to consider consistency and failure scenarios. However ACID transactions aren't available at a system level in a cloud native environment - an individual service can use ACID transactions, but when it comes to operations that span multiple services, there are no ACID transactions. So we need a new paradigm to allow developers to develop systems without having to deal with errors. Reactive seeks to be that paradigm, it seeks to be to cloud native systems what ACID transactions are to monoliths.
The way Reactive seeks to address this is by decoupling the processing of operations between services. Rather than making chains of REST calls through services, which strongly couple those services together and require the handling of errors and inconsistencies to be done manually, Reactive promotes asynchronous communication that decouples the availability of the services involved. This asynchronous communication can be built on top of infrastructure that can guarantee message delivery and handle retries for you, so, as a developer, you don't need to handle errors. And that's the aim, push the error handling and the recovery on the infrastructure, not on the developer. Reactive and ACID transactions have that in common.
Fundamental to this at an implementation level is streams, because streams provide a clean abstraction for managing flows of asynchronous messages, ensuring that they are handled in causal order, ensuring that downstream systems can push back rather than get overwhelmed, and providing a clean architectural abstraction for understanding the flow of data through a system. This is why in the MicroProfile Reactive effort, the first thing we have created is MicroProfile Reactive Streams Operators.
Next up we need a higher level abstraction for connecting these streams to the infrastructure that can deliver messages between services, databases, components, etc, and that's why the second specification we've focused on is MicroProfile Reactive Messaging. In the first iteration, this spec is targeted to offer messaging on top of infrastructure and/or protocols like Kafka, AMQP, MQTT etc. However, that's only the beginning. There's been a lot of talk about CQRS, and CQRS is itself a streaming pattern, it decouples commands from queries, putting an asynchronous messaging fence between them, traversed by, you guessed it, streams. And so, our hope for the messaging spec is that a CQRS spec will just be another messaging provider, offering a stream of events from the database to be published to other messaging providers such as Kafka, or consumed by updating a read side.
And if we do embrace this, then we can get back to a world where the developer can rely on the infrastructure to handle failures and inconsistency for them. Unfortunately we can't have the strong consistency that ACID transactions gave us, that's a given, but we can have guaranteed eventual consistency if we embrace a streaming architecture, using streams to propagate data from databases to message queues back to databases across services etc etc.
There's been a lot of discussion about the low level programming APIs, reactive streams, asynchronous IO etc, but I hope we don't lose site of the bigger vision that these APIs exist to support.
Regards,
James
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAKeeVe5d2rSAq_XfERxoKqOv%2BouUXH7rzU_8c9YuBttD5xgxPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages