Scenarios for messaging in microservices architecture

[andrew_s...@uk.ibm.com]

I'd like to kick off a discussion specifically on the scenarios for messaging we expect to see in a microservices architecture. I don't want to get bogged down in API discussions at this point. So, I'll kick this off with a short description of one way of using messaging that I see as appropriate for microservices architectures. I've started with a view that the messaging is "in front" of the microservices, so the messages are kind of commands. I think it's also entirely sensible to view messaging as "behind" the microservices, which I'd characterise as using the messages to represent events.

Comments and additional scenarios are welcome.

Andrew Schofield

Event Services, IBM Watson and Cloud Platform

[andrew_s...@uk.ibm.com]

[SCENARIO] Worker-offload (or cooperating consumers)

Brief Description

A microservice performs work asynchronously requested using messaging. Each request is represented by a message published on a topic, a named logical channel for message distribution. The microservice consumes the messages from the topic and performs the work. Multiple instances of the microservice can cooperatively consume the messages from the topic, with each message being processed by one instance of the microservice.

In JMS terms, the cooperating consumers would use a shared durable subscription. The shared subscription acts like a queue, but with the flexibility of publish/subscribe.

By using a topic as opposed to a queue, the same messages can be used for other purposes too. For example, an auditing microservice can consume the same messages using a different subscription.

Extension: Responses to work requests

Although it increases the coupling between requester and worker, it may be desirable for the microservice to send a response to each request. For the sake of resilience, it is preferable if the consumption of responses can also be handled by any of the requesters as opposed to a single, specific requester. Otherwise, the degree of coupling between requester and worker approaches that of a synchronous call.

Extension: Message ordering and concurrency

When a single microservice is performing the work, maintaining order is simple. Once concurrent execution of multiple workers comes into play, it's much more complicated. A pragmatic way of maintaining order when appropriate but permitted as much concurrency as possible is for the requestors to indicate which streams of messages have an ordering relationship by using some kind of identifier. Then, the messaging infrastructure can ensure that messages are processed in order where it matters while maximising concurrency where it does not.

[Steve Millidge]

Playing devil's advocate here. In this scenario are you not effectively describing an asynchronous invocation of a remote service where there are a pool of processing threads either distributed or in the same process. From my reading of the scenario messaging is a logical implementation choice rather than a necessity for the scenario.

Saying that I like the idea of driving and agreeing scenarios before diving into apis.

[James Roper]

He Andrew,

Thanks for kicking off this discussion.  I'll add one of my own scenarios.

[SCENARIO] CQRS Read-Side processing of event logs

Brief Description

A common pattern for persistence being used in microservices is event sourcing, which to be used effectively must be combined with CQRS for implementing queries that span multiple entities. Often, the CQRS read-sides will live in a separate services, and so events from the event log need to be published to a message broker so that those services can consume them.

Extension: Event log offset tracking

The event log is a persistent store of the events, and the way events would be consumed out of it would typically be done using queries that streamed the events as they were persisted. In the event of failure, when resuming publishing the events, the publisher will need to know where in the event log it was up to, so that it can resume publishing from that point. It could be considered a concern of the messaging API implementation to track this, and doing so would offer a very convenient end user API.

Extension: Distribution of publishing load

In some circumstances the resources required to publish the event log may be greater than what one single node can handle.  This could either be due to very high throughput on the event log, or it could also be due to the work required to transform the event log messages to messages to be published publicly do the message broker being expensive. In such circumstances, the event log may be partitioned, and the work of publishing the event log can then be sharded out to multiple nodes.  The messaging API implementation should be capable of distributing this work across multiple nodes.

[James Roper]

On Thursday, December 15, 2016 at 9:58:10 PM UTC+11, andrew_s...@uk.ibm.com wrote:

[SCENARIO] Worker-offload (or cooperating consumers)

Brief Description

A microservice performs work asynchronously requested using messaging. Each request is represented by a message published on a topic, a named logical channel for message distribution. The microservice consumes the messages from the topic and performs the work. Multiple instances of the microservice can cooperatively consume the messages from the topic, with each message being processed by one instance of the microservice.

In JMS terms, the cooperating consumers would use a shared durable subscription. The shared subscription acts like a queue, but with the flexibility of publish/subscribe.

By using a topic as opposed to a queue, the same messages can be used for other purposes too. For example, an auditing microservice can consume the same messages using a different subscription.

Extension: Responses to work requests

Although it increases the coupling between requester and worker, it may be desirable for the microservice to send a response to each request. For the sake of resilience, it is preferable if the consumption of responses can also be handled by any of the requesters as opposed to a single, specific requester. Otherwise, the degree of coupling between requester and worker approaches that of a synchronous call.

From what I've encountered, services tend to interact with messaging APIs in one of three ways - as sources, as sinks, or as transformers.  The initial use case you've described is of a sink (with an implied source somewhere).  This extension though, rather than viewing it as a sink of requests that sends responses, I think in many cases it may be better to view it as a (possibly side effecting) transformation of messages.  While the two things are basically describe the same process (we're just transforming requests to responses), if you think of processing a message in terms of a transformation where service A emits a stream of messages, service B transforms them, emitting a new stream, and service A consumes that new stream, the requirement for messages to be able to be handled by any node becomes implicit.  That is to say, take the "request/response" concept out of there and you get something that feels more natural and is implicitly resilient, and I think using this terminology will help guide developers to implement their services in such a way.

So for example, the order service emits messages to say that an order has been submitted.  A payment service will process these, and will emit success or failure messages, which will be consumed by the order service.  Effectively a request/response has happened, but by not using the request/response terminology, developers will naturally think more towards location independent messaging.

Extension: Message ordering and concurrency

When a single microservice is performing the work, maintaining order is simple. Once concurrent execution of multiple workers comes into play, it's much more complicated. A pragmatic way of maintaining order when appropriate but permitted as much concurrency as possible is for the requestors to indicate which streams of messages have an ordering relationship by using some kind of identifier. Then, the messaging infrastructure can ensure that messages are processed in order where it matters while maximising concurrency where it does not.

I think this requirement is going to be across the board for almost all scenarios.

[Ondrej Mihályi]

Hi,

...if you think of processing a message in terms of a transformation where service A emits a stream of messages, service B transforms them, emitting a new stream, and service A consumes that new stream, the requirement for messages to be able to be handled by any node becomes implicit. 

I like the transformer analogy. I can imagine that the concept is enhanced to include a notion of routing messages, where a request is routed through multiple transformers between a source and a sink. Then request-response can be expressed by a pipeline where the sink is the same as the source. However, if the sink is a public endpoint accessible by a synchronous interface (public REST service or a web frontend) or stateful protocol (SSE or WebSocket), it is not easy to ensure that the messages can be handled by any source node. It is necessary that the messages are routed to the very node that has access to the context of the original request. Am I wrong?

An example: Service A is a REST endpoint, service B is a transformer, that consumes messages from A and emits transformed messages back to A. If a client issues a REST call against service A, the client expects a response from the same node. The service A sends a message to B, which emits a message for A. If the final message is received by any node of the service A, it can only send the REST response if it's the original node. If it's not the original node, it does not have access to the REST client and must forward the message to the original node, which can send the REST response back to the client.

--Ondrej

Dňa piatok, 16. decembra 2016 1:50:05 UTC+1 James Roper napísal(-a):

[Ondrej Mihályi]

[SCENARIO] Flow-control and back pressure

Description - flow control:

Service A needs to retrieve a list of entities from service B and filter it. It only needs first N of the filtered entities. Clearly, the number of entities to retrieve from the service B is unknown at the beginning. Service A issues a message to start retrieval of the entities. Service B will start streaming messages, one for each entity. Service A consumes the messages from B, filters the data. When service A receives enough entities that match the filter, it will ignore subsequent messages. At this point, service B needs to be notified that no further messages will be consumed and therefore it may stop streaming data. Service A emits a "terminate" message to inform B that no more data is needed.

A similar scenario applies when the initial request is canceled by the user/client (e.g. when the browser navigates away from the current page, which is listening to SSE events).

Description - back pressure:

A similar situation, but the necessity to control the flow comes from the consumer (service B). When service B is slow to consume messages, the service B (or the message broker in case that it detects too many events in the queue) might inform service A about that. The service A may react to this, e.g. by slowing down the pace of producing messages or propagating the information upstream. If combined with flow control, service A could decrease the number of entities it needs and emit a "terminate" event earlier to relive pressure from service B.

--Ondrej

[James Roper]

On 17 December 2016 at 09:27, Ondrej Mihályi <ondrej....@gmail.com> wrote:

Hi,

...if you think of processing a message in terms of a transformation where service A emits a stream of messages, service B transforms them, emitting a new stream, and service A consumes that new stream, the requirement for messages to be able to be handled by any node becomes implicit. 

I like the transformer analogy. I can imagine that the concept is enhanced to include a notion of routing messages, where a request is routed through multiple transformers between a source and a sink. Then request-response can be expressed by a pipeline where the sink is the same as the source. However, if the sink is a public endpoint accessible by a synchronous interface (public REST service or a web frontend) or stateful protocol (SSE or WebSocket), it is not easy to ensure that the messages can be handled by any source node. It is necessary that the messages are routed to the very node that has access to the context of the original request. Am I wrong?

That's correct. The use cases I was thinking of specifically targeting were ones that support asynchronous messaging - asynchronous in the sense that it does not require the sending and receiving ends to be participating in the communication at the same time.  Generally, this implies the use of a message broker to allow for the temporal decoupling of services, but there may perhaps be other possibilities that allow for asynchronous messaging without the use of a message broker.

The same effect can be achieved if the sending side keeps a persistent record of its messages and the receiving side polls for new messages using REST or websockets, keeping track of what it has consumed, and perhaps such an implementation could be provided underneath the same API as is used by a message broker, certainly in the case of an event sourcing where the messages are already persisted, it would not be hard to do this.  But even in that context, there is no guarantee that the same node will handle the response messages - if a failure occurs that's one example, but lets say in the case of using an event log as a source of messages, generally the node that emits the message is not the node that will then publish the message (in Lagom, these are completely disconnected, it is only chance that it would be the same node, that chance being 1/n where n is the number of nodes), and so it would not be likely that the same node that sent the original request will receive the response.

An example: Service A is a REST endpoint, service B is a transformer, that consumes messages from A and emits transformed messages back to A. If a client issues a REST call against service A, the client expects a response from the same node. The service A sends a message to B, which emits a message for A. If the final message is received by any node of the service A, it can only send the REST response if it's the original node. If it's not the original node, it does not have access to the REST client and must forward the message to the original node, which can send the REST response back to the client.

I would consider a chaining of requests like that to be an anti-pattern.  Microservices should, as much as practical, be autonomous.  A system of services where services call other services in order to fulfill their own requests is better described as a distributed monolith, and in fact behaves worse than a monolith since all the network hops between services add latency and higher error rates.  Essentially you have a system with all the drawbacks in terms of lack of consistency, increased deployment complexity, and increased communication complexity, with few to none of the technical benefits that microservices can bring.  Service A should be able to handle a request without calling out to service B.  That may require denormalization/duplication of data, and it also may mean that when service A responds, rather than responding with the result of the operation, it will respond to say that the operation was submitted, and require the client to be asynchronously notified of success.  More generally what I've found is if a service does need to call out to another service to fulfill its requests, then responsibilities of and boundaries between those services have not been appropriately set, and some rearchitecture is necessary.

Of course, sometimes it's simply not practical to decouple services in that way, these architectural decisions are always about trade-offs, and sometimes the disadvantages of synchronous coupling between services does not the outweigh the advantages.  In those cases I'd say asynchronous messaging is inappropriate, and synchronous REST calls should be used instead, but I think that's beyond the scope of this discussion - we already have JAX-RS to do REST.

 

--Ondrej

Dňa piatok, 16. decembra 2016 1:50:05 UTC+1 James Roper napísal(-a):

On Thursday, December 15, 2016 at 9:58:10 PM UTC+11, andrew_s...@uk.ibm.com wrote:

[SCENARIO] Worker-offload (or cooperating consumers)

Brief Description

A microservice performs work asynchronously requested using messaging. Each request is represented by a message published on a topic, a named logical channel for message distribution. The microservice consumes the messages from the topic and performs the work. Multiple instances of the microservice can cooperatively consume the messages from the topic, with each message being processed by one instance of the microservice.

In JMS terms, the cooperating consumers would use a shared durable subscription. The shared subscription acts like a queue, but with the flexibility of publish/subscribe.

By using a topic as opposed to a queue, the same messages can be used for other purposes too. For example, an auditing microservice can consume the same messages using a different subscription.

Extension: Responses to work requests

Although it increases the coupling between requester and worker, it may be desirable for the microservice to send a response to each request. For the sake of resilience, it is preferable if the consumption of responses can also be handled by any of the requesters as opposed to a single, specific requester. Otherwise, the degree of coupling between requester and worker approaches that of a synchronous call.

From what I've encountered, services tend to interact with messaging APIs in one of three ways - as sources, as sinks, or as transformers.  The initial use case you've described is of a sink (with an implied source somewhere).  This extension though, rather than viewing it as a sink of requests that sends responses, I think in many cases it may be better to view it as a (possibly side effecting) transformation of messages.  While the two things are basically describe the same process (we're just transforming requests to responses), if you think of processing a message in terms of a transformation where service A emits a stream of messages, service B transforms them, emitting a new stream, and service A consumes that new stream, the requirement for messages to be able to be handled by any node becomes implicit.  That is to say, take the "request/response" concept out of there and you get something that feels more natural and is implicitly resilient, and I think using this terminology will help guide developers to implement their services in such a way.

So for example, the order service emits messages to say that an order has been submitted.  A payment service will process these, and will emit success or failure messages, which will be consumed by the order service.  Effectively a request/response has happened, but by not using the request/response terminology, developers will naturally think more towards location independent messaging.

Extension: Message ordering and concurrency

When a single microservice is performing the work, maintaining order is simple. Once concurrent execution of multiple workers comes into play, it's much more complicated. A pragmatic way of maintaining order when appropriate but permitted as much concurrency as possible is for the requestors to indicate which streams of messages have an ordering relationship by using some kind of identifier. Then, the messaging infrastructure can ensure that messages are processed in order where it matters while maximising concurrency where it does not.

I think this requirement is going to be across the board for almost all scenarios.

--
You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/0yYyOdVrZ_c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/01c4cbac-4340-499d-8628-e1721d6e7e1b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

James Roper
Software Engineer

Lightbend – Build reactive apps!
Twitter: @jroper

[Ondrej Mihályi]

Hi,

James Roper wrote:

On 17 December 2016 at 09:27, Ondrej Mihályi <ondrej....@gmail.com> wrote:

However, if the sink is a public endpoint accessible by a synchronous interface (public REST service or a web frontend) or stateful protocol (SSE or WebSocket), it is not easy to ensure that the messages can be handled by any source node. It is necessary that the messages are routed to the very node that has access to the context of the original request. Am I wrong?

That's correct. The use cases I was thinking of specifically targeting were ones that support asynchronous messaging - asynchronous in the sense that it does not require the sending and receiving ends to be participating in the communication at the same time.  

I admit that the example with a REST service is not a good pattern, because it has to wait synchronously to receive and process the message. A better example of what I meant was a browser receiving updates through WebSocket/SSE. It is necessary to send messages back to the browser through the same node which holds the session with the browser.

 

An example: Service A is a REST endpoint, service B is a transformer, that consumes messages from A and emits transformed messages back to A. If a client issues a REST call against service A, the client expects a response from the same node. The service A sends a message to B, which emits a message for A. If the final message is received by any node of the service A, it can only send the REST response if it's the original node. If it's not the original node, it does not have access to the REST client and must forward the message to the original node, which can send the REST response back to the client.

I would consider a chaining of requests like that to be an anti-pattern.  Microservices should, as much as practical, be autonomous.  A system of services where services call other services in order to fulfill their own requests is better described as a distributed monolith, and in fact behaves worse than a monolith since all the network hops between services add latency and higher error rates.  

I agree here that you might be right that a REST service should be autonomous and synchrnous waiting for a message can be considered an anti-pattern. But as I mentioned above, I would be more interested in supporting sending back data to a browser, which has to be done through the very node which holds the session with the browser. If a message is received by another service node, it cannot send an update to the browser. 

This could still be implemented with a topic by filtering of broadcast messages to receive only the messages that the node can forward to the browser. Other nodes would just simply ignore messages they cannot process, with the overhead of sending multiple messages instead of one. But that would imply at least a browser-session-id or request-corelation-id to match the messages with a node that has an open channel to the browser. Idea of a request-corelation-id has been discussed here and I think it is very valid for messaging: https://groups.google.com/d/msg/microprofile/fSnfHmVb_JA/9WpBoPPtFgAJ

In result, I think there is a use case to also address nodes of a service rather than just a service as a whole. The routing doesn't have to be implemented in the broker (a simple broadcast and filtering could do), but an API should make it easy to deliver a message to the very node that is interested in the message and not only to a random node of a service.

--Ondrej

[andrew_s...@uk.ibm.com]

Hi,

On Friday, December 16, 2016 at 11:00:17 PM UTC, Ondrej Mihályi wrote:

[SCENARIO] Flow-control and back pressure

Description - flow control:

Service A needs to retrieve a list of entities from service B and filter it. It only needs first N of the filtered entities. Clearly, the number of entities to retrieve from the service B is unknown at the beginning. Service A issues a message to start retrieval of the entities. Service B will start streaming messages, one for each entity. Service A consumes the messages from B, filters the data. When service A receives enough entities that match the filter, it will ignore subsequent messages. At this point, service B needs to be notified that no further messages will be consumed and therefore it may stop streaming data. Service A emits a "terminate" message to inform B that no more data is needed.

A similar scenario applies when the initial request is canceled by the user/client (e.g. when the browser navigates away from the current page, which is listening to SSE events).

 

I find this scenario surprising and a bit dangerous. If service A just requests an unbounded sequence of messages and then decides it's had enough, service B might have done a heap of wasted work and queued up a vast number of messages. You could try to get the consumption rate of service A to influence the production rate of service B, kind of lazy evaluation across a network, but service B needs to be quite sophisticated to be efficient and capable of lazily generating more data. I think it would be a lot better to do the filtering locally in service B.

Do you see support for browser/client disconnection as an important scenario to support for microservices messaging? For example, in the MQTT protocol, you can have a special message called a "will message" publishes when a client disconnects, thus enabling other clients to respond to the disconnect.

I suppose the bigger question is whether the notion of a messaging session is important for microservices. My view is that this concept should be avoided.

[James Carman]

Sounds like Apache Camel to me.

--


You received this message because you are subscribed to the Google Groups "MicroProfile" group.


To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

[Werner Keil]

Please don't mention all those other (Apache) projects here ;-)

Just kidding, at least for my part.

[Ondrej Mihályi]

Yes, something like Camel, but distributed across multiple services and without the centralized configuration of routes.

We're taking here about a pattern/usecase, which could be implemented using messaging. Camel is already an implementation of integration patterns, but might not match the requirements of microservice architectures. I don't know Camel too much, but I think it is a component running in a single JVM, communicating with other services, and it cannot be distributed. Correct me if I'm wrong.

--Ondrej

Dňa utorok, 3. januára 2017 13:36:45 UTC+1 James Carman napísal(-a):

[Ian Robinson]

I'd like to revive this discussion and wonder if it makes sense to start with something low-level, concrete and in common use in many frameworks, like the model provided by Kafka, and use that model as a basis to help identify the main scenarios that would benefit from a simpler API for Java microservices. Andrew is going to take a crack at that here, based on some of the work his team have been doing in IBM, to see if we can define a useful scope for a modern messaging API for MicroProfile and to get this discussion going again.

..Ian

[andrew_s...@uk.ibm.com]

My view is that Apache Kafka makes a really good starting point for messaging for MicroProfile. I wouldn't suggest that MicroProfile would adopt the Kafka APIs directly. For one thing, I've seen plenty of people struggle with the APIs and the myriad configuration options. It does however give a nice, concrete example that's used widely.

 

I would start with the Kafka Streams API (https://kafka.apache.org/documentation/streams) which is a client library for processing streams of messages. A Kafka Streams application is a long-running program that processes streams of messages. The processing logic of a Kafka Streams application consists of a processor topology - basically a directed graph of processor nodes. At the incoming edge of the topology are source processors which read from topics, and at the outgoing edge are sink processors which write to topics. The intervening processor nodes can filter, map, transform, or whatever.

 

I think this is at a much nicer level of abstraction than the Kafka Consumer API. In that case, you actually make calls to poll for batches of messages. Much nicer to define the code to handle a message and have the library worry about getting the messages.

 

You can run multiple instances of a Kafka Streams application on a multi-partition topic and the partitions are distributed among the running instances. If an instance crashes, the remaining instances take over that instance's partitions. So, you can scale it up, do rolling upgrades and so on. That's great, but I consider it to be a detail of the implementation, rather than part of a messaging API.

 

Kafka Streams includes the idea that a stream of events can be viewed as a table, and vice versa. That's really nice, but I suggest putting that aside an looking at the KStream part of the API.

 

If you are taking a microservices approach and want to use messaging to wire it all together, I think Kafka Streams is a good place to start.

[James Roper]

One big problem that we've had to solve with regards to Kafka, is that sure, once you get a message into Kafka, it's guaranteed to get to consumers at least once, and subsequent stream processing stages will also guarantee at least once processing, but how do you guarantee that a message gets into Kafka in the first place at the producer end? For example, I have to make a database update somewhere, and if that update is successful, I have to publish a message to Kafka. What if the update succeeds, and Kafka is down? Move publishing to inside the database transaction? What if the database transaction fails to commit?

I think a general purpose messaging API for technologies like Kafka must solve this problem. We solve it in Lagom by using event logs in the database, and then publishing from event logs and tracking the offset of messages published (essentially, we do it exactly the same way that Kafka itself handles at least once delivery). Making users do this themselves I think is unnecessarily complex and prone to errors, there's a lot of failure and distribution concerns that need to be addressed to do it well.

With regards to generic stream processing, there are already a number of good generic stream processing APIs out there, for example rxjava and akka streams, these provide all the map, filter, transform, fan in, fan out etc features that you could want.  I don't think it would be wise to create yet another generic stream processing API just for the purpose of abstracting over Kafka - instead, I think whatever is provided should be based on reactive streams, since then any of these technologies can be utilised, and not only that, but it means that any reactive streams compliant sink/source could participate in the underlying stream graph, you wouldn't be restricted for example to a graph that contained nothing but Kafka inputs and outputs, you could have kafka going into rabbitmq, you could merge a mongodb streamed query with a websocket stream in Play Framework to feed Kafka, etc, and this would all come for free if the proposal was based on reactive streams, no need to write adapters or anything, because the adapters already exist.

--

You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/0yYyOdVrZ_c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/92e668f4-e61e-4b8b-9fbe-32e79c0d4246%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[andrew_s...@uk.ibm.com]

I think you expressed it better than me. I don't want to invent a new stream processing API, but if we can feed messages into and out of existing frameworks in an elegant way, that would be good. Reactive Streams looks like an interesting option.

I don't agree about trying to span transactions across resource managers, which I think is the implication of trying to ensure a message is published if a database update succeeds. I might try to ensure that a synchronous REST API is called at least once successfully on every database update. I might try to ensure that a message is published successfully on every database update. I might want to call a bunch of webhooks when an action occurs. I view these as essentially equivalent to each other, but the middle one happens to be using messaging so the eventual recipients of the events can be running independently. It seems appropriate for a framework such as Lagom to help here, and you've chosen to put event logs in the database as a way of achieving this. We do similar things in our cloud storage services, basically building the ability to notify into the storage service and remembering the intent to notify along with the data.

Another way to achieve an integration between a database and Kafka is to treat the database as an event stream, feed the database changelog into a compacted topic, and then have event subscribers simply read that topic. That's really taking advantage of a very Kafka-specific feature, but it gives a pretty elegant solution which fits nicely with microservices. I suggest that a MicroProfile messaging API could be used here to implement the subscribers of the compacted topic without really understanding the existence of the database or even the compaction of the topic. The messages would have to have two parts - key and value - and an empty value would signify a deletion. 

Any suggestions for next steps?

Andrew Schofield

IBM Watson and Cloud Platform

To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/92e668f4-e61e-4b8b-9fbe-32e79c0d4246%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[James Roper]

On 5 June 2017 at 11:01, <andrew_s...@uk.ibm.com> wrote:

I think you expressed it better than me. I don't want to invent a new stream processing API, but if we can feed messages into and out of existing frameworks in an elegant way, that would be good. Reactive Streams looks like an interesting option.

I don't agree about trying to span transactions across resource managers, which I think is the implication of trying to ensure a message is published if a database update succeeds.

Yes, I'm definitely not implying distributed transactions, but rather whether there's a common API that can be extracted for the various techniques you've mentioned below.  Maybe there's not - I obviously have experience with the event log scenario, but I'd be interested in knowing the mechanics of the other scenarios to understand whether there are any commonalities between them.  And even if they are all very specific to the various solutions, my thought is that the best practice for ingesting data into the messaging API would be to use one of these techniques, and this assumption should guide what APIs get offered and how they are documented. My concern is that a general purpose at most once mechanism for publishing messages will become the default that everyone uses, rather than a last resort when no other at least once mechanism exists.

Have you seen many real world applications where people are doing a publish after commit approach?  My guess is that this is the way many, or even most people use Kafka, but I don't have the empirical data to back this up.  This is part of the problem with being a framework developer.

To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/fcfc8d89-e6f1-4d15-bf25-0a17c4935d30%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[tr...@redhat.com]

I'd like to toss a different perspective into this discussion of messaging for microservices by mentioning the AMQP protocol and AMQP routing.

In many microservice scenarios, the extra transactional exchange of a brokered messaging system (JMS, Kafka, etc.) is not necessary and in fact complicates the whole thing.  AMQP routing (see the Apache Qpid Dispatch Router project) provides a scalable network at layer 4 (TCP/IP) but allows direct transactional exchange between any two endpoints at layer 7 (AMQP).  In a scenario where there are multiple instances of a consumer/service, a producer/client transacts directly with the service instance and knows when the request has been received and processed.

The benefits of scale are also provided without the need to manage topic replicas.  Because the AMQP protocol formally handles the settlement of deliveries (in a transactional way), AMQP intermediaries (routers) are aware of the current state of a delivery and are also aware of how many outstanding deliveries are being handled by each service/consumer instance.  This allows an AMQP router network to choose a service instance based on its present rate of settlement (i.e. request completion) which is a far more effective method of load balancing than round robin or using out-of-band health checks.

The management of settlement also promises to aid in the smooth increase and decrease of service scale.  If a service instance is no longer needed, the AMQP network can quiesce it by taking it out of routing consideration while it completes its backlog of unsettled requests.  The network can then indicate when it is safe to shut down an instance without losing messages.

-Ted Ross
 Engineering, Red Hat Inc.

[James Roper]

Hi Ted,

If producers/consumers are talking directly to each other, what happens when all instances of a consumer are down for some reason?  Does this cause the failure to propagate to the producer, or is the producer isolated from the consumers failure?

One of the biggest problems that we find with microservices is that when you move from a monolith to many services, and you have temporal coupling between your services, the downtime ends up being the aggregate of the downtimes of all the services, which is far greater than the downtime of a single monolith. This is why a brokered messaging system can be desirable.

Regards,

James

--

You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/0yYyOdVrZ_c/unsubscribe.

To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/b1e36514-780e-45b9-9d77-f98e5244b6e4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[andrew_s...@uk.ibm.com]

Hi Ted,

I suppose there's no reason why you could not use store-and-forward with this scheme, but you do not have to if you don't want.

Could you help me understand why there's no temporal coupling between the source and target of an message? Doesn't "settlement" correspond to the completion of a request on the target system, which would seem pretty close to just making a synchronous HTTP request through a load balancer.

Andrew Schofield

IBM Watson and Cloud Platform

[andrew_s...@uk.ibm.com]

I spent a bit of time asking around about real-world examples of people using messaging with microservices. The response was that we've not seen all that much use of messaging and proper microservices yet. The signal's not that strong for what people are actually using and we often don't get close enough to see the actual code being written.

 

I've worked with a company where they divided their applications into ones that needed exactly-once delivery with two-phase commit and ones that could cope with at-least-once. They used different messaging solutions for the two cases. The latter were using a non-transactional, eventually-consistent database which is at best atomic to the level of a single document only. In terms of the API calls, that would be publish after update. This was Node.js and Kafka.

 

Another company I worked with were using Akka and Kafka, implementing an API by using microservices triggered by messages. Definitely not transactional.

 

We definitely see a variety of languages and runtimes being used for new code. Node.js is the one that gets most attention, probably next are Java (not JEE), Python and Go.

 

I don't think it's the case that everyone uses Kafka exclusively. At the open-source end of things, I see a lot of ActiveMQ and a bit less RabbitMQ.

Andrew Schofield

IBM Watson and Cloud Platform

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/fcfc8d89-e6f1-4d15-bf25-0a17c4935d30%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[tr...@redhat.com]

Hi James,

I think this depends on how you want it to be set up.  If you're using only direct messaging, the producers will experience back-pressure.  If temporal decoupling is desired for a particular address/service, brokers can be integrated into the router network to provide queueing.  The key difference in this case is that routing and queueing are separated.  Routers route and brokers queue.  Queueing is only used where desired and messaging brokers are not used to form networks for the purpose of routing, which they don't do particularly well.

In cases that are currently handled using HTTP/REST, or some RPC mechanism, direct messaging is appropriate.

-Ted

On Wednesday, June 14, 2017 at 11:22:12 PM UTC-4, James Roper wrote:

Hi Ted,

If producers/consumers are talking directly to each other, what happens when all instances of a consumer are down for some reason?  Does this cause the failure to propagate to the producer, or is the producer isolated from the consumers failure?

One of the biggest problems that we find with microservices is that when you move from a monolith to many services, and you have temporal coupling between your services, the downtime ends up being the aggregate of the downtimes of all the services, which is far greater than the downtime of a single monolith. This is why a brokered messaging system can be desirable.

Regards,

James

On 15 June 2017 at 06:15, <tr...@redhat.com> wrote:

I'd like to toss a different perspective into this discussion of messaging for microservices by mentioning the AMQP protocol and AMQP routing.

In many microservice scenarios, the extra transactional exchange of a brokered messaging system (JMS, Kafka, etc.) is not necessary and in fact complicates the whole thing.  AMQP routing (see the Apache Qpid Dispatch Router project) provides a scalable network at layer 4 (TCP/IP) but allows direct transactional exchange between any two endpoints at layer 7 (AMQP).  In a scenario where there are multiple instances of a consumer/service, a producer/client transacts directly with the service instance and knows when the request has been received and processed.

The benefits of scale are also provided without the need to manage topic replicas.  Because the AMQP protocol formally handles the settlement of deliveries (in a transactional way), AMQP intermediaries (routers) are aware of the current state of a delivery and are also aware of how many outstanding deliveries are being handled by each service/consumer instance.  This allows an AMQP router network to choose a service instance based on its present rate of settlement (i.e. request completion) which is a far more effective method of load balancing than round robin or using out-of-band health checks.

The management of settlement also promises to aid in the smooth increase and decrease of service scale.  If a service instance is no longer needed, the AMQP network can quiesce it by taking it out of routing consideration while it completes its backlog of unsettled requests.  The network can then indicate when it is safe to shut down an instance without losing messages.

-Ted Ross
 Engineering, Red Hat Inc.

On Thursday, December 15, 2016 at 5:55:15 AM UTC-5, andrew_s...@uk.ibm.com wrote:

I'd like to kick off a discussion specifically on the scenarios for messaging we expect to see in a microservices architecture. I don't want to get bogged down in API discussions at this point. So, I'll kick this off with a short description of one way of using messaging that I see as appropriate for microservices architectures. I've started with a view that the messaging is "in front" of the microservices, so the messages are kind of commands. I think it's also entirely sensible to view messaging as "behind" the microservices, which I'd characterise as using the messages to represent events.

Comments and additional scenarios are welcome.

Andrew Schofield

Event Services, IBM Watson and Cloud Platform

--
You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/0yYyOdVrZ_c/unsubscribe.

To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/b1e36514-780e-45b9-9d77-f98e5244b6e4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[tr...@redhat.com]

Hi Andrew,

That's correct, direct or queueing can be selected on a per-address basis.

In the direct case, there is temporal coupling between source and target, just as there is between source and queue, then queue and target in a decoupled system.

To contrast to HTTP, AMQP is asynchronous and full-duplex so it provides a bit more flexibility.

-Ted

[Robbie Gemmell]

I also think using Reactive Streams looks like an interesting option for messaging and wiring things together with different frameworks, particularly given inclusion of Flow in JDK9. The request granting looks a good way to regulate transmission of messages, and I like the ideas outlined previously around using either a Processor or an envelope based explicit ack/commit to convey that handling (be it receiving confirmation of send, or completing handling of a received message) has been completed and allow for at-least-once semantics through different processing stages.

In terms of the envelope idea, that also seems like a good fit for carrying content and related message metadata supported by many systems/protocols together as a unit through processing. There is a question there for me around how much of that is considered implementation specific detail and how much could be useful to look at common handling for to aid interop across implementations?

Robbie

[James Roper]

On 22 June 2017 at 02:46, Robbie Gemmell <robbie....@gmail.com> wrote:

I also think using Reactive Streams looks like an interesting option for messaging and wiring things together with different frameworks, particularly given inclusion of Flow in JDK9. The request granting looks a good way to regulate transmission of messages, and I like the ideas outlined previously around using either a Processor or an envelope based explicit ack/commit to convey that handling (be it receiving confirmation of send, or completing handling of a received message) has been completed and allow for at-least-once semantics through different processing stages.

In terms of the envelope idea, that also seems like a good fit for carrying content and related message metadata supported by many systems/protocols together as a unit through processing. There is a question there for me around how much of that is considered implementation specific detail and how much could be useful to look at common handling for to aid interop across implementations?

I think this is a really good discussion to have at some point. Adding generic maps for metadata is very flexible, but the risk is that implementations will start relying on their specific metadata always being handled, and then you lose inter-op. An example is message keys for partitioning, if that's done using a generic metadata mechanism, and each implementation uses their own specific metadata mechanism for specifying message keys and/or partitions, then you will always need a metadata translation stage in order to make two implementations work together.  But making message keys a first class API concern doesn't make sense for any message transports that don't have partitioning.

To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/2215c8d0-ecb7-4269-9be6-dd12029cb80b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.