Optimization vs Decoupling

[Joao Santos]

Hi all. I find myself (once again) posting to what is probably the wrong mailing list, but I guess a lot of you have experience in this field. Sorry if you feel this question is not appropriate to this group.

I'm trying to design my first low latency distributed system as a learning experience, but not necessarily ultra low latency/jitter sensitive. For scalability and redundancy reasons the architecture will be based on micro-services doing pub/sub on a messaging bus. Question is, should I go event based, command based, mixed or something else entirely?

First option is fully event based: each service subscribes to some other services event topic and outputs events itself. As an example, I get a request from the outside to create an account, and the gateway service will output an event "AccountCreationRequested" on the "GatewayEvents" topic. An accounting system will be listening on this topic and create the accounting records, outputting it's own events ("AccountCreated" on topic "AccountingEvents", etc). An analytics service would also subscribe to "GatewayEvents" and generate some statistics, outputting it's own events on topic "AnalyticsEvents". This system, while fast, will couple the accounting and analytics services to the gateway, as an example.

Second option is fully command based: each service receives commands on a service specific command queue and commands other systems to do something by writing to their command queue. As an example, I get a request from the outside to create an account, and the gateway service will send a command "CreateAccount" on topic "AccountingCommands" and a command "CreateAccount" on topic "AnalyticsCommands", addressing those systems specifically. The gateway service now has intimate knowledge of who it is working with, although it would still be a fast system.

Third option is a mixed system: each service receives commands on a service specific command queue and outputs state changes on a service specific event topic. This system will need a "orchestrator" to transform events into commands. As an example, I get a request from the outside to create an account, and the gateway service will output an event "AccountCreationRequested" on the "GatewayEvents" topic. An orchestrator is subscribed to that topic and, upon seeing the event, will send a "CreateAccount" command to topics "AccountingCommands" and "AnalyticsCommands". This system looks better from a service decoupling perspective, but does add an additional network hop/contention point/failure point.

I'd like to gather people's feelings on this. Do any of these make sense? Have you used anything like this? What were the problems with your system? What was really great about it?

I've read a lot about all of this but still I have mixed feelings, I'm even kind of lost. Sam Adams from LMAX mentions in a 2014 talk [1] that LMAX uses an event based system but then describes topics as "ExecutionVenueServiceInstructions" and shows pieces of code akin to async RPC (45:52 in to the talk). It does sound like a command sourcing system, not an event based one. Command sourcing/journaling isn't without it's problems as commands can return different results as opposed to event sourcing/journaling where there is not extra validation on the event after it has been emitted, it just changes the system's state.

Appreciate any feedback I can get.

[1] https://skillsmatter.com/skillscasts/5247-the-lmax-exchange-architecture-high-throughput-low-latency-and-plain-old-java

[Vitaly Davidovich]

Just curious - in first option, why not output AccountCreationRequested on a topic of the same name? That way your gateway simply normalizes/canonicalizes external requests into your domain specific stuff, but then downstream consumers aren't coupled to your gateway.

sent from my phone

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jan van Oort]

You already described the disadvantages of a mixed system in your post. Moreover, your system will, both in the fully command-based and in the fully event-based variety, be able to replay all or part of your history, e.g. in order to restore a certain system state in the case of corruption etc.  ( Of course, under condition of logging any and every event / command ). 

Commands tend to be more lightweight, as very often you can play with static instances, e.g. class Command has a static final field of type AccountCreationCommand , one other of type AccountDeletionCommand -- and you reuse these, all the time. 

The question is: what are you after: high scalability / maximum scale-out capabilities ? Then you need maximum decoupling, and as much statelessness as you can get. Or only low latency ? The design goal is not exactly clear to me. 

Fortuna audaces adiuvat - hos solos ? 

--

[Joao Santos]

For the simple example I described that would work, but on a complete system with lots of services that would lead to probably 10000 topics (one per event type). That might work, i'll have to think about it.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Vitaly Davidovich]

Assuming your domain is actually that complex and 10k events is not an implementation deficiency, I don't see an issue with that.  The other upside to being targeted is that downstream consumers don't need to peek into general messages to determine if the more fine grained event type is applicable to them or not (i.e. no need to inspect GatewayEvent if it's really AccountCreated or something else).  This can also improve routing/delivery of the message (i.e. if you're doing point-to-point, and not multicasting, then your router will have more specific destinations).  Intuitively, your gateway already knows the type of event that it received, there's no need to lose that information by packaging it into a coarse grained message which then has to be "unpacked" downstream.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Joao Santos]

I'm going for both. Low latency AND high scalability. Ideally i'd be able to, as an example, shard accounts where even account numbers would be handled by a compute cluster and odd numbers by a different cluster. As an exercise i'm trying to get it with as low a latency as possible but I have no hard targets right now (like 99% of responses in less than 20ms). I'm just trying to get a sense of what architectures have people designed for this kind of systems.

For the sake of having a design goal let's say i'm trying to replicate LMAX. What I got is they use command topics, with a "command id" byte inside the data packets. This leads to tight coupling between the command sender and the receiver. The sender is commanding a very specific system to do something. If a request for order creation comes in thru the broker side, the gateway instructs the risk management service to handle the order creation. The risk management service then instructs the execution venue service to handle the order. It's all command based. On the other hand a "Market Maker" isn't risk managed so if an order creation request comes from the market maker it is automatically routed to the execution venue, bypassing risk management. It seems very tightly coupled when compared to an event based system. I'm trying to understand why.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.