Micro Services and noSQL - Best practice to enrich data in micro service architecture

[Rails]

I want to plan a solution that manages enriched data in my architecture.
To be more clear, I have dozens of micro services.
let's say - Country, Building, Floor, Worker.
All running over a separate NoSql data store.

When I get the data from the worker service I want to present also the floor name (the worker is working on), the building name and country name.

Later, I would like to sort and filter workers by names of country and buildings.

(My entities all have GUID as ID. )

Solution1.
Client will query all microservices. each entity will hold only normalized data (building Id, CountryId )
Problem - multiple requests and making the client be aware of the structure.
I know multiple requests shouldn't bother me but I believe that returning a json describing the entity in one single call is better.

Solution 2.
Create an orchestration on the FE side that retrieves the data from multiple services.
Problem - if the data (entity names, for example) is not stored in the same document in the DB it is very hard to sort and filter by these fields.

Solution 3.
(DeNormalizing) Before saving the entity, e.g. worker, call all the other services and fill the relative data (Building Name, Country name).
Problem - when the building name is changed, it doesn't reflect in the worker service.

solution 4.
(DeNormalizing big time)

Create a process that subscribes to a broker and receives all entities change.
For each entity it updates all the relavent entities.
When an entity changes, let's say building name changes, it updates all the documents that hold the building name.
Problem: Each service has to know what can be updated. When a trailing update happens it shouldnt update the broker again (recursive update), so this can complicate to the microservices.

[Chris Richardson]

A few points:

• It's generally better to use an API gateway so that client doesn't have to make multiple calls (or be aware of the multiple services). See http://microservices.io/patterns/apigateway.html
• Whether or not fetching data from multiple services is efficient depends on the nature of the queries. Each N parallel requests is probably good where as N sequential requests is probably bad since it potentially results in high latency. If that's the case then you should consider denormalizing.
• If you denormalize then an event-driven architecture is probably best. e.g. building service publishes a BuildNameChanged event then subscribers can update their copy.
• You might want to consider using Command Query Responsibility Segregation (CQRS) and define denormalize views to support queries so that the services themselves are narrowly focussed on updates.
• Event sourcing (ES) is a good way to implement an event-driven architecture
• For information on event-driven microservices with ES+CQRS see my blog post http://plainoldobjects.com/2015/01/04/example-application-for-my-qconsf-presentation-about-building-microservices-with-event-sourcing-and-cqrs/

I hope this helps.

Chris

--
You received this message because you are subscribed to the Google Groups "microservices" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microservice...@googlegroups.com.
To post to this group, send email to micros...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microservices/210c5902-542b-4abb-9ffe-d1f993f9b388%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rails]

Wow. 

Took me sometime to reply to your messages because I had to take time to absorb everything in your slideshows, videos,  and documentation. 

You have identified and tackled a lot of our project problems. 

Building micro service architecture without solving atomicity and transaction-related problems just doesn't work.

It was a truly mind opening. 

Thanks.

[Rails]

I have an additional question:

I am planning to update my architecture following your presentation. 

The CQRS solution and the event driven architecture is something I would love to implement. 

Lets say I have a service (Account) that has several aggregates that deal with messages (AccountCreated, AccountFoo).

Upon a save (rest call), I validate the request in my gateway, generate an id and send the full entity to the broker (RabbitMQ) with AccountCreated as a topic and a json of the account as a body.

The Broker has a few listeners/consumers. One that saves it (CouchBase), one that sends to a denormalized material view (ElasticSearch) , One that publishes to external subscribers...

1. Was I making sense until now? Is it  following the guidelines?

2. Notice that I have implemented everything but the "Persists events NOT current state"
    Is it doable? saving a state really eases on the change. It's too weird to calculate all the events when I have a page that's shows getAllAccount.
    constant calculation of snapshots sounds like something I should try to avoid. 

3. How do I manage a logical transaction?
    Even in a non transactional system I have a transaction here. 

    It might fail to publish to external users, it might fail on sending it to the ES. Are there any recommended tools to manage the transaction?
    Retries/CircuitBreakers/Logging?

Thanks.