microservice messaging

[wge vwgolf]

Hi

I've used app engine for a little while now, and need to break up my monolith into separate micro-services.

The key decision is how the microservices communicate - the criteria is:

• It must be asynchronously, which then implies messaging.
  
• The messages ideally need to be "fire and forget".
  
• They also need to be pretty good in terms of performance, and packet size, but these two are smaller considerations.

I can think of  two ways I could do this in the GAE world

1) Use one GAE, split it into services, and have the services talk to each other via taskQueues (which I believe are async)
Disadvantage: Not completely separate in terms of database. So the same db meta data is used for both read and write.
i.e The services still share the same database (I believe). My requirement is to have a separate write db, separate read db. Each of these have different representations of the data. The readDB is eventually consistent with the write db.

2) Use multiple GAE projects.
 
This solves the above issue, but, what is the best way to get messaging working between them ? I've looked everywhere but cannot find a decent solution.
a) XMPP is too high level, and it also requires a response to come back.
b) Can I use GCM ? If so, how ?
c) Is it possible to use RabbitMQ / ZeroMQ somehow ? if so how ?


Would really appreciate help and advice with this.

Regards

[Jason Collins]

You might want to check out this series: https://cloud.google.com/appengine/docs/python/microservices-on-app-engine

I would suggest Task Queue for messages between modules/services of the same app, and Cloud PubSub for messages between apps on different projects. Cloud PubSub could be used uniformly for both cases, but Task Queue has more features for point-to-point deliveries (e.g., countdown, finer grained retry semantics, better queue visibility for DevOps).

[Alex Martelli]

If you want data isolation between services, *namespaces* can deliver it on datastore and memcache (as the series that Jason suggests mentions, perhaps without enough emphasis).

If you have no other need for namespaces (no multitenancy etc), just have each service set the namespace it uses -- service 'foo' can have namespace 'foo' and so forth.

If you do need to use namespaces for other purposes (e.g namespace 'usr1' for user 1, 'usr2' for user 2, etc), then have namespace-setting operations combine the service name with the "user namespace" -- namespace "foo-usr1" for "user 1 on service 'foo'", and so on.

(Personally, for microservices just as for other architectural and design patterns, I've often found that some hybrid/compromise approach can be superior to rigid application of the "purist" pattern -- but, that's a much broader subject, not really germane to this forum!-).

Alex

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/023b66f0-925d-4171-b3d3-67e788953f71%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.