Google Cloud Composer in a microservice orchestrated architecture
369 views
Skip to first unread message
Jack Lee
Jan 8, 2019, 11:53:35 AM1/8/19
to cloud-composer-discuss
I'm currently trying to figure out how we can employ a microservice architecture in our solution. For this reason I'm looking into Google Cloud Composer as the conductor in an orchestrated architecture.
My idea is that we expose a set of high-level REST APIs while keeping the internal microservices hidden from the outside world. When someone calls one of the high-level REST APIs then a message could be added to a PubSub queue which would somehow trigger a particular DAG to run. There's a DAG for each high-level external API call and each decribes a series of internal HTTP calls for completing the task. The PubSub in between is to decouple the external API from Cloud Composer since we have some long-running processes, and currently, my plan is to expose the high-level APIs using AppEngine Flexible which has a upper timeout limit of 1 hour. Naturally, the caller would have to subsequently call some jobstatus endpoint to fetch the status of a task.
My idea is that we expose a set of high-level REST APIs while keeping the internal microservices hidden from the outside world. When someone calls one of the high-level REST APIs then a message could be added to a PubSub queue which would somehow trigger a particular DAG to run. There's a DAG for each high-level external API call and each decribes a series of internal HTTP calls for completing the task. The PubSub in between is to decouple the external API from Cloud Composer since we have some long-running processes, and currently, my plan is to expose the high-level APIs using AppEngine Flexible which has a upper timeout limit of 1 hour. Naturally, the caller would have to subsequently call some jobstatus endpoint to fetch the status of a task.
So, my question is if Cloud Composer is the right fit for the job or whether I should look at something else (maybe too complex to setup, etc.)? The only product that I've seen which directly describes itself as a microservice orchestrator is Conductor. However, I'd prefer Google Cloud-solution if it's feasible.
Thanks.
Wilson Lian
Jan 8, 2019, 3:30:54 PM1/8/19
to Jack Lee, cloud-composer-discuss
Hi Jack,
As far as orchestrating a series of potentially long-running internal HTTP calls goes, Composer is a great choice. I would re-consider using Cloud Pub/Sub as the triggering mechanism for DAGs, though. The reason is that you'd have to use the PubSubPullOperator and have a worker constantly polling your topic waiting to kick off the DAG. One of the downsides of this (consuming a worker slot for polling) will be alleviated by AIRFLOW-2747, but there still remains the issue that in order for an external API call to be handled, a DAG needs to have already been started and waiting behind a Sensor. A better architecture might directly trigger DAG execution from your AppEngine Flex API handler via the Airflow REST API. This would be similar to how you can call the Airflow REST API from Google Cloud Functions.
best,
Wilson
--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/1f66db43-fdb4-4905-81e8-7fdb4e42f5cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Tomas Jansson
Jan 8, 2019, 6:44:37 PM1/8/19
to Wilson Lian, Jack Lee, cloud-composer-discuss
You could also do a something else, and that is having a cloud function that listens to the pubsub and then based on the message it will trigger airflow over REST for you.
That is sort of the way we actually plan to structure things in a re-write. We will create a series of small DAGs that are triggered over pubsub + cloud function as described. We will also have a convention that we send in a pubsub topic where a response should be posted when the DAG is finished.
This structure will allow us to create simpler DAGs for the flow composed by on operator type that sends a pubsub message and then wait for the response. We will also be able to have SLA on every single smaller DAG that does the actual work, and the pubsub abstraction also allows us to use other tools than airflow to do the work.
The cloud function can be generalized so it can trigger any DAG basically, as long as we stick to some convention.
That is sort of the way we actually plan to structure things in a re-write. We will create a series of small DAGs that are triggered over pubsub + cloud function as described. We will also have a convention that we send in a pubsub topic where a response should be posted when the DAG is finished.
This structure will allow us to create simpler DAGs for the flow composed by on operator type that sends a pubsub message and then wait for the response. We will also be able to have SLA on every single smaller DAG that does the actual work, and the pubsub abstraction also allows us to use other tools than airflow to do the work.
The cloud function can be generalized so it can trigger any DAG basically, as long as we stick to some convention.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAM_uUVs-0vycHAiMLKXr9smBrqG8pcV%2Bke%2B%3DrMOmz%2BjiNYa78g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
| Tomas Jansson Sr. Director of Software Engineering +47 91862293 | @tomasjansson | skype:mastoj Karl Johans gate 21, 0159 Oslo, Norway
|
Reply all
Reply to author
Forward
0 new messages