Deploy airflow multi-tenant env: one scheduler multiple nodes deployment

281 views
Skip to first unread message

r0ger

unread,
Apr 19, 2016, 1:24:03 PM4/19/16
to Airflow
We will have one pipeline per customer (1 DAG per customer). 
I am trying to think of how to build the entire workflow-setup with airflow.

We really dont want to introduce celery at this stage.


Specifically, I am trying to understand and implement the following:
1. How do I build my pipeline such that pipelines for certain set of customers go to Node1 while few go to node3 ...and so on.
2. Can I set up airflow such that there is one UI for all pipelines? 

If anyone has a similar set up, it will be worth discussing it here.
Reply all
Reply to author
Forward
0 new messages