We will have one pipeline per customer (1 DAG per customer).
I am trying to think of how to build the entire workflow-setup with airflow.
We really dont want to introduce celery at this stage.
Specifically, I am trying to understand and implement the following:
1. How do I build my pipeline such that pipelines for certain set of customers go to Node1 while few go to node3 ...and so on.
2. Can I set up airflow such that there is one UI for all pipelines?
If anyone has a similar set up, it will be worth discussing it here.