The maximum number of dag and maximum number of tasks that cloud composer can run

4 views
Skip to first unread message

Tak Shing Lai

unread,
Sep 6, 2021, 9:42:16 AMSep 6
to gce-discussion
Dear Sir,

I am new to composer and I want to ask two scenarios:
1. If I have a DAG which run frequently (say every 5 mins) with a heavy workload task (say crawling a complicated website). I can foresee there will be some jammed tasks(the new tasks created while the previous tasks are not yet completed).
You can assume the task is running in airflow directly. 
In this case, what should be the factor of limitation and what is the best practices of this case?(for example i can create another DAG?)

2. If I have a lot of DAG in one composer (say more than 500), is it better to add the number of scheduler ?

Thanks and Regards,
Tak
Reply all
Reply to author
Forward
0 new messages