Maximum DAG and maximum tasks cloud composer can run

[Tak Shing Lai]

Dear Sir,

I have two scenarios:

1. If I have a DAG which run frequently (say every 5 mins) with unpredictable workload(e.g. crawling a website which vary time by time). In this case, if one day a task run too long so that it jammed (i.e. the previous tasks is not completed and the new task start). I can foresee the concurrent running tasks will be increasing. May I know know there is a limitation of number of concurrent running tasks in composer?

2. If I have to create tons of DAG (say more than 300). What is the best practices for this case? Shall I increase the number of scheduler or even the number of composer ?

Thanks and Regards,

Tak

[Rafal Biegacz]

Hi Tak,

There is quite a lot of flexibility when it comes to adjusting Composer environment to your workloads/needs.

Everything is a matter of:

a) adjusting parallelism/concurrency parameters in your Composer environment which is sth closely related to Scheduler(s)
Please, take a look at: https://cloud.google.com/composer/docs/troubleshooting-scheduling#scaling_airflow_configuration for more information. You can also have more Schedulers if need be.

b) adjusting the size and capabilities of your Composer environment, 

In Composer 1 you can adjust the number of Airflow workers (i.e. nodes in Composer cluster) and machine type used for Composer cluster nodes.

In Composer 2, you can adjust the number of Airflow workers and CPU/memory used by them. You can use Autoscaling to accommodate peaks.

NOTE: Composer 1 is a GA offering, Composer 2 is Preview offering.

Please, take a look at these instructions: https://cloud.google.com/composer/docs/environment-scaling to read more.

Regards, Rafal.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/f34da456-9a58-4067-a49b-51781cba75b9n%40googlegroups.com.