Configuring Airflow Worker Queues on Composer

[ma...@chope.co]

Hi,

Quick question. Is there anyway to configure queues for Airflow workers on Cloud Composer?

Reason for this is because some of my DAG tasks have high memory requirements, so I intend to create a separate node-pool and schedule these high memory tasks only on workers in that pool. It seems like airflow queues are a good way to do that, but can't figure out where it can be configured post-creation of the worker.

Thanks.

[eliseop...@gmail.com]

+1

Unfortunately it seems that autoscaling, kubernetesExecutor or even spinning GCE instances are all features not quite ironed out yet.

[Crystal Qian]

Composer doesn't have built-in support for multiple node pools; perhaps creating an environment with a different machine type (this can't be changed after environment creation) might help with high-memory tasks.

Crystal