Configuring Airflow Config variables

2,166 views
Skip to first unread message

ma...@chope.co

unread,
Aug 1, 2018, 7:48:20 AM8/1/18
to cloud-composer-discuss
Hi,

Is there a way to configure Airflow configuration variables in composer? 

Am trying to increase the celeryd concurrency, right now default is at 6 which seems a little low.

The CLI doesn't support configuring through Env variables.


Torkjel Hongve

unread,
Aug 1, 2018, 8:13:12 AM8/1/18
to cloud-composer-discuss
Hi

I needed to do the exact same change today. Ended up changing celeryd_concurrency in the configmap holding the airflow configuration in the Kubernetes settings, then restarting the scheduler and worker pods manually. 

Torkjel

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsubscri...@googlegroups.com.
To post to this group, send email to cloud-composer-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/b5ae8f31-a97c-469e-a472-9c848f442f27%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

thibault clement

unread,
Aug 1, 2018, 10:19:13 PM8/1/18
to cloud-composer-discuss

Hi,


Never test, there is an Airflow Configuration Overrides section in Google Cloud console -> Composer -> [your environment]


In your case, section will be celery and key celeryd_concurrency



Thibault

ma...@chope.co

unread,
Aug 1, 2018, 10:56:18 PM8/1/18
to cloud-composer-discuss
Nice, tested this and it works. Would like to automate this step though rather than going to the WebUI.

Right now I have a script that sets up and customizes Composer environments (mount docker images, separate worker node pool). Would be good if there is a official solution to set these custom configurations programmatically. Will be trying out Torkjel's solution in the interim.

Thanks guys!

thibault clement

unread,
Aug 1, 2018, 11:05:00 PM8/1/18
to ma...@chope.co, cloud-compo...@googlegroups.com
If you use gcloud to set up your environment (I think you can do it with the API too) you can add airflow override arguments using a list of SECTION_NAME-PROPERTY_NAME=VALUE separate by comma.

For instance:

gcloud beta composer environments create YOUR_ENVIRONMENT \
--location europe-west1 \
--airflow-configs=celery-celeryd_concurrency=12 \


Thibault

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/d0c34e53-e46d-4b0d-b1a4-3c4691aad979%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Thibault Clément

+65 8753 5289 (Singapore)
+33 (0)6 66 11 77 39 (France)
thibau...@gmail.com

ma...@chope.co

unread,
Aug 1, 2018, 11:12:23 PM8/1/18
to cloud-composer-discuss
Yup, you're right, this is probably the way to go for my use case. 

I was exploring the CLI and discovered a way to update the environments post-creation step.

gcloud composer environments update [YOUR_ENVIRONMENT] --location=[LOCATION] --update-airflow-configs=[KEY=VALUE]


Where KEY takes the format of [section-variable], e.g. celery-celeryd_concurrency
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

Torkjel Hongve

unread,
Aug 2, 2018, 3:12:55 AM8/2/18
to cloud-composer-discuss
That's interesting, because when I try this in the UI I get an error message saying celeryd_concurrency is "blacklisted". Getting the same with the core/parallelism setting. This is in the Beta though. Maybe it was changed for GA.

This was reported, but went unanswered, on the alpha list: https://groups.google.com/forum/#!topic/cloud-composer-alpha/36HXZ8NBfiM

Torkjel

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsubscri...@googlegroups.com.


--
Thibault Clément

+65 8753 5289 (Singapore)
+33 (0)6 66 11 77 39 (France)
thibau...@gmail.com

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.
To post to this group, send email to cloud-composer-discuss@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/6a1661b8-f6b9-4301-81dd-09584672b4d1%40googlegroups.com.

Feng Lu

unread,
Aug 2, 2018, 3:48:02 AM8/2/18
to torkjel...@tapad.com, cloud-composer-discuss
Hi Torkjel, 

Celery related config blacklisting is updated in GA, please see our documentation here: https://cloud.google.com/composer/docs/concepts/airflow-configurations. 
You'll need to restart the worker pod for celery-* configs to be effective. 

Feng 

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.


--
Thibault Clément

+65 8753 5289 (Singapore)
+33 (0)6 66 11 77 39 (France)
thibau...@gmail.com

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/6a1661b8-f6b9-4301-81dd-09584672b4d1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAN77Tcvw0n%2BUuvxV3-MG5795ubRQ_RVA7ZwaWC3Bo%3DkBQRfcpA%40mail.gmail.com.

Torkjel Hongve

unread,
Aug 2, 2018, 4:43:27 AM8/2/18
to Feng Lu, cloud-composer-discuss
Thanks for the answer Feng

If I read that page correctly the celeryd_concurrency setting is still blacklisted, but there may be a way in using the celery-celery_config_options setting... Do you plan to introduce a more convenient way to sett the number of tasks per node? In our case all tasks will be light weight, just API calls to other services, so having only 6 per node seems too low.

Torkjel

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsubscri...@googlegroups.com.


--
Thibault Clément

+65 8753 5289 (Singapore)
+33 (0)6 66 11 77 39 (France)
thibau...@gmail.com

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsubscri...@googlegroups.com.
To post to this group, send email to cloud-composer-discuss@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsubscri...@googlegroups.com.
To post to this group, send email to cloud-composer-discuss@googlegroups.com.

Maria Janczak

unread,
Aug 2, 2018, 6:58:10 PM8/2/18
to torkjel...@tapad.com, Feng Lu, cloud-compo...@googlegroups.com
Hi Torkjel,

You have caught an error with our documentation, which we will fix. All celery configs EXCEPT celery-celery_app_name, celery-worker_log_server_port, celery-broker_url, celery-celery_result_backend, celery-default_queue, celery-celery_config_options should be updatable in environments version composer-1.0.0 (created July 19th or later). This should include celeryd_concurrency.

-Maria



To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.


--
Thibault Clément

+65 8753 5289 (Singapore)
+33 (0)6 66 11 77 39 (France)
thibau...@gmail.com

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAN77Tcv4SinEQOgQCqcKBAQGbmG5QCf%2BKu6g5giFWE2mRGqcqg%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages