Airflow recommends using an alternative result_backend (a database)

[Jocelyn]

Hi,

I have created an environment with image version composer-1.5.0-airflow-1.10.1. However, i get this warning in my logs: 

{default_celery.py:80} WARNING - You have configured a result_backend of redis://airflow-redis-service:6379/0, it is highly recommended to use an alternative result_backend (i.e. a database).

[2019-03-04 09:40:06,944] {base_task_runner.py:101} INFO - Job 4005: Subtask adwords_api [2019-03-04 09:40:06,944] {__init__.py:51} INFO - Using executor CeleryExecutor. 

I thought airflow/composer is configured to use Cloud SQL as the backend database.

Is this something to worry about? Any recommendations would be appreciated.

Regards

Jocelyn

[Feng Lu]

Great question, the concern with running Redis backend is that celery task messages may get lost across Redis restarts. Composer has provisioned Redis service using StatefulSet and saves a snapshot every 60 seconds to a persistent disk. So it's not a concern to use the Redis backend in Cloud Composer. 

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/163391ee-6fc8-4d8b-8510-a2aaed935c2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.