Infinite loop of Build Synchronization

[bikram....@essenceglobal.com]

My cloud composer stuck in infinite loop of build synchronization. I am seeing following set of logs multiple times per minute:

• Client closed local connection on 10.40.0.9:3306
• New connection for "xxxxx-tp:us-central1:us-central1-xxxx-xxxcf-sql"
• Client closed local connection on 10.40.0.9:3306
  
• Building synchronization state...
  
• Starting synchronization...
  
• ...
• INFO - Finding 'running' jobs without a recent heartbeat
  
• Failing jobs without heartbeat after 2018-09-06 04:30:03.091647
  
• ..
• Finding 'running' jobs without a recent heartbeat
  
• Building synchronization state...
  
• Starting synchronization...
  
• Removing file:///home/airflow/gcs/dags/dag1.pyc
  
• Removing file:///home/airflow/gcs/dags/dag2.pyc
  
• ...
• Building synchronization state...
  
• INFO - Heartbeating the process manager
  
• ..
• Processor for /home/airflow/gcs/dags/dag1.py finished
  
• ..
• {models.py:1594} ERROR - <TaskInstance: dag1.task1 2018-09-05 23:59:33.100734 [running]> killed as zombie
  

Any help is appreciated as I am seeing lengthy tasks getting killed without any exceptions in logs (due to zombie) as well as long wait/timeouts in accessing gcs buckets (data/ or dags/)  

[Feng Lu]

These sync logs are produced by the gcs-synd side car container and shouldn't matter for task execution. 

If you suspect a task got killed improperly, you may want to take a look at Composer stackdriver logs as described in [1]. Also could you quantify the "long" wait/timeouts in accessing gcs bucket (e.g., seconds vs minutes)? 

[1]. https://console.cloud.google.com/projectselector/logs?service=composer.googleapis.com

-------------------------------------------------------

essenceglobal.com
Facebook • Twitter • YouTube • Instagram

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/29cd6913-7e7d-4250-bcbe-507278864a7c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.