Log file not found

[Jonas Seemann]

Hi everyone,

I experience a problem with my dags I cannot really solve myself.

I have a couple of tasks which sometimes run, sometimes not. 

In the cases in which the tasks don't run the error is "Log file is not found"

Apparently the worker does not create the log file, because the log file really does not exist.

But if I wait a couple of minutes and then restart the task in most cases it suddenly works. 

I already tried the solutions suggested in this thread:

https://groups.google.com/g/cloud-composer-discuss/c/wOdZ3Kah5Ls

But nothing really works. 

Is this a known bug?

I use composer version 2.0.0 with Airflow Version 2.1.4

Thanks and best regards,

Jonas

[Rafal Biegacz]

Hi Jonas,

If the task is not run then the log won't be generated - as the file is generated by Airflow worker and if task is not run then, by definition, Airflow worker is not able to create a log.

The most important thing in your case, I think, it's to determine why a task is not run - there might be a couple of reasons.

1. If the task sits in the Task Queue for too long (in case of Composer 1/Airflow 1 it is 10 minutes) then the Scheduler is going to cancel this task and in such a case a task log won't be generated.

    and the task will be rescheduled once again.

2. If, because of any reason, Scheduler scheduled a task and Worker doesn't have access to DAG file then such a task won't be executed and again in such a case a task log won't be generated.

3. Please, also take a look at this troubleshooting guide for Airflow Scheduler: https://cloud.google.com/composer/docs/troubleshooting-scheduling

Regards, Rafal.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/6d96047a-3862-4d6b-a62d-50acd5351304n%40googlegroups.com.