Dag that runs a GcsSensorOperator creates always 1 running instance and queues 16 in parallel when deploying the dag code

[Max M]

Hello all,

we have upgraded our Composer from version Composer-1.17.2-Airflow-2.1.2 to Composer-2.0.22-Airflow-2.1.4 and got an issue with a dag that runs a sensor.

We have a dag which runs a custom GcsBucketSensor which inherits from BaseSensorOperator. This is a kind of dag which scans a Gcs Bucket. When some kind of file arrives there, the dag will trigger another dag. 

Now the issue:

The Airflow scheduler tries to run 17 DagRuns for each dag in parallel. The dag config of the dags looks as follows:

dag = DAG(
dag_id=Path(file).stem,
schedule_interval=timedelta(seconds=1),
start_date=datetime(2000, 1, 1, 1, 1),
default_args={},
max_active_runs=1,
catchup=False

Since it is only one active run allowed, 1 DagRun is running and the other 16 are queued. The running DagRun works fine and does what it has to do. There also aren't any errors in the logs or somewhere else. Furthermore if I try to close one scheduled DagRun the scheduler immediatly queues another DagRun. This means the scheduler tries permanently to run just 17 dag run instances in parallel. This problem did not appear in Composer v1. In Composer v1 there was only one instance running. And catchup is also on False so that it should prevent to queue instances in parallel.

Currently this issue does not destroy anything in our processes. However, it looks very weird in the logs and people will ask me questions about that. I want to know what the issue of that could be. 

Do you have an idea what could be the reason of this issue and how to fix it? I will be thankful for any help.

Kind regards,

Max