Externally Triggered Task

[Sens Critique]

Hello,

Would it be possible to use Airflow for externally triggered task, either itself or inside an ecosystem.

For example, if an event happens, I want to store it synchronically in MySQL, but then I want to design a task workflow to propagate this data right away in NoSQL databases, increment counters, maybe send an email to the user, etc ..

I was thinking that Airflow could listen to messages on a broker, like RabbitMQ or Kafka. Is it possible and how does it compare to Celery for instance ?

[AH]

Perhaps the ExternalTaskSensor or the SqlSensor  will do what you want? 

You can also have tasks sent out to workers via Celery using airflow.

[Maxime Beauchemin]

We're planning on supporting the CLI features through a REST API, would that work for you? There's also a task param `adhoc` that allows for on demand tasks that can live in your DAG but aren't triggered by the scheduler.

[Sens Critique]

I think I'll try a Celery integration, thanks ! (but yes probably a REST API would be great too)