Recommended DAG framework for Python?

Skip to first unread message

Ram Rachum

Jul 13, 2021, 5:14:31 AM7/13/21
to pyweb-il
Hi guys,

For my MARL research, I'm writing a DAG framework for Python, i.e. a framework that lets you submit a directed acyclic graph of tasks and have them executed concurrently.

While I'm having a great time designing my own solution, I'm wondering if there's a good package in Python that does that. I know that Apache Airflow exists, but I'm suspicious of it. They used it at a company I worked at, and I remember it took way too much CPU and was a pain to deal with. Looking at the tutorial, it doesn't look very Pythonic to me.

Does anyone have a DAG framework in Python that they can recommend? Any other insights will also be appreciated.


Avraham Serour

Jul 13, 2021, 9:54:36 AM7/13/21
I would take a look at ray (, not sure if it can do DAG but I think it is worth a look.
Are you familiar with that?

You received this message because you are subscribed to the Google Groups "PyWeb-IL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
To view this discussion on the web visit

Ram Rachum

Jul 14, 2021, 5:29:56 AM7/14/21
to pyweb-il
I'll check it out, thank you. 

Aur Saraf

Jul 14, 2021, 5:34:28 AM7/14/21
I'm guessing it's too heavyweight for your needs, but celery is a very mature distributed computation framework that supports DAGs.

Jul 14, 2021, 9:15:06 AM7/14/21
to PyWeb-IL
I'd take a look at and

That being said, I think Airflow is as close to a standard as you'd get, and if I were to choose a workflow framework in python I'd try to pick airflow.
(Not mentioning celery because 'mature' is a bit of an understatement, and personally I didn't find the DAGing abilities to be very convenient, although it is technically true that it is supported.)
Reply all
Reply to author
0 new messages