Recommended DAG framework for Python?

55 views
Skip to first unread message

Ram Rachum

unread,
Jul 13, 2021, 5:14:31 AM7/13/21
to pyweb-il
Hi guys,

For my MARL research, I'm writing a DAG framework for Python, i.e. a framework that lets you submit a directed acyclic graph of tasks and have them executed concurrently.

While I'm having a great time designing my own solution, I'm wondering if there's a good package in Python that does that. I know that Apache Airflow exists, but I'm suspicious of it. They used it at a company I worked at, and I remember it took way too much CPU and was a pain to deal with. Looking at the tutorial, it doesn't look very Pythonic to me.

Does anyone have a DAG framework in Python that they can recommend? Any other insights will also be appreciated.


Thanks,
Ram.

Avraham Serour

unread,
Jul 13, 2021, 9:54:36 AM7/13/21
to pywe...@googlegroups.com
I would take a look at ray (https://ray.io/), not sure if it can do DAG but I think it is worth a look.
Are you familiar with that?

--
You received this message because you are subscribed to the Google Groups "PyWeb-IL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyweb-il+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyweb-il/CANXboVYUENfkSZ1Nmxch13CjQzDoBzO12ATtZsbUmSkeSCdRKA%40mail.gmail.com.

Ram Rachum

unread,
Jul 14, 2021, 5:29:56 AM7/14/21
to pyweb-il
I'll check it out, thank you. 

Aur Saraf

unread,
Jul 14, 2021, 5:34:28 AM7/14/21
to pywe...@googlegroups.com
I'm guessing it's too heavyweight for your needs, but celery is a very mature distributed computation framework that supports DAGs.

yotam...@gmail.com

unread,
Jul 14, 2021, 9:15:06 AM7/14/21
to PyWeb-IL
I'd take a look at https://www.prefect.io/ and https://dagster.io/

That being said, I think Airflow is as close to a standard as you'd get, and if I were to choose a workflow framework in python I'd try to pick airflow.
(Not mentioning celery because 'mature' is a bit of an understatement, and personally I didn't find the DAGing abilities to be very convenient, although it is technically true that it is supported.)
Reply all
Reply to author
Forward
0 new messages