Parallelizing Multiple Tasks which share the same dependencies

[Nhat Bui Minh]

I have a Dags, where multiple data loads which share the same dependencies (image attached). The issue is when I set the multiple workers more than 2 to set up parallel processing , luigi fails to run the bottom task first since they are being triggered multiple times by the workers.

Does anyone know if executing this dag is possible in luigi ?

Thank you!

[Lars Albertsson]

Yes, what you attempt to do is a standard pattern, so something has
gone wrong. If you share a snippet of your code, we might help you to
figure out what.


Lars Albertsson
Data engineering entrepreneur
www.scling.com, www.mapflat.com
https://twitter.com/lalleal
+46 70 7687109

> --
> You received this message because you are subscribed to the Google Groups "Luigi" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/luigi-user/c6bf1499-2fee-4ff1-97f0-c1319508b6dcn%40googlegroups.com.