Split one TypedPipe into multiple branches

11 views
Skip to first unread message

Christian Pernillo

unread,
Feb 21, 2023, 10:35:22 AM2/21/23
to Scalding Development
Dear Scalding community, 

I'm struggling with a Scalding job that we need to optimize, the job contains one big TypedPipe that has to be filtered/mapped twice in a sequential way since the data inside of it is very similar but at the same time it has to be split in 2 different pipes. Basically what I need is what "partition" method does but instead of only splitting this big Pipe into 2 mutually exclusive branches I need either the capacity of sending one "row" to both branches or splitting the big Pipe into multiple branches to later append the pipes that need to be together.

The job that I mention goes something like this:
carbon.png
As I mention earlier, data in pipe1 could also be in pipe2 so `partition` method doesn't work for my case.

If someone helps me It would be so great, I've been looking everywhere and I couldn't find reference.

Thank you so much!
Reply all
Reply to author
Forward
0 new messages