question about translation to map-red

5 views
Skip to first unread message

Koert Kuipers

unread,
Dec 15, 2011, 5:54:31 PM12/15/11
to cascadi...@googlegroups.com
i have a few subassemblies that i run after each other: A, B, and C
i save the results to a sink after A and also after C to a different sink.

now strangely the translation to map-reduce boils down to 2 map-reduce jobs that are run in parallel:
source => A => sink1
source => A => B => C => sink2

this way subassembly A is run twice, which was not my intention. what am i doing wrong?

code looks something like this:

Pipe pipe = new A(new Pipe("source"));
Pipe left = pipe;
Pipe right = pipe;
right = new B(right);
right = new C(right);
final Map<String, Tap> sinkMap = new ImmutableMap.Builder<String, Tap>().put("left", leftSink).put("right", rightSink).build();
final Flow flow = flowConnector.connect(sourceTap, sinkMap, new Pipe("left", left), new Pipe("right", right));


Koert

unread,
Dec 15, 2011, 6:00:04 PM12/15/11
to cascading-user
i think i just found the answer in the thread called "Dumping pipe to
disk"
i have to use isSafe() on my method inside subassembly A.
best, koert
Reply all
Reply to author
Forward
0 new messages