PlannerException: union of steps have 3 fewer elements than parent assembly

[PUSHPAK GOHEY]

Hi Chris,

While testing below scenario with two '|' delimited input files having fields f1, f2 and f3-

Input file1 --> GroupBy(on f1) --> Every(First()) --> Merge() --> Output

Input file2 --> GroupBy(on f1) --> Every(First()) -->

Cascading throws following exception-

cascading.flow.planner.PlannerException: union of steps have 3 fewer elements than parent assembly: MapReduceHadoopRuleRegistry, missing: [Hfs["TextDelimited[['f1', 'f2', 'f3']]"]["../cascading-platform/src/test/resources/data/delimited-file1.txt"], Every(GroupBy Pipe 1)[First[decl:ARGS]], GroupBy(GroupBy Pipe 1)[by:[{1}:'f1']]]

at cascading.flow.planner.FlowPlanner.verifyResultInternal(FlowPlanner.java:622)

at cascading.flow.planner.FlowPlanner.verifyResult(FlowPlanner.java:561)

at cascading.flow.planner.rule.RuleSetExec.execPlannerFor(RuleSetExec.java:163)

at cascading.flow.planner.rule.RuleSetExec$3.call(RuleSetExec.java:336)

at cascading.flow.planner.rule.RuleSetExec$3.call(RuleSetExec.java:1)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

This is the similar exception Cascading threw for join pipe assembly that I had mentioned in this issue which is fixed in Cascading-wip-3.0.

So I ran the test on Cascading-wip-3.0 and I faced same exception.

Hence I have created a test case and raised a pull request. Could you please help me in resolving this issue.

Thanks,

Pushpak D Gohey

[Dave Maughan]

Hi,

We have come across the same issue in 3.0.2 and I've also tested in 3.1.0-wip-40.

For the above example, using the Hadoop2MR1FlowConnector, if you add a Checkpoint before one of the inputs to the Merge then it works. We're also not seeing this error when using the Hadoop2TezFlowConnector to build the Flow.

Thanks

- Dave

[Chris K Wensel]

Can you confirm this is the same problem?

https://github.com/cwensel/cascading/pull/44

If not, a new test case would be great so I can tackle them all at once.

ckw

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/895f9ddb-b09a-47fd-a599-9f1ef2e607e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

—

Chris K Wensel

ch...@wensel.net

[PUSHPAK GOHEY]

Hi Chris,

Yes, this is the same problem for which I raised the pull request.

Thanks for looking into this.

Pushpak D Gohey

[Vinay Shashank]

Hi Dave,

I know this conversation is quite old but I'm facing the same issue with Cascading 3.0.2. This issue is fixed in 3.1.0 but my I can't use that yet at my company. I followed using advice to use Checkpoint before the inputs to the Merge but it didn't work. Any other suggestions?

Thanks,

Vinay 

[Chris K Wensel]

sorry, you need to upgrade to 3.1.x. I don’t know of any workarounds, but others might.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.

Visit this group at https://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/323296e5-99f3-4896-9327-ff618eb8e55e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

—

Chris K Wensel

ch...@wensel.net