HashJoin after Merge throwing an IllegalStateException

60 views
Skip to first unread message

Ben Linsay

unread,
Jun 26, 2012, 5:41:06 PM6/26/12
to cascadi...@googlegroups.com
I'm getting a Hadoop mapper configuration error thrown when I try to put a Merge right before a HashJoin in my JUnit tests. It's all: "Caused by: java.lang.IllegalStateException: accumulated source conf property missing" and I'm not sure what's up. Things work if you replace the HashJoin with a CoGroup.

I reproduced the case here: https://gist.github.com/2999083

Chris K Wensel

unread,
Jun 26, 2012, 6:33:01 PM6/26/12
to cascadi...@googlegroups.com
OK, that's weird. will give it a look.

ckw

On Jun 26, 2012, at 2:41 PM, Ben Linsay wrote:

I'm getting a Hadoop mapper configuration error thrown when I try to put a Merge right before a HashJoin in my JUnit tests. It's all: "Caused by: java.lang.IllegalStateException: accumulated source conf property missing" and I'm not sure what's up. Things work if you replace the HashJoin with a CoGroup.

I reproduced the case here: https://gist.github.com/2999083

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/4t7PpFppUo8J.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.


Chris K Wensel

unread,
Jun 26, 2012, 8:11:46 PM6/26/12
to cascadi...@googlegroups.com
ok, can reproduce. may have a fix, more later.

chris

Chris K Wensel

unread,
Jun 27, 2012, 11:59:06 AM6/27/12
to cascadi...@googlegroups.com
this should be fixed in wip-321. please give it a spin and let me know if it resolves your issue.

ckw

Ben Linsay

unread,
Jun 27, 2012, 1:42:10 PM6/27/12
to cascadi...@googlegroups.com
it works perfectly. thanks a bunch for the fast turnaround!

On Wednesday, June 27, 2012 8:59:06 AM UTC-7, Chris K Wensel wrote:
this should be fixed in wip-321. please give it a spin and let me know if it resolves your issue.

ckw
On Jun 26, 2012, at 5:11 PM, Chris K Wensel wrote:

ok, can reproduce. may have a fix, more later.

chris

On Jun 26, 2012, at 3:33 PM, Chris K Wensel wrote:

OK, that's weird. will give it a look.

ckw

On Jun 26, 2012, at 2:41 PM, Ben Linsay wrote:

I'm getting a Hadoop mapper configuration error thrown when I try to put a Merge right before a HashJoin in my JUnit tests. It's all: "Caused by: java.lang.IllegalStateException: accumulated source conf property missing" and I'm not sure what's up. Things work if you replace the HashJoin with a CoGroup.

I reproduced the case here: https://gist.github.com/2999083

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/4t7PpFppUo8J.
To post to this group, send email to cascading-user@googlegroups.com.
To unsubscribe from this group, send email to cascading-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To post to this group, send email to cascading-user@googlegroups.com.
To unsubscribe from this group, send email to cascading-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To post to this group, send email to cascading-user@googlegroups.com.
To unsubscribe from this group, send email to cascading-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

Chris K Wensel

unread,
Jun 27, 2012, 1:53:50 PM6/27/12
to cascadi...@googlegroups.com
great news. 

I'll look at pushing out a 2.0.2 this weekend if nothing else comes up between now and then.

ckw

To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/Q-fhWyiauYUJ.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

George London

unread,
Dec 14, 2012, 4:37:05 PM12/14/12
to cascadi...@googlegroups.com
Hi there!

I'm new to Cascading, but already it's been a HUGE improvement over the hacked together mess I was using before.

Things had been going smoothly until I just hit what appears to be the same error as the original poster in this thread. My stack trace appears exactly the same (https://gist.github.com/4288776), and I'm also doing a HashJoin immediately after a merge. (Though I'm not sure that's what's causing it since the error message isn't attached to a line, and it would be fairly difficult to extract just the relevant part of the code.)

I just tried replacing the HashJoin with a CoGroup and it appears to now be working, though I won't be sure for a while since the problem operation happens about an hour into the process. But at least it got past the planner. 

The operation is pretty expensive, so I would prefer to use a HashJoin if possible.

I was using 2.1.0 out of Maven, and just tried upgrading to 2.1.1-wip-86 with no effect. 

Any ideas? I can try to reproduce in isolated code but that'll take a while.

Thanks!

-George





On Wednesday, June 27, 2012 1:53:50 PM UTC-4, Chris K Wensel wrote:
great news. 

I'll look at pushing out a 2.0.2 this weekend if nothing else comes up between now and then.

ckw

On Jun 27, 2012, at 10:42 AM, Ben Linsay wrote:

it works perfectly. thanks a bunch for the fast turnaround!

On Wednesday, June 27, 2012 8:59:06 AM UTC-7, Chris K Wensel wrote:
this should be fixed in wip-321. please give it a spin and let me know if it resolves your issue.

ckw
On Jun 26, 2012, at 5:11 PM, Chris K Wensel wrote:

ok, can reproduce. may have a fix, more later.

chris

On Jun 26, 2012, at 3:33 PM, Chris K Wensel wrote:

OK, that's weird. will give it a look.

ckw

On Jun 26, 2012, at 2:41 PM, Ben Linsay wrote:

I'm getting a Hadoop mapper configuration error thrown when I try to put a Merge right before a HashJoin in my JUnit tests. It's all: "Caused by: java.lang.IllegalStateException: accumulated source conf property missing" and I'm not sure what's up. Things work if you replace the HashJoin with a CoGroup.

I reproduced the case here: https://gist.github.com/2999083

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/4t7PpFppUo8J.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.
--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.
--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/Q-fhWyiauYUJ.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

Chris K Wensel

unread,
Dec 17, 2012, 11:00:27 AM12/17/12
to cascadi...@googlegroups.com
sorry, i'm going to need a bit more information.

if you can send me the dot file, that would be a start.

i've also updated the exception message a little in the next wip to identify the tap that the config information was expected for. this with the dot file should help.

next wip should be out in an hour or so.

chris

To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/M3WW9klrq6AJ.

To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

Chris K Wensel

unread,
Dec 18, 2012, 2:48:47 PM12/18/12
to cascadi...@googlegroups.com
ok, good news is I can repro this (after getting the dot offline). will work on a fix asap.

ckw
Reply all
Reply to author
Forward
0 new messages