Custom counters not generated on rhs of Hashjoin

57 views
Skip to first unread message

ANIKET MORE

unread,
Sep 1, 2016, 11:08:52 AM9/1/16
to cascading-user
Hi,

I tried to ran a cascading job with 2 input files and used a hashjoin pipe.I have created custom counters using each pipe to get the( Record Count) number of tuples flowing through a pipe/input.1st input file connected on lhs generated correct record count whereas for 2nd input file on rhs was not giving any record count.

After debugging the code, The getReporter method in HadoopFlowProcess class is returning two different reporter objects for the 2 input files, TaskReporter for lhs and NULL reporter for rhs.But we are expecting TaskReporter for both input files.   

For the input file connected on lhs of Hashjoin, TaskReporter is incrementing the custom counter by incrCounter method and 
for input file connected on rhs, Reporter is called to increment the counter(But this class have empty incrCounter method).

The above problem was faced when using Hadoop2MR1FlowConnector but when I ran the same job using Hadoop2TezFlowConnector, the 
counters were successfully generated for both the inputs.

Can someone please provide any solution for this problem? 

Cascading version : 3.1.0
Hadoop version : 2.6 

Thanks

Prabodh Mhalgi

unread,
Sep 13, 2016, 9:05:58 AM9/13/16
to cascading-user
I am facing similar issue. Did find any workaround / resolution for this?

Chris K Wensel

unread,
Sep 13, 2016, 2:00:33 PM9/13/16
to cascadi...@googlegroups.com
this looks like a bug in Cascading MR, Tez makes the ‘context’ available much earlier so shouldn’t be an issue there. 

we are assigning the reporter before we are actually given the actual reporter, so we are ending up with the NULL reporter on those streams.

i’ll see if I can make a fix against 3.1 so we an issue a maintenance release with it resolved. this may take a day or so.

ckw

-- 
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at https://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/029d84c3-3c33-409c-af67-742554deb506%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

—
Chris K Wensel




Chris K Wensel

unread,
Sep 13, 2016, 5:48:18 PM9/13/16
to cascadi...@googlegroups.com
OK, I do have a trivial fix i’m testing locally before pushing through the build/publish pipeline. should see a 3.1.2 wip out tomorrow morning.

ckw


For more options, visit https://groups.google.com/d/optout.

—
Chris K Wensel




Chris K Wensel

unread,
Sep 13, 2016, 11:24:02 PM9/13/16
to cascadi...@googlegroups.com
I had a trivial fix, but a bad test. Just re-worked it, probably won’t see a wip until tomorrow afternoon.

ckw


For more options, visit https://groups.google.com/d/optout.

—
Chris K Wensel




ANIKET MORE

unread,
Sep 28, 2016, 7:19:06 AM9/28/16
to cascading-user
Hi Chris,

Did you get a chance to work on this ?

Thanks,
Aniket

Chris K Wensel

unread,
Sep 28, 2016, 11:40:31 AM9/28/16
to cascadi...@googlegroups.com

For more options, visit https://groups.google.com/d/optout.

ANIKET MORE

unread,
Oct 4, 2016, 6:13:45 AM10/4/16
to cascading-user
Thank you Chris.

Chris K Wensel

unread,
Oct 4, 2016, 11:26:20 AM10/4/16
to cascadi...@googlegroups.com

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages