What is the best practice of setting reducer number in cascading ?

804 views
Skip to first unread message

Jeff Zhang

unread,
Jun 24, 2011, 3:56:10 AM6/24/11
to cascading-user
I know that I can set properties to set the reducer number, but this
method will cause each mapreduce job in flow has the same number of
reducer.
But some sub jobs do not need too many reducers. Actually I just want
some heuristic method, such as dynamically setting the reducer number
according the input size. Does cascading support such method ?


Jeff Zhang

Chris K Wensel

unread,
Jun 24, 2011, 9:49:17 AM6/24/11
to cascadi...@googlegroups.com
All jobs in a Flow will have the same number of reducers, currently no way to have it dynamic or overridden.

Your best option is to break your flow into multiple flows if you need them to have different properties.

chris

> --
> You received this message because you are subscribed to the Google Groups "cascading-user" group.
> To post to this group, send email to cascadi...@googlegroups.com.
> To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.
>

--
Chris K Wensel
ch...@concurrentinc.com
http://www.concurrentinc.com

-- Concurrent, Inc. offers mentoring, support for Cascading

Paul Lam

unread,
Feb 2, 2012, 7:17:22 AM2/2/12
to cascadi...@googlegroups.com
Has there been any update regarding the number of reducers in a flow?

Ken Krugler

unread,
Feb 2, 2012, 10:34:07 AM2/2/12
to cascadi...@googlegroups.com
On Feb 2, 2012, at 4:17am, Paul Lam wrote:

Has there been any update regarding the number of reducers in a flow?

I don't remember the context here - could you provide more details?

-- Ken

--------------------------
Ken Krugler
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr




Chris K Wensel

unread,
Feb 2, 2012, 12:04:49 PM2/2/12
to cascadi...@googlegroups.com

Cascading 2 has the ability to let individual Pipe instances override the current MR job properties, or the whole Flow properties..

so calling groupBy.getProcessConfigDef() will let you override the default reducers for that job.

ckw

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.
--
Chris K Wensel

Paul Lam

unread,
Feb 2, 2012, 12:09:52 PM2/2/12
to cascadi...@googlegroups.com
Basically, is Cascading able to intelligently set the number of reducers for each Flow?

Chris K Wensel

unread,
Feb 2, 2012, 12:11:57 PM2/2/12
to cascadi...@googlegroups.com
Basically, is Cascading able to intelligently set the number of reducers for each Flow?


Cascading has no magic in it that you would end up working against..

ckw

--
Chris K Wensel
Reply all
Reply to author
Forward
0 new messages