What is the best practice of setting reducer number in cascading ?

Showing 1-7 of 7 messages
What is the best practice of setting reducer number in cascading ? Jeff Zhang 6/24/11 12:56 AM
I know that I can set properties to set the reducer number, but this
method will cause each mapreduce job in flow has the same number of
reducer.
But some sub jobs do not need too many reducers. Actually I just want
some heuristic method, such as dynamically setting the reducer number
according the input size. Does cascading support such method ?


Jeff Zhang
Re: What is the best practice of setting reducer number in cascading ? Chris K Wensel 6/24/11 6:49 AM
All jobs in a Flow will have the same number of reducers, currently no way to have it dynamic or overridden.

Your best option is to break your flow into multiple flows if you need them to have different properties.

chris

> --
> You received this message because you are subscribed to the Google Groups "cascading-user" group.
> To post to this group, send email to cascadi...@googlegroups.com.
> To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.
>

--
Chris K Wensel
ch...@concurrentinc.com
http://www.concurrentinc.com

-- Concurrent, Inc. offers mentoring, support for Cascading

Re: What is the best practice of setting reducer number in cascading ? Paul Lam 2/2/12 4:17 AM
Has there been any update regarding the number of reducers in a flow?
Re: What is the best practice of setting reducer number in cascading ? kkrugler 2/2/12 7:34 AM

On Feb 2, 2012, at 4:17am, Paul Lam wrote:

Has there been any update regarding the number of reducers in a flow?

I don't remember the context here - could you provide more details?

-- Ken

--------------------------
Ken Krugler
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr




Re: What is the best practice of setting reducer number in cascading ? Chris K Wensel 2/2/12 9:04 AM

Cascading 2 has the ability to let individual Pipe instances override the current MR job properties, or the whole Flow properties..

so calling groupBy.getProcessConfigDef() will let you override the default reducers for that job.

ckw

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.


Re: What is the best practice of setting reducer number in cascading ? Paul Lam 2/2/12 9:09 AM
Basically, is Cascading able to intelligently set the number of reducers for each Flow?
Re: What is the best practice of setting reducer number in cascading ? Chris K Wensel 2/2/12 9:11 AM
Basically, is Cascading able to intelligently set the number of reducers for each Flow?


Cascading has no magic in it that you would end up working against..

ckw