--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/74899b8e-e81e-4606-bcb3-f15b01ab1e76%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
From: Gnanesh Gujulva
Sent: April 21, 2015 6:19:46am PDT
To: cascadi...@googlegroups.com
Subject: Re: Is there a way to specify the number of mappers to be used when running a cascading job?
Thanks Cyberlord. I am tried the following setting for reduces and mappers. It does not seem to change the number of mappers or reducers being spawned. Anything else that I can try? What should I look in the logs to see if this property is being passed on to the Hadoop job controller.-Gnanesh
On Monday, April 20, 2015 at 5:35:33 PM UTC-4, Cyberlord wrote:Yes Gnanesh, you can use the following property to set number of reducers.Properties properties = new Properties();properties.setProperty("mapreduce.job.reduces", "1000");Flow flow = new Hadoop2MR1FlowConnector( properties ).connect.....For number of mappers, you can adjust your split size.RegardsOn Mon, Apr 20, 2015 at 2:26 PM, Gnanesh Gujulva <sendg...@gmail.com> wrote:Hi,I am trying to run a join of two data sets using Cascading. The join seems to take a long time when run on a hadoop cluster machine. On investigating, the cascading job spawns only one mapper even when the input data is large. I believe that is the reason for the slowness in processing the data. Is there a way to specify programmaticaly the number of mappers/reducers in Cascading. I tried the following but it still is creating only one mapper.Properties properties = new Properties();properties.setProperty("mapreduce.job.maps", "21");AppProps.setApplicationJarClass( properties, Main.class );Hadoop2MR1FlowConnector flowConnector = new Hadoop2MR1FlowConnector( properties );Anyone have success in configuring the number of mappers/reducers from Cascading API?ThanksGnaneshCascading API version: 2.6.0Hadoop platform: Hortonworks 2.6
--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/c00cd681-b57b-46ed-8eb1-ecc948ca5682%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/c00cd681-b57b-46ed-8eb1-ecc948ca5682%40googlegroups.com.