configure number of reducers

[Dexin Wang]

I tried to set the number or reducers by doing:

Properties properties = new Properties();
properties.put("mapred.reduce.tasks", 10);
FlowConnector.setApplicationJarClass( properties,
MyClass.class );

but it doesn't seem to have any effect -- my jobs still have the same
number of reducers as before when it's not set.

Even if this works, how do I configure different number of reducers
for different jobs. My flow has a handful of jobs and I'd like to use
different number of reducers on them. Mainly, some of them will be
global sort/count, so I want just one reducer, while I want the
default number reducers configured by the cluster for some other jobs.

[Chris K Wensel]

What version of Cascading?

> --
>
> You received this message because you are subscribed to the Google
> Groups "cascading-user" group.
> To post to this group, send email to cascadi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cascading-user?hl=
> .
>
>

--
Chris K Wensel
ch...@concurrentinc.com
http://www.concurrentinc.com

[William Wang]

This is 1.0.9. It's not a big deal for this particular case, I'm just doing a "cat xxx/part-* | sort", heh.

But just curious how to set the various job parameters in general. Thanks.

[Chris K Wensel]

Looks like this was fixed in 1.0.13, per CHANGES.txt

1.0.13

  Fixed bug where non-String j.u.Property values where not being copied to the internal o.a.h.m.JobConf instance.