using multiple reducers on an aggregator

[Jayant Durgad]

I have a custom Aggregator which I need to run in multiple reduce tasks for performance purposes (by setting mapred.reduce.tasks in the stepconfigdef). However when I do this, I am seeing that all tuples belonging to the same group key are not being sent to the same reduce task. Therefore the output of my aggregation is incorrect. Things work correctly when I use a single reducer (mapred.reduce.tasks = 1). The group key is a composite key of 8-9 fields, which are either Long or String or Enum. I have supplied the appropriate serializations for all the Enum classes used using TupleSerializationProps. So is this a problem of serialization & comparison, or is there something wrong in using Aggregator in this manner?

Fields groupFields = new Fields(a,b,c,d,e);

pipe = new GroupBy(pipe, groupFields);

Aggregator<Context> aggregator = new CustomAggregator<Context>(new Fields(z));

pipe = new Every(pipe, new Fields(f), aggregator, new Fields(a,b,c,d,e,z));

pipe.getStepConfigDef().setProperty("mapred.reduce.tasks", 10);

[Chris K Wensel]

Sorry, enums are not supported as keys.

Search the mail-list for an explanation.

ckw

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at https://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/c3d82367-64c8-4b90-8f70-2f73c001e370%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jayant Durgad]

Thanks, that was the problem.