My cascading program sends 30 MB files as output

[Srinivas Pachari]

I have a cascading job that outputs 30 25MB files. Is there anyway I can reduce it to 128 mb files each. I tried -Dmapreduce.job.reduces=1. It does not seem to work. I set the number of reducers to 1 and that did not work. Any guidance would be helpful

[Chris K Wensel]

If there are no reducers in the final MR job, then that property will be ignored.

you may need to introduce a final GroupBy at the end of the pipeline to force a reducer.

ckw

On Aug 11, 2017, at 6:58 AM, Srinivas Pachari <sriniv...@gmail.com> wrote:

I have a cascading job that outputs 30 25MB files. Is there anyway I can reduce it to 128 mb files each. I tried -Dmapreduce.job.reduces=1. It does not seem to work. I set the number of reducers to 1 and that did not work. Any guidance would be helpful

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at https://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/eae184b9-4aca-49b4-9424-887604d7f8cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

—

Chris K Wensel

415-203-5022

ch...@wensel.net

https://www.linkedin.com/in/cwensel

[Srinivas Pachari]

Hello Chris,

I have tried that and it does provide 1 file at that step.

But when the sink job is executed, each split seems to be directly written to a file. The sink job also looks like a map reduce job. Is there a way to configure the sink job's reducers?

Regards,

Srinivas

[Chris K Wensel]

you will need to provide a code sample in order to understand what you are attempting.

ckw

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/7a7bd294-3d02-47b8-b016-2b89ded4f925%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.