Scoobi - CombineInputFormat

16 views

Skip to first unread message

Deepak Jain

unread,

Oct 21, 2014, 4:46:18 PM10/21/14

to scoobi...@googlegroups.com

I have a scoobi MR job that takes multiple text files as input. It takes 60 files as input and most of them have only 1 or 2 lines. The number of lines is equalent to number of mappers. I want to reduce this by using CombineInputFileFormat and specify 64MB for mapreduce.input.fileinputformat.split.maxsize so that only 1 mapper is used.

How does scoobi allow me to specify CombineInputFileFormat as InputFormat. Today i see the following

mapreduce.job.inputformat.class	com.nicta.scoobi.impl.mapreducer.ChannelsInputFormat	job.xml ⬅ programatically
scoobi.input.formats	26;org.apache.hadoop.mapreduce.lib.input.TextInputFormat

Regards,

Deepak

Reply all

Reply to author

Forward

0 new messages