an alternate implementation to support CombineFileInputFormat

[Sangjin Lee]

https://github.com/cwensel/cascading/pull/19

Per earlier email thread, I'm playing with the idea of adding the CFIF support directly into Hfs. The multi-source support naturally falls from it. The temp hfs is also addressed via the same configuration.

The main addition is Hfs.handleCombineFileInputFormat() and several setter/getter methods. I shied away from creating a separate inner class for this so as not to create an additional object per Hfs...

I'd appreciate your feedback and suggestions on it. Thanks!

Sangjin

[Chris K Wensel]

OK, this has been incorporated.

Note to users, CombinedHfs has been removed in favor of pushing that functionality directly into Hfs, and enabled via a property setting passed to the FlowConnector or set in the default Hadoop settings.

If enabled, TempHfs will also use the split combining inside a Flow.

What we aren't doing is setting the max split size but leaving it the default of zero, which tells hadoop to combine multiple files from the same rack into a block. I'm unsure if there is an upper limit to this per file or by bytes, but seems it could be non-intuitive and maybe problematic.

One option is to add an additional flag that automatically makes the max split size the same as the current default block size. this lets each map task deal with the same amount of data and a likely reduced throughput, esp when some files fill multiple blocks and some don't.

If this seems like a reasonable addition, comment.  

ckw

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Chris K Wensel

ch...@concurrentinc.com

http://concurrentinc.com