Loading 2TB file

[Bala Kasaram]

Hi Team,

I am working for POC for Gobblin -> https://www.1digitalblog.com/gobblin-universal-distributed-big-data-integration-framework/

Which will load delimited files and fixed width files load into HDFS with validation.  but need some basic guide to send text files to hdfs kind code. any help.

But Can I load 2TB files [txt files] in my external harddisk into HDFS/Amazon S3 using Gobblin with high speed. I have 1GB internet speed. I want to know whether Gobblin will send data load cluster [multiple threads] or It will load 2TB file as same as linux copy command - single thread? 

[Shirshanka Das]

Gobblin's file-based source currently partitions (and multi-threads / distributes ) based on number of files. It doesn't split large files. 

HTH

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-user...@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/e2dfe7e9-e99e-490e-8692-13691a1476de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.