Parallel Transfer of a File From Local HOST to IRODS Using Jargon.

[yada...@umn.edu]

Hi,

Do you have some examples of parallel file transfer from a local host machine to IRODS using Jargon. I tried using ParallelPutFileTransferStrategy for it, but I ended up having some exceptions. Are there any examples which can I be pointed at ?

Thanks

Pranjul

[Conway, Mike]

you just need to call DataTransferOperations.putOperation().  

like this, from the unit tests

@Test

public void testPutOneFile() throws Exception {

// generate a local scratch file

String testFileName = "testPutOneFile.txt";

String absPath = scratchFileUtils

.createAndReturnAbsoluteScratchPath(IRODS_TEST_SUBDIR_PATH);

String localFileName = FileGenerator

.generateFileOfFixedLengthGivenName(absPath, testFileName,

32 * 1024);

String targetIrodsFile = testingPropertiesHelper

.buildIRODSCollectionAbsolutePathFromTestProperties(

testingProperties, IRODS_TEST_SUBDIR_PATH + '/'

+ testFileName);

File localFile = new File(localFileName);

// now put the file

IRODSAccount irodsAccount = testingPropertiesHelper

.buildIRODSAccountFromTestProperties(testingProperties);

IRODSFileFactory irodsFileFactory = irodsFileSystem

.getIRODSFileFactory(irodsAccount);

IRODSFile destFile = irodsFileFactory

.instanceIRODSFile(targetIrodsFile);

DataTransferOperations dataTransferOperationsAO = irodsFileSystem

.getIRODSAccessObjectFactory().getDataTransferOperations(

irodsAccount);

dataTransferOperationsAO.putOperation(localFile, destFile, null, null);

assertionHelper.assertIrodsFileMatchesLocalFileChecksum(

destFile.getAbsolutePath(), localFile.getAbsolutePath());

}

On Aug 8, 2013, at 1:42 PM, <yada...@umn.edu>

 wrote:

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
 
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Mike Conway

DICE Center

Jargon, Java, Interface Developer

michael...@unc.edu

------------------------------------------------

Google voice/video: Michael....@gmail.com

Skype: michael.c.conway

[yada...@umn.edu]

Hi,

Will it be the most efficient way to transfer files between host machine and IRODS server ? My file sizes would roughly be around 20-40 GB. Eventually the IRODS server would be DFC-Main and the host machines would be somewhere at Minnesota.

Thanks

Best Regards

[Conway, Mike]

It should be fairly equivalent to the iput command…there are a few overheads in the Java code that the C code does not have…for example managing status callbacks and the like.

I'd be happy to help you tune this if you do use it.

MC

On Aug 8, 2013, at 2:08 PM, <yada...@umn.edu>

 wrote:

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
 
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[yada...@umn.edu]

Thanks Mike.

As if now, we are going to use it. Let me come back with some numbers.

Regards

Pranjul.

On Thursday, August 8, 2013 12:42:00 PM UTC-5, yada...@umn.edu wrote: