Transfer speed of Globus and iRODS is slow

[Sanju Timsina]

Hello,

We are using the Globus endpoint to transfer files. The transfer speed is about 4 to 6 MB/s(for 2GB and 11 GB files).  Earlier we had the Globus server on a different server than the iRODS server. For testing, we tried installing the iRODS server on the same server as the Globus server but the performance hasn't improved. 

These are the irods and Globus version we are using:

irods: 4.2.8

Globus: 5.4.23.

Right now, we have 2 resource servers, one provider server, irods database, and Globus each on different servers.

Please suggest if there is anything that we can do to improve the performance.

Thank you,

Sanju Timsina

[James, Justin Kyle]

Couple of questions.  Are you doing any checksuming of the files on transfer? It is possible that we are doing duplicate reads because the client has to download the entire file to do the checksum (but only the first time).

Also, the original Globus iRODS plugin seems to be single threaded and we are working on moving that to multithreaded transfers.  That work has not been completed though.

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Sanju Timsina <timsina...@gmail.com>
Sent: Thursday, June 17, 2021 11:42 AM
To: iRODS-Chat <irod...@googlegroups.com>
Subject: [iROD-Chat:19823] Transfer speed of Globus and iRODS is slow

 

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/6a9d0711-ce34-4c5b-9237-a73cdaaff685n%40googlegroups.com.

[Sanju Timsina]

Hi James,

Yes, It seems I was checksumming the files on transfer. I will try transferring without doing any checksum.

Thank you,

Sanju

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/BN8PR03MB470595F6682CB5D0BA6E6D15C70E9%40BN8PR03MB4705.namprd03.prod.outlook.com.

[Sanju Timsina]

There was not much difference in the transfer speed without doing a checksum on files too.

Thanks,

Sanju

[Mauro Tridici]

Hi Sanju,

I’m not an iRODS expert, but I would suggest to do the following before checking GLOBUS and iRODS behaviour.

Please, make some bw tests:

- run iperf2 test involving source and destination servers and save the throughput value;

- run a scp/rsync remote copy (writing a 2GB file in a temporary folder on the same storage) and save the throughput value;

- run a scp/rsync remote copy (writing a 11GB file in a temporary folder on the same storage) and save the throughput value;

- run a scp/rsync remote copy (writing a 5GB file in a temporary folder on the same storage) and save the throughput value;

- run a scp/rsync remote copy (writing a 500MB file in a temporary folder on the same storage) and save the throughput value.

If the obtained values are not the ones you expected, I think you should check your network segments.

I hope it helps.

Mauro

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/7cf87b2f-347d-47f6-8eb7-031b3ca1843dn%40googlegroups.com.