Transfer multiple files in parallel

[Vitor Rezende Da Costa Aguiar]

Hello,

I'm currently running a file transfer with globus cli in which 4 files are being transferred simultaneously, no matter how many CPUs I use. I wonder if I can increase the number of files in parallel.

My endpoint is a remote CentOS server. In a slurm interactive session with 16 CPUs I have `globusconnectpersonal -start` running. In a different session a submit the transfer request. I've played with the number of CPUs in both, but only four files are transferred.

I'd appreciate any comments.

Thank you,

Vitor

[John Bresnahan]

Vitor,

Have you tried to alter the settings through the webapp? The link https://app.globus.org/file-manager/collections/<collection id>/servers/network will take you there. You can then select "custom" and there are a few options to work with including Max Concurrency.

John

[Vitor Rezende Da Costa Aguiar]

Hi John,

thanks for pointing that out. You right that I have Maximum Concurrency set to 4.

I don't see a "custom" option, but a button "Edit Network Use". When I click it, I get this message: "Network Use may only be changed on an endpoint subscribed to a Globus subscription. See www.globus.org/subscriptions for details."

Maybe that is because I'm using globus connect personal instead of the server version?

Thanks,

Vitor

[Vas Vasiliadis]

Hi Vitor,

The network parameters can only be changed on a subscribed endpoint (both Personal and Server). Your institution has a subscription, so you can add your personal endpoint to that subscription. I will follow up privately with more details.

Thanks,
Vas

> --
> You received this message because you are subscribed to the Google Groups "Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@globus.org.

[Vitor Rezende Da Costa Aguiar]

Thank you, John and Vas!

[john kennedy]

Hi Vitor,

The docs also indicate that concurrency/parallelism on both endpoints is considered.

https://docs.globus.org/globus-connect-server/v5/#setting_endpoint_network_use_options

"""

For a given transfer, the concurrency is calculated as the smallest value across the MaximumConcurrency values of both endpoints, and the maximum of the PreferredConcurrency of both endpoints. Parallelism is also calculated similarly, with an additional consideration for transfers with high latency (trans-oceanic transfers) where the parallelism is set to minimum of the Maximum Parallelism value set for both endpoints.

"""

Vas may have pointed this out in private communication but I thought I'd add a note here.

cheers

John