Globus transfer times out GSHError

[Tue Boesen]

This is my first time using Globus, so this might just be a dumb rookie mistake, but I'm trying to download a very large dataset (11 TB), that someone has shared with me Globus.

However when I try to download this dataset I get the following error:

{
"context": [
{
"operation": "Directory List / File Scan",
"path": "/raw/"
}
],
"error": {
"details": "Error\nEndpoint: ProteinNet12 (b57a05f6-d8b9-11ea-9ef6-0aba3c43875b)\nServer: transfer07.o2.rc.hms.harvard.edu:2811\nCommand: MLSC /raw/\nMessage: The operation timed out\n---\nDetails: Timeout waiting for response\n",
"type": "GSHError"
}
}  

Obviously I get a timeout, but I can't figure out whether this happens because the folder is full of millions of files that add up to 11 TB, so a file scan times out because there is just too many files, for the system to scan before time out, or whether it just means that there is no connection at all.

A note on the files I'm trying to get. It is a raw dataset from a scientific article, that the author has shared with me, but I don't know anyhing about the format of it, except that it is roughly 11TB, so it could be 1 file with that size or millions of files.

Does anyone have any suggestions as to what I should try here?

Cheers

Tue

[Eli Dart]

Are you sure the local packet filters for your DTN are permitting the GridFTP data connections?

Are you able to download from other DTNs via Globus?

Thanks,

Eli

--


To unsubscribe from this group and stop receiving emails from it, send an email to user-discuss...@globus.org.

--

Eli Dart, Network Engineer                          NOC: (510) 486-7600

ESnet Science Engagement Group                           (800) 333-7638

Lawrence Berkeley National Laboratory 

[Tue Boesen]

I just ran the tutorial test and was able to transfer those files to my desktop, but at that point I also noticed that the original file transfer is actually still on-going even though it initially reported failure.

I can see that it has downloaded 200GB already. So I guess the failure was just a timeout while trying to do an initial filescan (which probably makes sense since the folder contains millions of files from what I can see so far)

So thank you for the help.

Cheers

Tue

To unsubscribe from this topic, visit https://groups.google.com/a/globus.org/d/topic/user-discuss/uOkWlkPwGDs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to user-discuss...@globus.org.

[Eli Dart]

Yeah....I’ve seen cases where directories have so many files that simple operations (like running ‘ls’) take minutes.....

Glad it’s working for you....automated failure recovery is a good thing :)

Cheers,

Eli

[Gigi Kennedy]

Hi Tue,

We’re glad you reached out to this list! We are so fortunate to have users willing to share their wealth of Globus knowledge and desire to help.

Something I’d like to add to this - 

In this issue, the network may be experiencing other transfer traffic and therefore encountering networking issues. And you’re transferring to a Globus Connect Personal (GCP) endpoint. However, despite these faults you’re encountering, we show your transfer is progressing. You do not need to stop or cancel a transfer if you are encountering a fault, or a few faults. The fault may be intermittent, or one that needs your intervention, such as extending authentication on an endpoint. However, once past the fault, Globus will retry and then the transfer should continue. When a job has not progressed for 3 days, the transfer will expire. However, as long as it's making progress, your transfer will continue.

If you feel the amount of faults is excessive, and that your transfer is not making progress, we suggest you cancel and resubmit the transfer with the Transfer option - "sync - only transfer new or changed files”. This will skip over files that were already successfully transferred.

An additional note, here are the limitations that the Globus transfer service does have -https://docs.globus.org/faq/transfer-sharing/#are_there_any_limits_on_using_the_file_transfer_service

Best regards,

  Gigi

Globus Help Desk