Incomplete downloads through globus

[Scott Rifkin]

I went through the Globus download process and downloaded my folder from CIPRES earlier in the month. However, I just noticed that not everything transferred.  In particular, if I had a folder nested within another folder, the subfolder didn’t transfer. Also it didn’t transfer all the tasks. For example, in the image below you can see that the Sep2021Analyses folder has Data, Tasks, Fingers_2021, and confSubClades2 as subfolders.

Globus transfered Data and some (but not all) of the Tasks but didn’t transfer Fingers_2021 and confSubClades2

Here is a screenshot of the globus source endpoint for this Sep2021Analyses:

If I click on the Fingers_2021 subfolder, the send to globus button doesn’t appear:

I think I can get around this by going into each task individually, viewing the output, selecting all the checkboxes and then exporting to globus.  When I did this for one of the tasks in Fingers_2021, my globus transfer directory made a new Finger_2021 folder and populated it with the output from the specific task I transfered.  

Is there a way to do this without individually going into each task?

Thanks,

Scott

[Mark Miller]

Hi Scott,

Thanks for the inquiry,

Here is our developer's reply to your question.

CIPRES can't support download of sub-folders. Here is the reason:

If we support download of sub-folders, somebody can put all his folders under one base folder and try to download the base folder. And that defeats our rule of one folder at a time to avoid causing SDSC network problem.  For example, if I have 50 folders in my account, I will put all of them under a base folder call “all_my_results”. If we support download of sub-folders, I can just download the base folder “all_my_results” and I will get all my 50 folders. This could swamp our server for a long time, potentially.

So, sorry to say, we are constrained to just one folder at a time.

Sorry for that.

Best,

Mark Miller