Using GLOBUS as per RSYNC
202 views
Skip to first unread message
Andor Kiss
Sep 21, 2021, 8:06:08 AM9/21/21
to Discuss
Hello,
I copied a genome assembly project (12 TB) from my local HPC cluster to our lab workstation using RSYNC - to ensure that all my symlinked files and directories withiin the 12 TB were preserved.
When I tried to copy this "backup" from my workstation to Ohio Supercomputer Center, I was recommended to try GLOBUS. Using the GUI, I tried the transfer, but only 7 TB was moved. All the symlinked files/directories were not transferred.
Is there an equivalent command in GLOBUS to this command in RSYNC?
$ rsync -ar --append-verify --stats user:source target
TIA
Michael Link
Sep 21, 2021, 11:56:59 AM9/21/21
to dis...@globus.org
Hi Andor,
Globus will ignore symlinks discovered as part of a recursive transfer.
When part of the submitted paths (entries that were selected in the
web interface), Globus will treat symlinks as normal files/directories,
copying the data they point to.
More info here:
https://docs.globus.org/faq/transfer-sharing/#how_does_globus_handle_symlinks
How did you calculate the size of the data set? I ask because, while
the Globus behavior will be different than that rsync command, it should
not have copied less data than the rsync command. I believe the
difference is only that rsync will have created the symlinks on the
destination, and Globus will not have. In the rsync case, the
additional symlinks should not increase the size of the dataset (beyond
the negligible size of the symlink itself).
Mike
On 9/21/2021 7:06 AM, Andor Kiss wrote:
>
> Hello,
>
> I copied a genome assembly project (12 TB) from my local HPC cluster
> to our lab workstation using RSYNC - to ensure that all my symlinked
> files and directories withiin the 12 TB were preserved.
>
> When I tried to copy this "backup" from my workstation to Ohio
> Supercomputer Center, I was recommended to try GLOBUS. Using the GUI, I
> tried the transfer, but only 7 TB was moved. All the symlinked
> files/directories were /not /transferred.
Globus will ignore symlinks discovered as part of a recursive transfer.
When part of the submitted paths (entries that were selected in the
web interface), Globus will treat symlinks as normal files/directories,
copying the data they point to.
More info here:
https://docs.globus.org/faq/transfer-sharing/#how_does_globus_handle_symlinks
How did you calculate the size of the data set? I ask because, while
the Globus behavior will be different than that rsync command, it should
not have copied less data than the rsync command. I believe the
difference is only that rsync will have created the symlinks on the
destination, and Globus will not have. In the rsync case, the
additional symlinks should not increase the size of the dataset (beyond
the negligible size of the symlink itself).
Mike
On 9/21/2021 7:06 AM, Andor Kiss wrote:
>
> Hello,
>
> I copied a genome assembly project (12 TB) from my local HPC cluster
> to our lab workstation using RSYNC - to ensure that all my symlinked
> files and directories withiin the 12 TB were preserved.
>
> When I tried to copy this "backup" from my workstation to Ohio
> Supercomputer Center, I was recommended to try GLOBUS. Using the GUI, I
> tried the transfer, but only 7 TB was moved. All the symlinked
Andor J Kiss
Sep 21, 2021, 12:45:41 PM9/21/21
to dis...@globus.org
Okay, thanks. Yeah, there's thousands of symlinks created in the FALCON
assembler, so I'll have to re-do the transfer via RSYNC.
Not sure about the size - maybe it was compressed vs uncompressed?
/A
assembler, so I'll have to re-do the transfer via RSYNC.
Not sure about the size - maybe it was compressed vs uncompressed?
/A
Reply all
Reply to author
Forward
0 new messages