hidden (dot) files not subject to --exclude? (Globus CLI)

[Jason Sydes]

Hi there Globus Community,

It's entirely possible I'm doing this wrong. 

I'm trying to transfer 3 files using the Globus CLI (m*subreads.bam, m*subreads.bam.pbi, and barcodes.fa). It would be nice if I could avoid having to make 3 separate transfers, and instead be able to rely upon a mix of --recursive, --exclude and --include. (Yes, I could always use that approach as a work around if needed.)

No matter what approach I take, it seems like I cannot --exclude the hidden (dot) files and folders. Here are three examples:

globus transfer --recursive $ENDPOINT1:/large/data $ENDPOINT2:/large/data \
    --include='m*subreads.bam*' --include='barcodes.fa' \
    --exclude='*' --exclude='.hidden'

globus transfer --recursive $ENDPOINT1:/large/data $ENDPOINT2:/large/data \
    --exclude='.hidden' \
    --include='m*subreads.bam*' --include='barcodes.fa' \
    --exclude='*'

globus transfer --recursive $ENDPOINT1:/large/data $ENDPOINT2:/large/data \
    --exclude='.??*' \
    --include='m*subreads.bam*' --include='barcodes.fa' \
    --exclude='*'

For all three of those examples, all the .hidden folders get transferred over, even though I specifically don't want them to be transferred (my specific application requires the .hidden folder to be transferred at a later step, independent of the initial transfer).

(Note: Yes, the correct three "normal" / "desired" files get transferred over just fine). 

I'm curious if this is a bug, a feature, or if I'm just making a some sort of syntactic mistake(s)?

I'm running the latest version of Globus CLI:

❯ globus version
Installed version:  3.27.0
Latest version:     3.27.0

You are running the latest version of the Globus CLI

 

Thanks in advance for any assistance / insight you might be able to provide,

Jason

[J. Lewis Muir]

On 03/30, Jason Sydes wrote:
> For all three of those examples, all the .hidden folders get transferred
> over, even though I specifically don't want them to be transferred (my
> specific application requires the .hidden folder to be transferred at a
> later step, independent of the initial transfer).
>
> (Note: Yes, the correct three "normal" / "desired" files get transferred
> over just fine).
>
> I'm curious if this is a bug, a feature, or if I'm just making a some sort
> of syntactic mistake(s)?

Hi, Jason!

I've noticed the same. It's a real shame. In my experience, the
--include and --exclude options only apply to files, not directories!

You've probably already seen it, but the "Include and Exclude" section
of the "globus transfer" CLI reference

https://docs.globus.org/cli/reference/transfer/

says:

The --include and --exclude options are evaluated in order
together to determine which files are transferred during recursive
transfers. Earlier --include and --exclude options have priority over
later such options, with the first option that matches the name of
a file being applied. A file that does not match any --include or
--exclude options is included by default, making the --include option
only useful for overriding later --exclude options.

For example, "globus transfer --include *.txt --exclude *" will only
transfer files ending in .txt found within the directory structure.

And for the --include and --exclude options, it says:

--include TEXT

Include files found with names that match the given pattern in
recursive transfers. Pattern may include "*", "?", or "[]" for
Unix-style globbing. This option can be given multiple times along
with --exclude to control which files are transferred, with earlier
options having priority.

--exclude TEXT

Exclude files found with names that match the given pattern in
recursive transfers. Pattern may include "*", "?", or "[]" for
Unix-style globbing. This option can be given multiple times along
with --include to control which files are transferred, with earlier
options having priority.

You'll notice that it only talks about files, never directories....

Regards,

Lewis

[Jason Sydes]

Ah, sharp reading Lewis! I didn't notice that it only applies to files, not directories. 

Thanks for pointing that out Lewis! Well, that's too bad, but at least I have clarity now, and can move forward with my work around.

Thanks again!

Jason

PS Globus developers: Could this be a feature request? Sure would be nice to have...

[Brigitte Raumann]

Jason,

The Globus Transfer service supports filtering of both files and directories.  However, as noted, the Globus CLI only supports filtering of files.  The Transfer service filter rule logic can quickly become complex when applying multiple filters, particularly when applying a mix of file and directory filters.  We intentionally omitted the option to filter directories in the CLI so that users could more easily reason about the impact of a filter on their transfer.  That said, we can add CLI support for applying filters to directories to the product backlog, especially now that we know there is a use case for it.

Although you can't use the CLI, you can use the Globus web app at app.globus.org to apply filters on directories. You will find the option to apply filter rules under Transfer & Timer Options in the File Manager.  I recommend you familiarize yourself with the logic that the Transfer service uses for filtering and confirm that your transfers are behaving as expected.  Also, as a side note, the filter rules under the Transfer & Timer Options apply to transfers only and are independent of the filters that can be applied to directory listings in the web app.  

Brigitte

On behalf of the Globus Team

[Jason Sydes]

Hi Brigitte,

Thanks for your response! I just ended up switching back to the Globus Python SDK, which doesn't have restrictions on files/folders use in filters.

I had been hoping to use the Globus CLI, just to "simplify" debugging a little bit (you could say it's easier to copy/paste troublesome command line invocations of the Globus CLI rather than running it through Python, at least that was my hope?). I'm not sure I had really good reasons for wanting the CLI besides that...

But in the end, the SDK is actually relatively easy to use, and the filters work fine. So I went with the SDK, and it seems to be working fine.

Thanks,

Jason

---

From: Brigitte Raumann <brau...@globus.org>
Sent: Wednesday, April 3, 2024 7:29 PM
To: Discuss <dis...@globus.org>
Cc: Jason Sydes <sy...@uoregon.edu>; J. Lewis Muir <jlm...@imca-cat.org>; Discuss <dis...@globus.org>
Subject: Re: [Globus Discuss] hidden (dot) files not subject to --exclude? (Globus CLI)