quota on shared endpoint?

[Wong-Barnum, Mona]

Hi:

Can quota be setup for a shared endpoint? We would like to be able to limit our user upload amount…is that possible?

Mona

*********************************************
Mona Wong
Web & Mobile Application Developer
San Diego Supercomputer Center

Believing we are in control is an
illusion that brings suffering.
*********************************************

[Karl Kornel]

On Wednesday, May 27, 2020 at 2:13:13 PM UTC-7, Wong-Barnum, Mona wrote:

Hi:

        Can quota be setup for a shared endpoint?  We would like to be able to limit our user upload amount…is that possible?

I don't think so,  Enforcing quotas is more of a function of the underlying file system.  But one way that shared endpoints can help is in the way that all files transferred to the shared endpoint have the file owner/group set to be that of the user who created the shared endpoint.  So if you have the ability to enable quotas on the underlying file system, that might do toe job!

Or, a more-involved solution: You could give a non-native Globus App access to the host endpoint. I think that would allow the App to pull the list of shared endpoints, and pull the list of transfers, to identify endpoints that have too much data coming in.

Or maybe there are other ways?  That's all I could think of off-hand.  I hope it helps!

[Wong-Barnum, Mona]

Hi Karl:

Thanks for your answer & ideas!

FYI, we do have a science gateway app and was hoping that there would a Globus solution to the quota issue so quota can be enforced BEFORE a transfer, not afterwards (:  I can see how this could be difficult for the Globus API to implement…performance would be a big issue.

cheers,

Mona

--
To unsubscribe from this group and stop receiving emails from it, send an email to developer-disc...@globus.org.

[Karl Kornel]

On Wednesday, May 27, 2020 at 5:47:47 PM UTC-7, Wong-Barnum, Mona wrote:

Hi Karl:

Thanks for your answer & ideas!

FYI, we do have a science gateway app and was hoping that there would a Globus solution to the quota issue so quota can be enforced BEFORE a transfer, not afterwards (:  I can see how this could be difficult for the Globus API to implement…performance would be a big issue.

 Oh, you're working on adding this to a science gateway app?   That gives me an idea that would be crazily inefficient, but might do what you want.

If you route the user through the Browse Endpoint helper, your app will get the list of files & directories the user wants to transfer to your shared endpoint, as well as the UUID of the source.  As long as you got a Globus Transfer token from the user, you could then do one or more directory list operations to check the size of the files to be transferred.  Assuming it is small enough, then you could set up a write ACL on the shared endpoint (giving the user access) and submit the transfer.

There is still a race condition, because the user could change the size of files before they are transferred.  And it's not the most efficient, because you might have to do many directory listings.  But it might work for you!

[Wong-Barnum, Mona]

Good morning Karl and thanks again!  That’s a good idea though we are expecting large data transfers…many TB per transfer…still might be worth trying to see if the performance hit outweighs the need to limit user transfers (:

Mona

--
To unsubscribe from this group and stop receiving emails from it, send an email to developer-disc...@globus.org.

*********************************************
    Mona Wong
    Web & Mobile Application Developer
    San Diego Supercomputer Center

    "I've learned that people will forget
    what you said, people will forget
    what you did, but people will never
    forget how you made them feel."
                                --- Maya Angelou
*********************************************

[Eli Dart]

Hmmmm....

I've blown quota using Globus transfers many times.

When this happens, the Globus transfer stops (but does not fail), goes into an error condition, and notifies me. I can then either cancel the transfer, or clear some space. If I clear enough space so that the transfer can proceed, the Globus error recovery logic simply continues the transfer after a minute or so, and the transfer completes normally.

Are you trying to protect users from themselves, or ensure that an automated workflow fits within quota constraints? In either case, it seems to me that the user should be aware of what they are doing, and make allowances to manage their space.  For example, I could start a big transfer that won't fit in quota, but shuttle files out of my storage area once they complete, thus making more space. If the data portal I was using to launch the transfer didn't give me a way to subset the transfer into bite-sized pieces, I would still be able to get my data moved. If you enforce quota at job start time, my options are limited.

Just trying to think about multiple angles here....

Thanks,

Eli

--

Eli Dart, Network Engineer                          NOC: (510) 486-7600

ESnet Science Engagement Group                           (800) 333-7638

Lawrence Berkeley National Laboratory 

[Wong-Barnum, Mona]

I think our current goal is to protect the storage space that we have which must be shared by every user of our gateway.  Most of our users are not (yet) tech savvy but giving options for power users is definitely a good idea!

Mona

On May 28, 2020, at 11:29 AM, Eli Dart <da...@es.net> wrote:

Hmmmm....

I've blown quota using Globus transfers many times.

When this happens, the Globus transfer stops (but does not fail), goes into an error condition, and notifies me. I can then either cancel the transfer, or clear some space. If I clear enough space so that the transfer can proceed, the Globus error recovery logic simply continues the transfer after a minute or so, and the transfer completes normally.

Are you trying to protect users from themselves, or ensure that an automated workflow fits within quota constraints? In either case, it seems to me that the user should be aware of what they are doing, and make allowances to manage their space.  For example, I could start a big transfer that won't fit in quota, but shuttle files out of my storage area once they complete, thus making more space. If the data portal I was using to launch the transfer didn't give me a way to subset the transfer into bite-sized pieces, I would still be able to get my data moved. If you enforce quota at job start time, my options are limited.

Just trying to think about multiple angles here....

Thanks,

Eli

*********************************************
    Mona Wong
    Web & Mobile Application Developer
    San Diego Supercomputer Center

    "Nothing is impossible, the word
    itself says 'I'm possible'!"
                              --- Audrey Hepburn
*********************************************

[Eli Dart]

Hi Mona,

I'm curious.....is there one global quota for everything in all science gateways? Or is each gateway instance (e.g. for a given scientific collaboration) given its own storage allocation?

Thanks,

Eli

--
To unsubscribe from this group and stop receiving emails from it, send an email to developer-disc...@globus.org.

[Rick Wagner]

Hi Eli,

Generally, a science gateway will have a single pool of storage for all of its users. So the gateway may need to check the incoming transfer size to avoid overwhelming its storage, even if other gateways or system users aren’t impacted.

—Rick

[Wong-Barnum, Mona]

Yes what Rick said (:

We do have separate quote for each user…just a matter of how it is enforced…enforcing it BEFORE a transfer starts is probably not a good option for large amount of data; for now we are enforcing it after the transfer.

Mona

> On May 28, 2020, at 2:29 PM, Rick Wagner <richard....@gmail.com> wrote:
>
> Hi Eli,
>
> Generally, a science gateway will have a single pool of storage for all of its users. So the gateway may need to check the incoming transfer size to avoid overwhelming its storage, even if other gateways or system users aren’t impacted.
>
> —Rick

*********************************************
Mona Wong
Web & Mobile Application Developer
San Diego Supercomputer Center

"I am not a product of my
circumstances. I am a product of
my choices."
*********************************************