Protocol Support Roadmap/Timeline

[Matt Snyder]

Hello Globus Group,

There is some confusion here on the roadmap/timeline for protocol support.  We've heard gridftp support will be phased out.  What does this mean and when?  Does this only refer to the obsolete globus toolkit?  If so, when? 

Also, confusingly, we've heard protocol support is switching to https.  Does this just refer to the "small" data download via browser and communication on the control channel with the globus SaaS?  The data channel between globus transfer nodes is unaffected, correct?

Any response or links to some information would be appreciated.

Have a great day!

-Matt

[Vas Vasiliadis]

Hi Matt,

There are _no_ plans to phase out GridFTP support within the Globus service - anything you may have heard to the contrary is incorrect.

Support for the Globus Toolkit ended in January 2018 [1], so perhaps that’s where there is some confusion. As part of that transition, users of the open source GridFTP implementation in the Globus Toolkit were asked to migrate to the Globus service and use Globus Connect Server [2] instead.

We continue to use GridFTP as the protocol for managed file transfers. We also added HTTPS support to the Globus service [3] for downloading “small” files directly via the browser (as you correctly pointed out), but that is in no way replacing GridFTP as a protocol.

Please feel free to reach out to me directly and discuss, if you have other concerns.

Thanks,
Vas

[1] https://github.com/globus/globus-toolkit/blob/globus_6_branch/support-changes.md
[2] https://docs.globus.org/globus-connect-server/
[2] https://docs.globus.org/globus-connect-server/v5/https-access-collections/

> --
> You received this message because you are subscribed to the Google Groups "Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@globus.org.

[Cherry, Andrew J.]

Hi Vas,

This does lead me to another question, and I think this is a good context to ask this question for the sake of clarity.  As a resource provider, I have fielded the "is GridFTP is going away?" question more than once, and I think terminology muddies the waters since different people may mean different things when they say "GridFTP protocol" -- they could be thinking about specific implementations (since there are now different methods of handling the control channel with v4 vs v5 along with different auth mechanisms) and not necessarily about the base transfer protocol under the hood.  

When I see this sort of question from an end user or site administrator, sometimes I find that they are not actually asking whether the underlying file transfer protocol is changing.  Often what they're really asking about is compatibility of server implementations with the Globus cloud service and whether they are going to experience vendor lock-in.  And I've also heard some users express concern about losing an open source high throughput file transfer mechanism.

We know that Globus is no longer maintaining the Globus Toolkit and the GSI authentication layer it relies on, but there does exist a third party open source fork of the Globus Toolkit code, the Grid Community Toolkit (maintained and supported by OSG).  I expect that there are probably sites embedded in the OSG ecosystem that are using this implementation in order to continue supporting workflows that rely on the legacy clients, and it seems probable that there might be instances of this that have also have some transfers brokered by the Globus service, particularly if they were previously running the toolkit or the OSG packages and not the full GCSv4 version with the overlying autoconfiguration pieces.

I think the more pertinent question for an end user or site admin is this one: once GCSv5 (which uses https for the control channel and native OAuth for authentication) reaches critical mass and GCSv4 begins to phase out, will the Globus cloud service continue to maintain compatibility with "traditional" GSI-based GridFTP servers like the Grid Community Toolkit?  Or will we eventually reach the point where GCSv5 is the only server option that works with your service?  This was perhaps a clearer picture when Globus maintained the toolkit and announced end of support, but it's a more ambiguous situation when we have third party server implementations, and I don't know if I've actually seen a clear statement about it.

Thanks....

-Andrew

---

From: dis...@globus.org <dis...@globus.org> on behalf of Vas Vasiliadis <v...@uchicago.edu>
Sent: Wednesday, January 27, 2021 9:19 AM
To: Matt Snyder <maat...@gmail.com>
Cc: Discuss <dis...@globus.org>
Subject: Re: [Globus Discuss] Protocol Support Roadmap/Timeline

 

[Matt Snyder]

Thanks Vas for the clear response.

[Vas Vasiliadis]

Hi Andrew,

Thanks for raising some important questions.

To your first concern, OSG has decided that GridFTP and Globus will not be part of their infrastructure in the future; their transition plan is described here: https://opensciencegrid.org/technology/policy/gridftp-gsi-migration/. As much as we would like to see continued use of Globus within the high energy physics community, we respect their decision to invest in building bespoke tools.

Regarding the broader question of supporting GSI-based GridFTP servers, we're guided by the voices of the community. We have shared plans for transitioning to a new security model based on OAuth, and there's no demand for ongoing support of both approaches from our users. The majority of legacy GridFTP servers are within the high energy physics community and, with their move away from GridFTP to other transfer tools, we are hard pressed to invest our limited resources in maintaining multiple security models and versions. We will continue to support GSI until the transition from GCSv4 to GCSv5 is complete. From that point forward, only the new security model in GCSv5 will be supported within the Globus service.

In the same spirit of being user driven, our focus has always been to make advanced data management capabilities available to every researcher, regardless of technical ability, while also reducing (ideally, eliminating!) the burden on system administrators that provide these capabilities to their users. It’s with this goal in mind that we introduced modern authN/authZ approaches like OAuth to the community a few years ago, and started the effort to rebuild Globus Connect Server. In addition to removing the friction of managing X.509 certs, GCSv5 also simplifies packaging, reduces firewall requirements by using port 443 for the control channel, and provides tools that give admins a lot more flexibility in managing their Globus deployments.

You also raised the spectre of vendor lock-in which I don’t believe is an issue in this context. The vast majority of Globus users simply use the web app to move and share data. Hence there are no barriers to switching to another service since Globus doesn’t own any of the user’s data -- the storage systems “behind” all Globus endpoints are under the sole control of their respective owners. Likewise, for those institutions that have integrated Globus capabilities into other systems (portals, science gateways, etc.), switching might require a bit more effort (e.g. to use another provider’s APIs) but that's no different than, for instance, deciding to use a different library in their code. Also, the fact that we use open standards like OAuth and OIDC means that any auth infrastructure they build is easily portable between solutions.

I hope this clarifies things and provides some added perspective. Happy to continue the conversation if you have other questions.

Cheers,
Vas

> On Jan 27, 2021, at 8:59 AM, 'Cherry, Andrew J.' via Discuss <dis...@globus.org> wrote:
>
> Hi Vas,
>
> This does lead me to another question, and I think this is a good context to ask this question for the sake of clarity. As a resource provider, I have fielded the "is GridFTP is going away?" question more than once, and I think terminology muddies the waters since different people may mean different things when they say "GridFTP protocol" -- they could be thinking about specific implementations (since there are now different methods of handling the control channel with v4 vs v5 along with different auth mechanisms) and not necessarily about the base transfer protocol under the hood.
>
> When I see this sort of question from an end user or site administrator, sometimes I find that they are not actually asking whether the underlying file transfer protocol is changing. Often what they're really asking about is compatibility of server implementations with the Globus cloud service and whether they are going to experience vendor lock-in. And I've also heard some users express concern about losing an open source high throughput file transfer mechanism.
>
> We know that Globus is no longer maintaining the Globus Toolkit and the GSI authentication layer it relies on, but there does exist a third party open source fork of the Globus Toolkit code, the Grid Community Toolkit (maintained and supported by OSG). I expect that there are probably sites embedded in the OSG ecosystem that are using this implementation in order to continue supporting workflows that rely on the legacy clients, and it seems probable that there might be instances of this that have also have some transfers brokered by the Globus service, particularly if they were previously running the toolkit or the OSG packages and not the full GCSv4 version with the overlying autoconfiguration pieces.
>
> I think the more pertinent question for an end user or site admin is this one: once GCSv5 (which uses https for the control channel and native OAuth for authentication) reaches critical mass and GCSv4 begins to phase out, will the Globus cloud service continue to maintain compatibility with "traditional" GSI-based GridFTP servers like the Grid Community Toolkit? Or will we eventually reach the point where GCSv5 is the only server option that works with your service? This was perhaps a clearer picture when Globus maintained the toolkit and announced end of support, but it's a more ambiguous situation when we have third party server implementations, and I don't know if I've actually seen a clear statement about it.
>
> Thanks....
>
> -Andrew
>

[Cherry, Andrew J.]

Hello Vas-

Thank you very much for the detailed answer!

-Andrew

[Doug Benjamin]

Hi Vas,

  Is GCSv5 open source?  Is there a specification document available for GCSv5? I ask because at BNL Scientific Data and Computing Center (SDCC) the majority of the storage for the LHC ATLAS experiment, KEK Belle II  and BNL RHIC experiments Phenix, sPhenix use dCache. Currently dCache gridftp servers work with GCSv4 servers. We are using a Globus endpoint (GCSv4) at BNL to transfer data between US HPC centers (NERSC, ALCF) and the rest of WLCG data centers worldwide. Instead of maintaining independent storage, it would be more efficient to be able to use dCache storage when Globus migrates to exclusively GCSv5.

Regards,

Doug Benjamin

[Vas Vasiliadis]

Hi Doug,

CGSv5 is covered by the Globus Connect Community Source Code License (https://www.globus.org/legal/source-license). Your question regarding dCache support is timely - I recently spoke with Paul Millar at DESY about this and we’re going to discuss further to see what’s possible. Stay tuned :-)

Cheers,
Vas

[Ryan Abernathey]

Doug, to answer your original question...

Is GCSv5 open source?

...with clarity: the license referred to here (https://www.globus.org/legal/source-license) is most certainly not an open source license. 

[Ian Foster]

To add my voice to Vas’ — we are definitely interested in GCS interface to dCache, so that we can link the worlds of US and other HPC centers that run GCS and the physics centers that run dCache. I suspect that we could get some funding for the required work, if we can find someone on the dCache side who can work with us. 

On Feb 17, 2021, at 10:42 AM, Doug Benjamin <dougbenja...@gmail.com> wrote:

[Doug Benjamin]

Thank you all for the responses. It is helpful.