Server v5 installation on CentOS 6

[Chandler Sobel-Sorenson]

Hi all,

I understand CentOS 6 is ancient and EOL and does not meet the official Requirements for GCS, but we have an old cluster storage server with this OS needing to share its data.  Right now, due to international location, Globus is the only option that would work, short of purchasing a 20TB disk and copying and mailing the data.

Anyway I don't expect support if anything goes wrong but just thought I'd check in to see if there is some sort of impossible configuration that GCS requires that would prevent it functioning with CentOS 6, before I invest too much time in trying to get it running?

Thanks

[Karl Kornel]

Hi Chandler,

If I remember correctly, part of GCSv5 is implemented as an Apache module, so that’s might be your biggest issue (I assume that CentOS 6’s Apache API is different).

Could you run a newer-OS VM, with its own public IP (or shared, if needed) and read-only access to the data?

~ Karl

On Nov 20, 2023, at 2:55 PM, Chandler Sobel-Sorenson <sc...@arizona.edu> wrote:



--
You received this message because you are subscribed to the Google Groups "Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@globus.org.

[Chandler Sobel-Sorenson]

Thanks Karl.  Hopefully don't have to go down that route.  

I did get GCP running and published to Globus.  Our IT dep't added the endpoint to our Globus Plus Users group, which is what it asked for when I tried to share some data from it.  This is different than our main GCSv5 endpoint, which is managed under our campus subscription.  I thought this would be good to try since we may never transfer data from this server again after this, and GCP is supposed to work around the firewall, so don't need to bother NetOps dep't with Firewall Change requests and so on.

 I created a guest collection on a subdirectory and shared with the intended recipient, and they were able to open and browse the collection.  However, problem with them receiving the data. 

They sent me a screenshot of the Task and errors produced: 

Aborting: Error: The endpoint '<foo> (x)' needs to be upgraded. It does not currently support UDT NAT traversal. Exited due to signal 6 (SIGABRT)

internal transfer service error - please contact sup...@globus.org

I don't know what UDT is but it sounds like a firewall issue.  Although our server is behind a firewall, I was under the impression GCP figured out ways to work around that.  I also just downloaded GCP the other day, version 3.2.2, so not sure what needs to be upgraded.  Google couldn't find any results for "UDT NAT traversal" so guess I'm the first to have this strange error.

Anyone have any ideas what the issue is and possible solutions?  Thought I'd check here first.  

Does it maybe have to do with the endpoint being public/private?  It started out as private but told me to make it public so could be added to Globus Plus Users group, so I did that, then changed back to private after it was added.  The guest collection says it's Public though and don't see a way to change that.  Globus does some magical things I don't quite understand it all but this is all I could think off top of my head.  Maybe make the endpoint public too would help?  I'm not sure what info exactly becomes public but probably not an issue since our main GCSv5 endpoint is listed as public already.

Or: Is it something the recipient needs to e-mail Globus support about, as indicated in the error message?

Thanks

[Karl Kornel]

Hi Chandler,

 

Is the '<foo> (x)' endpoint your newly-created GCP endpoint, or the intended recipient’s endpoint?

 

~ Karl

[Chandler Sobel-Sorenson]

Oh yes sorry about that.  It's indeed my GCP endpoint or, rather, the "foo" and "x" refer to the name and UUID for the Guest Collection itself that I created.  Thanks

[Lev Gorenstein]

Hi Chandler,

Please check out this suggestion in the GCP troubleshooting guide: https://docs.globus.org/globus-connect-personal/troubleshooting-guide/#gcp-network-troubleshooting-step1 and try transferring couple small files between your GCP collection and the Globus Tutorial Endpoint (in both directions).

If transfers to the tutorial endpoint work, then there is a good chance that the problem is on the recipient's side.  if it fails, you'll know your side also has some contribution.  At the very least, it would help deciding which one of you should start drafting that email to sup...@globus.org :)

Also, is the destination endpoint also a GCP (and maybe the recipient should try to update their version of the software first)? GCP-to-GCP adds an extra complexity for firewalls and NAT traversal, and while GCP tries very, sometimes such transfers are not possible (see more details and suggestions in this section: https://docs.globus.org/globus-connect-personal/troubleshooting-guide/#troubleshooting_ice_negotiation_issues).  

Lev

P.S. Finally, if push comes to shove, maybe consider a two-step workaround described in the above link:

1. Transfer from the source GCP to an intermediate landing zone on a GCS collection (maybe your local cluster scratch or something similar... inside your data center, so fast);
2. Then create the share off of that landing zone and have the recipient transfer from there instead.
3. Delete the intermediate when done.

Alternatively, this can be done by the recipient themselves (transfer from your GCP to an interim landing zone on their cluster, then to their final GCP). Sounds longer, but might be easier and quicker overall (especially if it's a "we just need it once" scenario).  However, if you need to do it multiple times, consider this handy Two-Stage Transfer Flow.

--

Lev Gorenstein
Solutions Architect
Globus // University of Chicago

e: l...@globus.org

[Michael Link]

Hi Chandler,

The current release of GCP is built to support glibc 2.17, which covers
most active distros (back to centos 7). I assume your CentOS 6 has an
updated glibc to have gotten it even partially working. UDT is the
protocol that allows transferring between GCP endpoints, and the failure
simply means the driver for that couldn't be loaded.

We don't support CentOS 6, but for a short term setup you should be able
to use an older version that was built for that platform. See
https://docs.globus.org/ca-update-2022/#globus_connect_personal_v2 to
find a link to version 3.0.4. You'll have to follow the steps to update
the trust roots in that version, but after that it should have no
problem loading your existing endpoint config.

Mike

[Chandler Sobel-Sorenson]

Hey all, thanks for the feedback.

I was able to transfer some files successfully to the Globus Tutorial Endpoint using the File Manager, indicating everything is fine on my end.

However, last night, before I saw your messages, on a whim, I started up a globusconnect -trace process, and it indicates otherwise!

In fact, as soon as I open the Guest Collection to list the contents, these are first messages printed:

Got connection from ('127.0.0.1', 50580)
GCP-3.2.2L
Sock fd: 8
[30596] Tue Nov 28 11:03:12 2023 :: Unable to load UDT driver:
globus_xio: driver activation failed.
globus_extension_module: Couldn't dlopen libglobus_xio_udt_driver.so in /newwing/wing2/users/scar/.local/src/globusconnectpersonal-3.2.2/gt_amd64/lib (or LD_LIBRARY_PATH): file not found

Why bro?  It's right there:

$ ls -l /newwing/wing2/users/scar/.local/src/globusconnectpersonal-3.2.2/gt_amd64/lib/libglobus_xio_udt_driver.so
-rwxr-xr-x 1 scar admin 11888432 May 16  2023 /newwing/wing2/users/scar/.local/src/globusconnectpersonal-3.2.2/gt_amd64/lib/libglobus_xio_udt_driver.so

$

I checked globusconnect -start process environment (/proc/PID/environ), which is comprised of 2 python scripts, relaytool, then ssh.  The first python script, gc-ctrl.py, has LD_LIBRARY_PATH="/opt/mpich-install/lib:" while the following 3 processes have LD_LIBRARY_PATH="/opt/mpich-install/lib:/newwing/wing2/users/scar/.local/src/globusconnectpersonal-3.2.2/gt_amd64/lib".

I tried unset'ing LD_LIBRARY_PATH and then started GCP like: LD_LIBRARY_PATH="/newwing/wing2/users/scar/.local/src/globusconnectpersonal-3.2.2/gt_amd64/lib" globusconnect -start.  Now, gc-ctrl.py, has LD_LIBRARY_PATH="/newwing/wing2/users/scar/.local/src/globusconnectpersonal-3.2.2/gt_amd64/lib" and the second python script, gc.py, gets double the fun with LD_LIBRARY_PATH="/newwing/wing2/users/scar/.local/src/globusconnectpersonal-3.2.2/gt_amd64/lib:/newwing/wing2/users/scar/.local/src/globusconnectpersonal-3.2.2/gt_amd64/lib" but relaytool and ssh get triple trouble with LD_LIBRARY_PATH="/newwing/wing2/users/scar/.local/src/globusconnectpersonal-3.2.2/gt_amd64/lib:/newwing/wing2/users/scar/.local/src/globusconnectpersonal-3.2.2/gt_amd64/lib:/newwing/wing2/users/scar/.local/src/globusconnectpersonal-3.2.2/gt_amd64/lib".

Despite all that, bro is still not getting a clue and still can't dlopen libglobus_xio_udt_driver.so.  Why?

Mike, you're absolutely right that I've more than likely hacked a newer glibc in here at some point to squeeze more life out of our server (it can handle it it's been in air conditioning its whole life).  strings /lib64/libc.so.6 | grep GLIBC does spit out GLIBC_2.17.

relaytool is the only Globus ELF here, python and ssh have been working as needed.

The dependent libraries are:

$ ldd relaytool
linux-vdso.so.1 =>  (0x00007ffe3df20000)
libcrypto.so.1.1 => not found
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003042c00000)
libm.so.6 => /lib64/libm.so.6 (0x0000003040800000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003041c00000)
libc.so.6 => /lib64/libc.so.6 (0x0000003040400000)
/lib64/ld-linux-x86-64.so.2 (0x0000555e17e40000)

$

I guess the problem is missing libcrypto then?  We happen to have a copy at /opt/anaconda2/lib/libcrypto.so.1.1 maybe that's what bro needs... umm wow I think it helped?  LD_LIBRARY_PATH="/opt/anaconda2/lib" globusconnect -start and I don't see the Unable to load UDT driver error anymore from the trace program.  My hard work payed off 😁

Other thoughts?  I guess I'll ask our colleague to try the transfer again.  If it works then maybe next time I'll try grabbing the older GCP 3.0.4 to see if it runs any easier...

[Karl Kornel]

Hi Chandler,

 

libcrypto is one of the two libraries that make up OpenSSL.  And it’s not surprising if OpenSSL is one of the things that have gone through major changes from RHEL 6 to 7!

 

OpenSSL is used by GridFTP to secure the data channel of a transfer, when transfer encryption is enabled.  (I think it’s also used for securing GridFTP control traffic from Globus HQ, but maybe that was in GCSv4 only?).  You might want to copy over the corresponding libssl.so too, just in case: OpenSSL provides libcrypto for the crypto routines, and libssl for protocols.

 

As for testing, how about spinning up Globus Connect Personal on a wifi-connected work machine, giving yourself access to the share (if needed), and doing a transfer?  If your work wifi is anything like ours, it’s firewalled off from the rest of the campus network.  Do a large-ish transfer, and once it gets going, look for active UDP connections.  If you see them, that means UDP is working!

 

~ Karl

 

From: dis...@globus.org <dis...@globus.org> on behalf of Chandler Sobel-Sorenson <sc...@arizona.edu>
Date: Tuesday, November 28, 2023 at 12:27 PM
To: Discuss <dis...@globus.org>
Subject: Re: [Globus Discuss] Server v5 installation on CentOS 6

[Chandler Sobel-Sorenson]

Our collaborator was able to successfully download/receive the data! 🎉

Have a good weekend!

-Chandler