Query: genome browser interaction with globus links

[Simon Butcher]

Hi,

One of our users told us that they are using https://genome.ucsc.edu/cgi-bin/hgCustom but when they supply a link to a file hosted on Globus (globus.org), they get an error message. I was able to confirm this  - it seems to be a quirk of globus that doesn't allow direct wget of globus links

e.g.

wget https://92bd86.8540.dn.glob.us/KMtest2020-08-18/sample_9.bw

 

doesn't get the actual file.

I spoke to Globus about it, and have requested it as a feature request. However , as Globus is a popular research data tool, i wondered if you would consider in the meantime, allowing globus links by modifying the way your tool does wget commands?

wget --header "X-Requested-With: XMLHttpRequest" https://92bd86.8540.dn.glob.us/KMtest2020-08-18/sample_9.bw

The above command was suggested by Globus as a workaround. It would be great if you could consider it

regards

Simon Butcher

Head of Research Applications

QMUL

Add Custom Tracks - BLAT An annotation data file in one of the supported custom track formats may be uploaded by any of the following methods: (Preferred) Enter one or more URLs for custom tracks (one per line) in the data text box.The Genome Browser supports both the HTTP and FTP (passive-only) protocols. Click the "Browse" button directly above the URL/data text box, then choose a custom track file from your local ... genome.ucsc.edu

[Matthew Speir]

Hi, Simon. 

Thank you for your question about using Globus links with UCSC.

Unfortunately, it doesn't seem to be as simple as adding that header to our requests. One of our engineers modified our code to send that header when we access Globus links but was still getting errors when trying to access it using the UCSC Genome Browser:

Error Expected Partial Content 206.
https://92bd86.8540.dn.glob.us/KMtest2020-08-18/sample_9.bw;byterange=0-:
416 Requested Range Not Satisfiable. rangeStart=0 rangeEnd=-1 Can't get
data socket for https://92bd86.8540.dn.glob.us/KMtest2020-08-18/sample_9.bw

It seems that Globus isn't honoring byte-range requests without an end byte, however, the end byte is optional according to the standard: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range. You can see this by trying this request (which fails with "Unknown error"):

curl --header "X-Requested-With: XMLHttpRequest" 'https://92bd86.8540.dn.glob.us/KMtest2020-08-18/sample_9.bw' -H "Range: bytes=0-"

but adding an end to the range request works:

curl --header "X-Requested-With: XMLHttpRequest" 'https://92bd86.8540.dn.glob.us/KMtest2020-08-18/sample_9.bw' -H "Range: bytes=0-5"

If you have any further questions, please reply to gen...@soe.ucsc.edu. All messages sent to that address are archived on a publicly-accessible Google Groups forum. If your question includes sensitive data, you may send it instead to genom...@soe.ucsc.edu.

Training videos & resources: http://genome.ucsc.edu/training/index.html

Want to share the Browser with colleagues? Host a workshop: http://bit.ly/ucscTraining

---

Matthew Speir

UCSC Cell Browser, Quality Assurance and Data Wrangler

Human Cell Atlas, User Experience Researcher

UCSC Genome Browser, User Support

UC Santa Cruz Genomics Institute

Revealing life’s code.

--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Public Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome+un...@soe.ucsc.edu.
To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome/AM6PR07MB46162D9E2863AF17AA70359EC1889%40AM6PR07MB4616.eurprd07.prod.outlook.com.

[Simon Butcher]

Hi Matthew,

Thanks for that information. Good news - I passed this onto Globus and they replied:

That looks to be a bug in our collections http interface. I have it added to our fix log which should be available in an upcoming release.

regards

simon

---

From: Matthew Speir <msp...@ucsc.edu>
Sent: 19 February 2021 23:17
To: Simon Butcher <s.bu...@qmul.ac.uk>
Cc: gen...@soe.ucsc.edu <gen...@soe.ucsc.edu>
Subject: Re: [genome] Query: genome browser interaction with globus links

 

[Simon Butcher]

Some good news from Globus on the support ticket:

We plan to release a fix next week that covers (1) range requests in the format "Range: <digit>-" and (2) a fix for curl/wget/genome.ucsc.edu to be able to download from a guest collection with a public ACL without a redirect page.

You can see this behavior here:

curl --range 2-  https://g-15be48.485e8.8540.sandbox2.zones.dnsteam.globuscs.info/file

Range support isn't fully rfc-compliant, but it should be sufficient for your use case.

simon

---

From: Matthew Speir <msp...@ucsc.edu>
Sent: 19 February 2021 23:17
To: Simon Butcher <s.bu...@qmul.ac.uk>
Cc: gen...@soe.ucsc.edu <gen...@soe.ucsc.edu>
Subject: Re: [genome] Query: genome browser interaction with globus links

 

[Luis Nassar]

Hello, Simon.

Great news indeed.

Thank you for keeping us in the loop. And let us know if there is anything else we can help with.

I hope this is helpful. Please include gen...@soe.ucsc.edu in any replies to ensure visibility by the team. All messages sent to that address are archived on our public forum. If your question includes sensitive information, you may send it instead to genom...@soe.ucsc.edu.

Lou Nassar
UCSC Genomics Institute

--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Public Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome+un...@soe.ucsc.edu.

To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome/AM6PR07MB4616E77B926FD190D345D30DC1659%40AM6PR07MB4616.eurprd07.prod.outlook.com.