BigWig.

31 views
Skip to first unread message

dan

unread,
Mar 27, 2015, 12:47:11 PM3/27/15
to gen...@soe.ucsc.edu
Hello,

I am trying to upload BigWig files to my sessions on the browser.
My files are stored on copy (www.copy.com) which I use to generate public links. I then use the command below to upload the file:

track type=bigWig name="Melanocytes-" bigDataUrl=https://copy.com/4yJ9p3LQrNjB visibility=full smoothingWindow=4 windowingFunction=mean maxHeightPixels=100:50:8

I was able to do so up until about 10 days ago and then I started receiving messages that the link could not be open. I am trying to figure out where is the problem.

The link below is to one of my files. As you can see, the link is active and will direct you to the file. However, it doesn’t work on the browser


Please advise.

Thank you so much.

dan.




Hiram Clawson

unread,
Mar 27, 2015, 12:56:17 PM3/27/15
to dan, gen...@soe.ucsc.edu
Good Morning Dan:

We have received similar reports from other copy.com users.
Evidently copy.com has changed the access permissions, it does not
honor 'byte-range requests' which is a necessary option to use the big*
file formats.

Compare the copy.com response with the Apache server response below:

> $ time curl -I https://copy.com/k6euKF0mYGWAlFY7
> HTTP/1.1 200 OK
> Server: nginx
> Content-Type: application/octet-stream
> Content-Length: 196961177
> Connection: keep-alive
> Keep-Alive: timeout=20
> Access-Control-Allow-Origin: *
> Access-Control-Allow-Methods: POST, GET, PUT, PATCH, DELETE, OPTIONS
> Access-Control-Allow-Headers: X-AUTHORIZATION-ANON, X-AUTHORIZATION, X-CLIENT-VERSION, X-CLIENT-TYPE, X-CLIENT-OSVERSION, X-CLIENT-HWID, X-CLIENT-NAME, X-API-VERSION, X-REQUESTED-WITH, X-HTTP-METHOD-OVERRIDE, CONTENT-TYPE
> Access-Control-Max-Age: 1728000
> Cache-Control: must-revalidate, post-check=0, pre-check=0, private
> Date: Fri, 27 Mar 2015 16:51:27 GMT
> Content-Disposition: attachment; filename="Melanocytes-H3K4me3-q0.05.bigWig"
> Content-Transfer-Encoding: binary
> Pragma: public
> Last-Modified: Fri, 27 Mar 2015 15:24:47 GMT
> ETag: "24333da9b41f9d70af6319a111557f05"
> X-Content-Type-Options: nosniff
> X-Frame-Options: SAMEORIGIN
>
> real 0m21.631s

Note the 'Accept-Ranges: bytes' option this Apache server is configured for:

> $ time curl -I http://genome.ucsc.edu/goldenPath/help/examples/bigWigExample.bw
> HTTP/1.1 200 OK
> Date: Fri, 27 Mar 2015 16:53:10 GMT
> Server: Apache/2.2.15 (CentOS)
> Last-Modified: Mon, 14 Jun 2010 23:42:43 GMT
> ETag: "100391-4dd0f29-489060b8642c0"
> Accept-Ranges: bytes
> Content-Length: 81596201
> Connection: close
> Content-Type: text/plain; charset=UTF-8
>
> real 0m0.087s

--Hiram

On 3/27/15 8:33 AM, dan wrote:
> Hello,
>
> I am trying to upload BigWig files to my sessions on the browser.
> My files are stored on copy (www.copy.com <http://www.copy.com/>) which I use to generate public links. I then use the command below to upload the file:
>
> track type=bigWig name="Melanocytes-" bigDataUrl=https://copy.com/4yJ9p3LQrNjB visibility=full smoothingWindow=4 windowingFunction=mean maxHeightPixels=100:50:8
>
> I was able to do so up until about 10 days ago and then I started receiving messages that the link could not be open. I am trying to figure out where is the problem.
>
> The link below is to one of my files. As you can see, the link is active and will direct you to the file. However, it doesn’t work on the browser
>
> https://copy.com/k6euKF0mYGWAlFY7 <https://copy.com/k6euKF0mYGWAlFY7>

Brian Lee

unread,
Mar 27, 2015, 1:46:33 PM3/27/15
to Hiram Clawson, dan, gen...@soe.ucsc.edu

Dear Dan,

Thank you for using the UCSC Genome Browser and your question about your custom tracks that are not loading.

We have heard reports from other users also experiencing this issue with copy.com/files that used to work on the browser. Other free services like dropBox that once worked on the browser changed their configurations to disallow the byte range requests needed for the data to be accessed by the browser, it appears copy.com may have likewise changed some configurations recently. While byte range requests are very efficient for internet access, as it allows the browser to read arbitrary bits of the file without having to read the entire file, some of these free providers find all these requests expensive for their servers to honor. However when big files might represent several gigabytes of data, doing the byte range request which only fetches the tiny bits of the huge files to be displayed is much less taxing in the big picture, so some system administrators are willing to enable this request once they understand the bigger picture. You might want to contact copy.com and ask them about changes they may have made.

It is definitely worth investigating with copy.com why these changes are happening, but is likely similar to DropBox that they have discontinued some assepct of support for files they are storing for free. We've seen some people have success with an Amazon Cloud account on S3 storage, or often if you are a paying client of these services they will work to resolve your needs.

Another option is if you do not need to share these files remotely is to use our now available Virtual Machine Genome Browser in a Box, where you install a VM of the UCSC Browser and share files locally on your laptop, skipping the need to run anything over the internet. GBiB is free for non-commercial use, read more here: https://groups.google.com/a/soe.ucsc.edu/d/msg/genome/xhucY1YVv88/K2gyAwEgEmcJ

Thank you again for your inquiry and using the UCSC Genome Browser. If you have any further questions, please reply to gen...@soe.ucsc.edu. All messages sent to that address are archived on a publicly-accessible forum. If your question includes sensitive data, you may send it instead togenom...@soe.ucsc.edu.

All the best,

Brian Lee
UCSC Genome Bioinformatics Group


--


Reply all
Reply to author
Forward
0 new messages