External track-db fails to import with: "Couldn't open:..."
Romain Grégoire, Mr
Hi,
I'm providing track-db files to the USCS Genome Browser, but some of them fail to import with a "Couldn't open:..." error. I'm clueless as to why some of them fail, because they are all generated by the exact same API. The error only happens when I try to open more than 9 tracks (and it doesn't depend on which tracks are selected, the error only starts to arise when I have 10 tracks provided).
E.g. the first hub/genome/track-db files here contain 9 tracks, and can be opened fine:
http://dev3.epigenomesportal.ca/api/ucsc/hub/5420
http://dev3.epigenomesportal.ca/api/ucsc/genome/5420
http://dev3.epigenomesportal.ca/api/ucsc/track-db/5420
but the second hub/genome/track-db files here fail to import with a "Couldn't open" error (10 tracks):
http://dev3.epigenomesportal.ca/api/ucsc/hub/5421
http://dev3.epigenomesportal.ca/api/ucsc/genome/5421
http://dev3.epigenomesportal.ca/api/ucsc/track-db/5421
Here are links to open the tracks above in a UCSC Genome Browser
https://genome.ucsc.edu/cgi-bin/hgTracks?db=hg19&hubClear=http://dev3.epigenomesportal.ca/api/ucsc/hub/5421 (doesnt
work)
Thank you,
Romain
Cath Tyner
Thank you for contacting the UCSC Genome Browser support team, and thanks for including all of the relevant links for troubleshooting!
For the hub you are having problems with (5421), I can replicate the issue you are having; I also cannot connect to this hub.
I have copied and placed your 5421 hub files into my public directory. After doing so, and without changing the hub files or structure, I was able to connect without issues to your hub. This is making me wonder about a possible permissions difference between your 5420 and 5421 hub.
Your files on my server work fine for me:
https://genome.ucsc.edu/cgi-bin/hgTracks?db=hg19&hubClear=http://hgwdev.cse.ucsc.edu/~cath/hubs/MLQ20709/hub/5421
Are you seeing any differences in permissions that might explain this issue?
. . .
--
---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Public Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome+un...@soe.ucsc.edu.
To post to this group, send email to gen...@soe.ucsc.edu.
Visit this group at https://groups.google.com/a/soe.ucsc.edu/group/genome/.
To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome/YQBPR0101MB145848CB44B08488DED9AC18E80A0%40YQBPR0101MB1458.CANPRD01.PROD.OUTLOOK.COM.
For more options, visit https://groups.google.com/a/soe.ucsc.edu/d/optout.
Cath Tyner
A track hub opens multiple connections for each track, and it's possible that your open limit is set to 10.
If you can share your apache error_log, we may be able to assist further.
Romain Grégoire, Mr
Hi Cath,
Thank you so much for your time!
So, regarding your question, it's not a an Apache, it's a NodeJS server generating the track-db files, and there is no permission of any kind set up.
Also, I doubt that it is related to the number of connections because all the bigWig files in that track-db are hosted on a different server (https://datahub-3fl67ocb.udes.genap.ca).
Is there a way to troubleshoot and get a more verbose message than "Couldn't open"?
I've noticed that if I run the kentUtils command line utility `hubCheck` on that second link, it returns the exact same message: "Couldn't open ...". Is this tool being used in the backend? I've tried to get it to print more verbose details, but it doesn't print anything more with the -verbose=2 flag set.
I've also tried compiling `hubCheck` from source to try to troubleshoot more, but the current github version print a different error message:
Sent: Thursday, December 14, 2017 7:32:47 PM
To: Romain Grégoire, Mr
Cc: gen...@soe.ucsc.edu
Subject: Re: [genome] External track-db fails to import with: "Couldn't open:..."
Matthew Speir
Thanks for checking all of those things for possible issues.
Can you ensure that each of the files in your hub accepts byte-range requests? One of our engineers noticed that your file http://dev3.epigenomesportal.ca/api/ucsc/track-db/5421 does not allow byte-range requests.
I hope this is helpful. If you have any further questions, please reply to gen...@soe.ucsc.edu. All messages sent to that address are archived on a publicly-accessible Google Groups forum. If your question includes sensitive data, you may send it instead to genom...@soe.ucsc.edu.
Matthew Speir
UCSC Genome Bioinformatics Group
To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome/YQBPR0101MB1458C5D080D7C7CC8456FAC3E80B0%40YQBPR0101MB1458.CANPRD01.PROD.OUTLOOK.COM.
Romain Grégoire, Mr
Hello Matthew,
Thank you, your last hint did the job. Setting the headers 'Accept-Range: bytes' and 'Content-Length:
...' on the hub/genome/track-db files did solve the issue.
It should be noted that those headers were already set on the bigWig/bigBed files, and that the hub/genome/track-db files don't really accept byte-range requests.
Thanks all.
Romain
Sent: Tuesday, December 19, 2017 4:38:44 PM
To: Romain Grégoire, Mr; Cath Tyner