Help needed debugging BigWig querying error

4 views
Skip to first unread message

Andy Yates

unread,
Feb 4, 2026, 4:30:30 AM (8 days ago) Feb 4
to UCSC Genome Browser Public Support
Hi there

Over at EBI we're moving some of our object storage to a new system which involves moving a set of BigWig files. These two URLs point to the same BigWig file:
When I pass the first URL into bigWigInfo (version 492) I get the following response

version: 4
isCompressed: yes
isSwapped: 0
primaryDataSize: 1,344,201
primaryIndexSize: 145,480
zoomLevels: 7
chromCount: 4301
basesCovered: 12,245,021
mean: 2.819623
min: 1.000000
max: 6513.000000
std: 34.968281

When passing in the second I get this error


Rerunning with the verbose flag set to 4 gives this

./bigWigInfo -verbose=4 https://livingobjects.ebi.ac.uk/wbps-jbrowse-tracks/rnaseqer/WBPS19/ancylostoma_caninum_prjna72585/A_caninum_9.3.2.ec.cg.pg/SRR332/SRR332576.bw
### kent source version 492 ###
udcfileOpen(https://livingobjects.ebi.ac.uk/wbps-jbrowse-tracks/rnaseqer/WBPS19/ancylostoma_caninum_prjna72585/A_caninum_9.3.2.ec.cg.pg/SRR332/SRR332576.bw, /var/folders/z4/503ywxqj7k53xfs320l_vbv80000gp/T//udcCache)
UDC dir: /var/folders/z4/503ywxqj7k53xfs320l_vbv80000gp/T//udcCache/https/livingobjects.ebi.ac.uk/wbps-jbrowse-tracks/rnaseqer/WBPS19/ancylostoma_caninum_prjna72585/A_caninum_9.3.2.ec.cg.pg/SRR332/SRR332576.bw
checking http remote info on https://livingobjects.ebi.ac.uk/wbps-jbrowse-tracks/rnaseqer/WBPS19/ancylostoma_caninum_prjna72585/A_caninum_9.3.2.ec.cg.pg/SRR332/SRR332576.bw
HTTP HEAD for https://livingobjects.ebi.ac.uk/wbps-jbrowse-tracks/rnaseqer/WBPS19/ancylostoma_caninum_prjna72585/A_caninum_9.3.2.ec.cg.pg/SRR332/SRR332576.bw
Couldn't open https://livingobjects.ebi.ac.uk/wbps-jbrowse-tracks/rnaseqer/WBPS19/ancylostoma_caninum_prjna72585/A_caninum_9.3.2.ec.cg.pg/SRR332/SRR332576.bw

When passing the URLs into curl and requesting HEAD (see after this for details) curl operates as expected. The new URL has one extra header "content-security-policy: block-all-mixed-content" otherwise nothing seems strange.

Any help is appreciated as I feel like this is finding its way into the net.c head functions but not succeeding. 

Thanks

Andy

% curl -v --head 'https://uk1s3.embassy.ebi.ac.uk/wbps-jbrowse-tracks/rnaseqer/WBPS19/ancylostoma_caninum_prjna72585/A_caninum_9.3.2.ec.cg.pg/SRR332/SRR332576.bw'
* Host uk1s3.embassy.ebi.ac.uk:443 was resolved.
* IPv6: (none)
* IPv4: 193.62.54.231
*   Trying 193.62.54.231:443...
* Connected to uk1s3.embassy.ebi.ac.uk (193.62.54.231) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=GB; ST=Cambridgeshire; O=EMBL-European Bioinformatics Institute; CN=*.embassy.ebi.ac.uk
*  start date: Mar  6 00:00:00 2025 GMT
*  expire date: Mar  6 23:59:59 2026 GMT
*  subjectAltName: host "uk1s3.embassy.ebi.ac.uk" matched cert's "*.embassy.ebi.ac.uk"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Organization Validation Secure Server CA
*  SSL certificate verify ok.
* using HTTP/1.x
> HEAD /wbps-jbrowse-tracks/rnaseqer/WBPS19/ancylostoma_caninum_prjna72585/A_caninum_9.3.2.ec.cg.pg/SRR332/SRR332576.bw HTTP/1.1
> Host: uk1s3.embassy.ebi.ac.uk
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 2585656
Content-Length: 2585656
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Last-Modified: Tue, 02 Jan 2024 11:58:12 GMT
Last-Modified: Tue, 02 Jan 2024 11:58:12 GMT
< x-rgw-object-type: Normal
x-rgw-object-type: Normal
< ETag: "8fd83fb0da7209c747329fba9ba79e32"
ETag: "8fd83fb0da7209c747329fba9ba79e32"
< x-amz-request-id: tx00000000000001635d0de-00698309da-7518e06c-default
x-amz-request-id: tx00000000000001635d0de-00698309da-7518e06c-default
< Content-Type: binary/octet-stream
Content-Type: binary/octet-stream
< Date: Wed, 04 Feb 2026 08:56:58 GMT
Date: Wed, 04 Feb 2026 08:56:58 GMT
< Connection: Keep-Alive
Connection: Keep-Alive
<

Now for the one that doesn't work

% curl -v --head 'https://livingobjects.ebi.ac.uk/wbps-jbrowse-tracks/rnaseqer/WBPS19/ancylostoma_caninum_prjna72585/A_caninum_9.3.2.ec.cg.pg/SRR332/SRR332576.bw'
* Host livingobjects.ebi.ac.uk:443 was resolved.
* IPv6: (none)
* IPv4: 193.62.193.150
*   Trying 193.62.193.150:443...
* Connected to livingobjects.ebi.ac.uk (193.62.193.150) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=GB; ST=Cambridgeshire; O=EMBL-European Bioinformatics Institute; CN=*.livingobjects.ebi.ac.uk
*  start date: Dec 14 00:00:00 2025 GMT
*  expire date: Dec 14 23:59:59 2026 GMT
*  subjectAltName: host "livingobjects.ebi.ac.uk" matched cert's "livingobjects.ebi.ac.uk"
*  issuer: C=GB; O=Sectigo Limited; CN=Sectigo Public Server Authentication CA OV R36
*  SSL certificate verify ok.
* using HTTP/1.x
> HEAD /wbps-jbrowse-tracks/rnaseqer/WBPS19/ancylostoma_caninum_prjna72585/A_caninum_9.3.2.ec.cg.pg/SRR332/SRR332576.bw HTTP/1.1
> Host: livingobjects.ebi.ac.uk
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< accept-ranges: bytes
accept-ranges: bytes
< content-length: 2585656
content-length: 2585656
< content-security-policy: block-all-mixed-content
content-security-policy: block-all-mixed-content
< content-type: binary/octet-stream
content-type: binary/octet-stream
< etag: "8fd83fb0da7209c747329fba9ba79e32"
etag: "8fd83fb0da7209c747329fba9ba79e32"
< last-modified: Thu, 29 Jan 2026 08:49:44 GMT
last-modified: Thu, 29 Jan 2026 08:49:44 GMT
< server: envoy
server: envoy
< vary: Origin
vary: Origin
< x-amz-request-id: 1890FF5423C263A5
x-amz-request-id: 1890FF5423C263A5
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< x-amz-meta-btime: 2024-01-02T11:58:12.235Z
x-amz-meta-btime: 2024-01-02T11:58:12.235Z
< x-amz-meta-mtime: 1704196692.235
x-amz-meta-mtime: 1704196692.235
< date: Wed, 04 Feb 2026 08:56:33 GMT
date: Wed, 04 Feb 2026 08:56:33 GMT
< x-envoy-upstream-service-time: 2837
x-envoy-upstream-service-time: 2837
<

Maximilian Haeussler

unread,
Feb 4, 2026, 5:06:26 AM (8 days ago) Feb 4
to Andy Yates, UCSC Genome Browser Public Support
Hi Andy, 

as far as I know, we use HTTP 1.0, which has never been a problem during all these years and has not come up recently either.

If I use HTTP 1.0 on your link with curl, I get an error:

$ curl --http1.0 -I https://livingobjects.ebi.ac.uk/wbps-jbrowse-tracks/rnaseqer/WBPS19/ancylostoma_caninum_prjna72585/A_caninum_9.3.2.ec.cg.pg/SRR332/SRR332576.bw
HTTP/1.1 426 Upgrade Required
date: Wed, 04 Feb 2026 09:55:47 GMT
server: envoy
connection: close

I changed HTTP/1.0 to HTTP/1.1 in our net.c and then this bigWigInfo command worked, but I don't know if there any other implications of this change. 

So I asked Claude, you're using a proxy, Envoy, which has HTTP 1.0 switched off by default. 
The developers agreed that supporting HTTP 1.0 makes little difference:

You can allow http/1 with accept_http_10 in your Envoy config:

Mark has a version somewhere of net.c that uses libcurl, but in a test a few years ago, that was slower when loading a .hal file than with the old net.c and so we never made the switch to libcurl (which supports http/1.1 and many more things, e.g. persistent connections and pooling and HTTP/2, etc)

best
Max

--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Public Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome+un...@soe.ucsc.edu.
To view this discussion visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome/8be48b50-a189-4e2a-a7db-98cce31a688en%40soe.ucsc.edu.

Galt Barber

unread,
Feb 4, 2026, 2:21:49 PM (7 days ago) Feb 4
to Maximilian Haeussler, Andy Yates, UCSC Genome Browser Public Support
I ran it though our own test utility that uses our network library and it reported this:

HTTP/1.1 426 Upgrade Required

This seems to be saying HTTP/1.1 is required.
You might be able to relax that.

One of the big improvements of http 1.1 was the client provides a host in the http request so that servers hosting multiple domains work correctly. We have always supplied the host in the request.

-Galt


Ar Céad 4 Feabh 2026 ag 02:06, scríobh 'Maximilian Haeussler' via UCSC Genome Browser Public Support <gen...@soe.ucsc.edu>:

Andy Yates

unread,
Feb 4, 2026, 4:30:23 PM (7 days ago) Feb 4
to Galt Barber, Maximilian Haeussler, UCSC Genome Browser Public Support
Thanks Galt. We'll chase up with our admins and see what we can do. Unfortunately it's at a layer that's not within my team's direct control so will see what we can do. For the moment you can assume it's a solved problem

Andy

Andrew Yates - Genomics Technology Infrastructure Team Leader, Joint-PI Ensembl Project
EMBL’s European Bioinformatics Institute (EMBL-EBI)
Wellcome Genome Campus
Hinxton, Cambridge
CB10 1SD, United Kingdom
Tel: +44-(0)1223-492538
Fax: +44-(0)1223-494468
https://www.ebi.ac.uk/
https://www.ensembl.org/

I sometimes send emails out of hours; I do not expect a response outside of your working hours.

Reply all
Reply to author
Forward
0 new messages