bug in kent/src/lib/https.c affecting big* binaries

[Pak, Theodore]

Hi UCSC Genome Browser team,

While using bigBedToBed and related binaries, I noticed that the program sometimes terminates early while fetching a lot of data from large bigBed files at HTTPS URLs. Fetching the same files from HTTP and locally was OK. (My test case was https://pakt01.u.hpc.mssm.edu/test/hg38.refGene.idx.bb)

This was utilizing the official binaries at http://hgdownload.soe.ucsc.edu/admin/exe/ on Mac x86_64; at present the Linux x86_64 binaries aren't compiled with OpenSSL.

I tracked this down in a debugger to a SIGPIPE being thrown when mustCloseFd() was being called in kent/src/lib/udc.c:243. This is because the pthread created for SSL encryption/decryption tries to keep writing to the socketpair() created in kent/src/lib/https.c and this throws a SIGPIPE signal. By default a SIGPIPE ends the process.

The easiest fix was to add one line to kent/src/lib/https.c which is illustrated in this commit (I was writing a new tool for bigBed searching when I discovered the bug, which utilizes much of the code in jkweb.a): https://github.com/powerpak/bigBedSearch/commit/9a56cb49dbac58e7345c8950dfb80f78ec4b6c91

This fix simply ignores SIGPIPE. This seems safe because you are already checking for error codes in netConnectHttpsThread() and terminating the thread when the pipe closes. 

There are more graceful and specific ways of doing this with SO_NOSIGPIPE on Mac and MSG_NOSIGNAL on Linux: http://stackoverflow.com/questions/108183/how-to-prevent-sigpipes-or-handle-them-properly That may be nicer if you use jkweb.a inside processes that should be quitting if other pipes break.

Sending this to you so hopefully you can patch a fix into the main repo.

Cheers,

Ted Pak

MD/PhD Student, Kasarskis Lab

Icahn School of Medicine at Mount Sinai

[Luvina Guruvadoo]

Hello Ted,

Thank you for bringing this to our attention. We hope to have this resolved soon and patched into a future release.

If you have any further questions, please reply to genome...@soe.ucsc.edu. All messages sent to that address are archived on a publicly-accessible forum. If your question includes sensitive data, you may send it instead to genom...@soe.ucsc.edu.

Regards,

Luvina

--

Luvina Guruvadoo
UCSC Genome Browser
http://genome.ucsc.edu

--