ParallelFetch > 0 gives timeout errors on rocky9

[Hoover, David (NIH/CIT) [E]]

Hi all,

We've been hosting a GB mirror on centos7 and seen no problems with custom track loading from remote webservers using bigDataUrl.

However, we've tried running it on rocky8 and rocky9, and in both cases increasing parallelFetch in hg.conf to anything above 0 causes timeout errors:

Error expected end of input reading http header on https://...
Can't get data socket for https://...

Reloading the page over and over ultimately causes the errors to go away, but this is very tedious.

If we run with parallelFetch=0, the response is generally much slower on rocky8 or rocky9 than when running with parallelFetch=20 on centos7. The errors are much rarer, but they still occur. Creating custom tracks always gives errors and is horrendously slow.

The same kinds of errors occur with track hubs pulled from external remote web servers.

We've ruled out any proxy issues, the host is directly connected to the internet.

We've ruled out any crypto-policies issues.

We have selinux disabled.

Is there any special thread management that is done?

What is your apache configuration? We are using prefork:

LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

I can't figure out how to implement the worker or event MPM anyway.

Can we use another webserver, nginx perhaps?

Pulling at threads,
David
--
David Hoover, Ph.D.
Computational Biologist
High Performance Computing Services,
Center for Information Technology,
National Institutes of Health
12 South Dr., Rm 2N207
Bethesda, MD 20892, USA
TEL: (+1) 301-435-2986
Email: hoov...@hpc.nih.gov

[Luis Nassar]

Hi, David.

We apologize for the delay in response here.

Have you found any solution or are you still having issues here with ParallelFetch?

Please include gen...@soe.ucsc.edu in any replies to ensure visibility by the team. All messages sent to that address are archived on our public forum. If your question includes sensitive information, you may send it instead to genom...@soe.ucsc.edu.

--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Mirror-Specific Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome-mirro...@soe.ucsc.edu.
To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/3d2fb235-3fa1-4817-9f90-a5fabf3d9d34%40hpc.nih.gov.

[Hoover, David (NIH/CIT) [E]]

Unfortunately no. The only situation that works is building the browser and running on centos7. Both rocky8 and rocky9 VMs give rise to random connection stalls. I even tried running the browser from within a docker container using the Dockerfile (from UCSC, running Ubuntu Fossa), but this also has connection stalls.

Centos7 is being deprecated in June, and after that we can't continue to support the browser on our site unless a solution is found.

David

On 4/12/2024 7:35 PM, Luis Nassar wrote:
> Hi, David.
>
> We apologize for the delay in response here.
>
> Have you found any solution or are you still having issues here with ParallelFetch?
>

> Please include gen...@soe.ucsc.edu <mailto:gen...@soe.ucsc.edu> in any replies to ensure visibility by the team. All messages sent to that address are archived on our public forum. If your question includes sensitive information, you may send it instead to genom...@soe.ucsc.edu <mailto:genom...@soe.ucsc.edu>.

> Email: hoov...@hpc.nih.gov <mailto:hoov...@hpc.nih.gov>

>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Mirror-Specific Support" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to genome-mirro...@soe.ucsc.edu <mailto:genome-mirror%2Bunsu...@soe.ucsc.edu>.
> To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/3d2fb235-3fa1-4817-9f90-a5fabf3d9d34%40hpc.nih.gov <https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/3d2fb235-3fa1-4817-9f90-a5fabf3d9d34%40hpc.nih.gov>.

[Mark Diekhans]

David, I believe you need to get an network administrator who
can do packet tracing and try to identify the root cause of the
problem. Trying various different configurations has not
narrowed down the cause. Trying other experiments without more
information is likely to just waste time.

I doubt that this is directly a result of the Centos upgrade,
rather some configuration change that came along with the
upgrade.

> To unsubscribe from this group and stop receiving emails from it, send an email to genome-mirro...@soe.ucsc.edu.
> To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/aebf4656-1eab-4086-9663-892a22d8800c%40hpc.nih.gov.

[Hoover, David (NIH/CIT) [E]]

Hi all,

I've managed to get the official UCSC docker solution working, after banging my head against the wall and figuring out the correct constellation of hg.conf parameters and reverse proxy directives in our parent webserver.

And yet we have another problem. The docker container runs Ubuntu, and our security scanners are complaining about vulnerabilities with openssl.

You all said that rocky8 and rocky9 work perfectly on your end. Would it be possible to change the OS of the docker container to either rocky8 or rocky9?

David