Hi all,
I just built a docker container using the standard procedure for v482 and the executables were compatible with the OS of the container. I guess whatever changes were implemented fixed our problem. I haven't tried Max's "mirror" & "dev" trick yet, I merely used the generic "install" command.
However... another issue arose. When I compile the utilities for local use on our cluster, the builds got stuck trying to access curl functions:
/usr/local/GCC/11.3.0/bin/gcc -O -g -lcurl -std=c99 -Wall -Wformat -Wimplicit -Wreturn-type -Wuninitialized -std=c99 -Wall -Wformat -Wimplicit -Wreturn-type -Wuninitialized -std=c99 -Wall -Wformat -Wimplicit -Wreturn-type -Wuninitialized
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DMACHTYPE_x86_64 -DUSE_HIC -Wall -Wformat -Wimplicit -Wreturn-type -Wuninitialized -I../inc -I../../inc -I../../../inc -I../../../../inc -I../../../../../inc -I../../../../htsl
ib -I/usr/include/freetype2 -I/usr/include/libpng16 -DUSE_FREETYPE -I/usr/include/libpng16 -o edwQaWigSpotFromRa.o -c edwQaWigSpotFromRa.c
/usr/local/GCC/11.3.0/lib/gcc/x86_64-redhat-linux/11.3.0/../../../../x86_64-redhat-linux/bin/ld: ../../../lib/x86_64/jkhgap.a(cart.o): in function `curlPostUrl':
/lscratch/58449581/kent/src/hg/lib/cart.c:1480: undefined reference to `curl_easy_init'
/usr/local/GCC/11.3.0/lib/gcc/x86_64-redhat-linux/11.3.0/../../../../x86_64-redhat-linux/bin/ld: /lscratch/58449581/kent/src/hg/lib/cart.c:1487: undefined reference to `curl_easy_setopt'
/usr/local/GCC/11.3.0/lib/gcc/x86_64-redhat-linux/11.3.0/../../../../x86_64-redhat-linux/bin/ld: /lscratch/58449581/kent/src/hg/lib/cart.c:1488: undefined reference to `curl_easy_setopt'
/usr/local/GCC/11.3.0/lib/gcc/x86_64-redhat-linux/11.3.0/../../../../x86_64-redhat-linux/bin/ld: /lscratch/58449581/kent/src/hg/lib/cart.c:1489: undefined reference to `curl_easy_setopt'
/usr/local/GCC/11.3.0/lib/gcc/x86_64-redhat-linux/11.3.0/../../../../x86_64-redhat-linux/bin/ld: /lscratch/58449581/kent/src/hg/lib/cart.c:1490: undefined reference to `curl_easy_setopt'
/usr/local/GCC/11.3.0/lib/gcc/x86_64-redhat-linux/11.3.0/../../../../x86_64-redhat-linux/bin/ld: /lscratch/58449581/kent/src/hg/lib/cart.c:1491: undefined reference to `curl_easy_perform'
/usr/local/GCC/11.3.0/lib/gcc/x86_64-redhat-linux/11.3.0/../../../../x86_64-redhat-linux/bin/ld: /lscratch/58449581/kent/src/hg/lib/cart.c:1492: undefined reference to `curl_easy_cleanup'
collect2: error: ld returned 1 exit status
I found a workaround by including -lcurl in the following files:
kent/src/inc/
common.mk
kent/src/inc/userApp.mk
kent/src/hg/orthoMap/makefile
I had hoped setting CFLAGS=-lcurl would have worked, but CFLAGS is not universally propagated throughout the build stack. The last two makefiles used COPT and COPTS rather than CFLAGS for extra gcc flags. I don't know exactly why we specifically need -lcurl to enable the curl libraries.
Anyhow, we are now happily using v482. Thanks for all you guys do.
David
On 2/5/2024 11:22 AM, 'Maximilian Haeussler' via UCSC Genome Browser Mirror-Specific Support wrote:
> Hi David,
>
> OK, I've spent time on this, but still can't reproduce it:
>
> (1) Took an empty Rocky 8 VM on my laptop, installed and built the CGIs there, but can access hubs just fine. I'm not getting the error that you saw. I tried the Alfa hub, as this is also on the NIH campus:
>
https://ftp.ncbi.nlm.nih.gov/snp/population_frequency/TrackHub/latest/hub.txt <
https://ftp.ncbi.nlm.nih.gov/snp/population_frequency/TrackHub/latest/hub.txt>. It displays without errors...
>
> (2) So I added a command "dev" to our installation script. If you run this after the "mirror" command, it will install all requirements, clone the git repo and build all the CGIs from scratch. This should help you build the CGIs yourself in a fresh VM or Docker container and may also help us track down problems: you can now compare the (pretty simple) yum and make commands in that script with what you use. For example, there is no static linking with any dependencies if you look at the linker command (see below).
> Here is the version of the script with the "dev" command, it will soon land on github but I am sending you a link right now so you can test it:
>
https://hgwdev.gi.ucsc.edu/~max/davidHoover/browserSetup.sh <
https://hgwdev.gi.ucsc.edu/~max/davidHoover/browserSetup.sh>
>
> On a fresh VM or Docker container, you should be able to run these three commands to make a browser mirror and build everything:
>
> wget
https://hgwdev.gi.ucsc.edu/~max/davidHoover/browserSetup.sh <
https://hgwdev.gi.ucsc.edu/~max/davidHoover/browserSetup.sh>
> >
https://genome.ucsc.edu/goldenPath/help/mirror.html#docker-installation-instructions <
https://genome.ucsc.edu/goldenPath/help/mirror.html#docker-installation-instructions> <
https://genome.ucsc.edu/goldenPath/help/mirror.html#docker-installation-instructions <
https://genome.ucsc.edu/goldenPath/help/mirror.html#docker-installation-instructions>>
> > Can you elaborate what you mean with "accommodate our local mirror" ?
> >
> > I just tried an Ubuntu 22 a few days ago and ran the browserSetup.sh script in it and it just worked. I didn't use the docker commands, ran the browserSetup.sh -b install manually but I had a working mirror 2 minutes later.
> >
> > I wanted to try Rocky 8 next and unfortunately got distracted by grant reviews...
> >
> >
> >
> > On Mon, Jan 29, 2024 at 3:38 PM Hoover, David (NIH/CIT) [E] <
hoov...@hpc.nih.gov <mailto:
hoov...@hpc.nih.gov> <mailto:
hoov...@hpc.nih.gov <mailto:
hoov...@hpc.nih.gov>>> wrote:
> >
> > How up to date is the Docker procedure for the Genome Browser? How difficult would it be to create and use a docker image of the Genome Browser that does not have mysql running internally? We have a substantial mariadb server already full with UCSC data and constantly updated, as well as lots of users' custom tracks. The instructions at
https://genome.ucsc.edu/goldenPath/help/mirror.html#docker-installation-instructions <
https://genome.ucsc.edu/goldenPath/help/mirror.html#docker-installation-instructions> <
https://genome.ucsc.edu/goldenPath/help/mirror.html#docker-installation-instructions <
https://genome.ucsc.edu/goldenPath/help/mirror.html#docker-installation-instructions>> are very generic and would need to be elaborated to accommodate our local mirror.
> >
> > David
> >
> > On 1/22/2024 2:44 PM, Christopher Lee wrote:
> > > Hi Belinda and David,
> > >
> > > Unfortunately we are just completely stumped on what is going on here,
> > > and we do not have the resources to continue looking further. The only
> > > thing we can say is that we ourselves are upgrading to Rocky 9 soon
> > > and have verified at least minimal function with that operating
> > > system, so you can try upgrading and see if that fixes the problem.
> > >
> > > As far as theories, we suspect that somehow the MySQL/MariaDb
> > > installation could be misconfigured, even though that seemed to have
> > > been ruled out already.
> > >
> > > If you do find the cause of the problem and/or a fix, we would of
> > > course love to hear about it and accept a pull request or fix.
> > >
> > > Thanks,
> > > Christopher Lee
> > > UCSC Genomics Institute
> > >
> > > On Tue, Jan 16, 2024 at 12:07 PM Belinda M. Giardine
> > >>
https://www.bx.psu.edu/~giardine/tests/tmp/ucsc_cgis/ <
https://www.bx.psu.edu/~giardine/tests/tmp/ucsc_cgis/> <
https://www.bx.psu.edu/~giardine/tests/tmp/ucsc_cgis/ <
https://www.bx.psu.edu/~giardine/tests/tmp/ucsc_cgis/>>
> > >>
> > >> If you mean you want to be able to access the pages on our server the
> > >> public version is
> > >>
https://main.genome-browser.bx.psu.edu/ <
https://main.genome-browser.bx.psu.edu/> <
https://main.genome-browser.bx.psu.edu/ <
https://main.genome-browser.bx.psu.edu/>>
> > >>
> > >> Belinda
> > >>
> > >>
> > >> On Tue, 16 Jan 2024, Jairo Navarro Gonzalez wrote:
> > >>
> > >>> Hello,
> > >>>
> > >>> Thank you for using the UCSC Genome Browser and reporting your issues.
> > >>>
> > >>> Would it be possible to place the binaries in a web-accessible location?
> > >>> This way, our engineers can have a reproducible case to debug the issue.
> > >>>
> > >>> If you have any further questions, please reply to
gen...@soe.ucsc.edu <mailto:
gen...@soe.ucsc.edu> <mailto:
gen...@soe.ucsc.edu <mailto:
gen...@soe.ucsc.edu>>.
> > >>> All messages sent to that address are archived on a publicly accessible
> > >>> Google Groups forum.
> > >>> If your question includes sensitive data, you may send it instead to
> > >>>
genom...@soe.ucsc.edu <mailto:
genom...@soe.ucsc.edu> <mailto:
genom...@soe.ucsc.edu <mailto:
genom...@soe.ucsc.edu>>.
> > >>>
> > >>> Jairo Navarro
> > >>> UCSC Genome Browser
> > >>>
> > >>> On Tue, Jan 2, 2024 at 8:56?AM Belinda M. Giardine <
giar...@bx.psu.edu <mailto:
giar...@bx.psu.edu> <mailto:
giar...@bx.psu.edu <mailto:
giar...@bx.psu.edu>>>
> > >>> wrote:
> > >>>
> > >>>> We are building our own binaries. It is running on a VM.
> > >>>>
> > >>>> Belinda
> > >>>>
> > >>>>
> > >>>> On Fri, 22 Dec 2023, Maximilian Haeussler wrote:
> > >>>>
> > >>>>> David and Belinda,
> > >>>>> thank you for your patience with this, it's hard to debug if we don't
> > >>>> have
> > >>>>> access to a machine and can't reproduce the problem. We don't think it's
> > >>>>> Rocky 9 itself (trying with a VM).
> > >>>>> Are you both building your own binaries from the source code or are you
> > >>>>> using our pre-built binaries?
> > >>>>> thanks
> > >>>>> Max
> > >>>>>
> > >>>>> On Mon, Dec 18, 2023 at 8:31?PM Belinda M. Giardine <
giar...@bx.psu.edu <mailto:
giar...@bx.psu.edu> <mailto:
giar...@bx.psu.edu <mailto:
giar...@bx.psu.edu>>
> > >>>>>>
genome.ucsc.edu <
http://genome.ucsc.edu> <
http://genome.ucsc.edu <
http://genome.ucsc.edu>>
> > >>>>>>>>>>> Genome Browser Mirror-Specific Support <
genome...@soe.ucsc.edu <mailto:
genome...@soe.ucsc.edu> <mailto:
genome...@soe.ucsc.edu <mailto:
genome...@soe.ucsc.edu>>
> > >>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Now I am not so sure what is happening.? Here are some
> > >>>> observations:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * The timeout issue holds not just for custom tracks, for the
> > >>>> track
> > >>>>>>>>>>>> hubs as well
> > >>>>>>>>>>>> * On the UCSC site, I can load tracks and a hub hosted on our
> > >>>> local
> > >>>>>>>>>>>> dataserver (
hpc.nih.gov <
http://hpc.nih.gov> <
http://hpc.nih.gov <
http://hpc.nih.gov>>), and these display normally
> > >>>>>>>>>>>> Email:
hoov...@hpc.nih.gov <mailto:
hoov...@hpc.nih.gov> <mailto:
hoov...@hpc.nih.gov <mailto:
hoov...@hpc.nih.gov>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ---
> > >>>>>>>>>>>> You received this message because you are subscribed to the Google
> > >>>>>>>>>>>> Groups "UCSC Genome Browser Mirror-Specific Support" group.
> > >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
> > >>>>>> send
> > >>>>>>>>>>>> an email to
genome-mirro...@soe.ucsc.edu <mailto:
genome-mirror%2Bunsu...@soe.ucsc.edu> <mailto:
genome-mirror%2Bunsu...@soe.ucsc.edu <mailto:
genome-mirror%252Buns...@soe.ucsc.edu>>.
> > >>>>>>>>>>>> To view this discussion on the web visit
> > >>>>>>>>>>>>
> > >>>>>>
> > >>>>
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/88325cba-ecc8-489d-bfbd-f9ed7f97ddc7%40hpc.nih.gov <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/88325cba-ecc8-489d-bfbd-f9ed7f97ddc7%40hpc.nih.gov> <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/88325cba-ecc8-489d-bfbd-f9ed7f97ddc7%40hpc.nih.gov <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/88325cba-ecc8-489d-bfbd-f9ed7f97ddc7%40hpc.nih.gov>>
> > >>>>>> .
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> David Hoover, Ph.D.
> > >>>>>>>>> Computational Biologist
> > >>>>>>>>> High Performance Computing Services,
> > >>>>>>>>> Center for Information Technology,
> > >>>>>>>>> National Institutes of Health
> > >>>>>>>>> 12 South Dr., Rm 2N207
> > >>>>>>>>> Bethesda, MD 20892, USA
> > >>>>>>>>> TEL:
(+1) 301-435-2986
> > >>>>>>>>> Email:
hoov...@hpc.nih.gov <mailto:
hoov...@hpc.nih.gov> <mailto:
hoov...@hpc.nih.gov <mailto:
hoov...@hpc.nih.gov>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>>
> > >>>>>>>>> ---
> > >>>>>>>>> You received this message because you are subscribed to the Google
> > >>>>>> Groups
> > >>>>>>>>> "UCSC Genome Browser Mirror-Specific Support" group.
> > >>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
> > >>>> send
> > >>>>>> an
> > >>>>>>>>> email to
genome-mirro...@soe.ucsc.edu <mailto:
genome-mirror%2Bunsu...@soe.ucsc.edu> <mailto:
genome-mirror%2Bunsu...@soe.ucsc.edu <mailto:
genome-mirror%252Buns...@soe.ucsc.edu>>.
> > >>>>>>>>> To view this discussion on the web visit
> > >>>>>>>>>
> > >>>>>>
> > >>>>
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/afd2916f-4681-4426-21f4-ddb5405fecfc%40hpc.nih.gov <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/afd2916f-4681-4426-21f4-ddb5405fecfc%40hpc.nih.gov> <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/afd2916f-4681-4426-21f4-ddb5405fecfc%40hpc.nih.gov <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/afd2916f-4681-4426-21f4-ddb5405fecfc%40hpc.nih.gov>>
> > >>>>>> .
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> David Hoover, Ph.D.
> > >>>>>>> Computational Biologist
> > >>>>>>> High Performance Computing Services,
> > >>>>>>> Center for Information Technology,
> > >>>>>>> National Institutes of Health
> > >>>>>>> 12 South Dr., Rm 2N207
> > >>>>>>> Bethesda, MD 20892, USA
> > >>>>>>> TEL:
(+1) 301-435-2986
> > >>>>>>> Email:
hoov...@hpc.nih.gov <mailto:
hoov...@hpc.nih.gov> <mailto:
hoov...@hpc.nih.gov <mailto:
hoov...@hpc.nih.gov>>
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>>
> > >>>>>>> --- You received this message because you are subscribed to the Google
> > >>>>>> Groups
> > >>>>>>> "UCSC Genome Browser Mirror-Specific Support" group.
> > >>>>>>> To unsubscribe from this group and stop receiving emails from it, send
> > >>>>>> an
> > >>>>>>> email to
genome-mirro...@soe.ucsc.edu <mailto:
genome-mirror%2Bunsu...@soe.ucsc.edu> <mailto:
genome-mirror%2Bunsu...@soe.ucsc.edu <mailto:
genome-mirror%252Buns...@soe.ucsc.edu>>.
> > >>>>>>> To view this discussion on the web visit
> > >>>>>>>
> > >>>>>>
> > >>>>
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/5cd7a537-2f76-05bb-f779-ddc8807fae7b%40hpc.nih.gov <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/5cd7a537-2f76-05bb-f779-ddc8807fae7b%40hpc.nih.gov> <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/5cd7a537-2f76-05bb-f779-ddc8807fae7b%40hpc.nih.gov <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/5cd7a537-2f76-05bb-f779-ddc8807fae7b%40hpc.nih.gov>>
> > >>>>>> .
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>>
> > >>>>>> ---
> > >>>>>> You received this message because you are subscribed to the Google
> > >>>> Groups
> > >>>>>> "UCSC Genome Browser Mirror-Specific Support" group.
> > >>>>>> To unsubscribe from this group and stop receiving emails from it, send
> > >>>> an
> > >>>>>> email to
genome-mirro...@soe.ucsc.edu <mailto:
genome-mirror%2Bunsu...@soe.ucsc.edu> <mailto:
genome-mirror%2Bunsu...@soe.ucsc.edu <mailto:
genome-mirror%252Buns...@soe.ucsc.edu>>.
> > >>>>>> To view this discussion on the web visit
> > >>>>>>
> > >>>>
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/alpine.DEB.2.21.2312181326170.20267%40uniport.bx.psu.edu <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/alpine.DEB.2.21.2312181326170.20267%40uniport.bx.psu.edu> <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/alpine.DEB.2.21.2312181326170.20267%40uniport.bx.psu.edu <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/alpine.DEB.2.21.2312181326170.20267%40uniport.bx.psu.edu>>
> > >>>>>> .
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>> --
> > >>>>
> > >>>> ---
> > >>>> You received this message because you are subscribed to the Google Groups
> > >>>> "UCSC Genome Browser Mirror-Specific Support" group.
> > >>>> To unsubscribe from this group and stop receiving emails from it, send an
> > >>>> email to
genome-mirro...@soe.ucsc.edu <mailto:
genome-mirror%2Bunsu...@soe.ucsc.edu> <mailto:
genome-mirror%2Bunsu...@soe.ucsc.edu <mailto:
genome-mirror%252Buns...@soe.ucsc.edu>>.
> > >>>> To view this discussion on the web visit
> > >>>>
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/alpine.DEB.2.21.2401021122170.4445%40uniport.bx.psu.edu <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/alpine.DEB.2.21.2401021122170.4445%40uniport.bx.psu.edu> <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/alpine.DEB.2.21.2401021122170.4445%40uniport.bx.psu.edu <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/alpine.DEB.2.21.2401021122170.4445%40uniport.bx.psu.edu>>
> > >>>> .
> > >>>>
> > >>>
> >
> > --
> > David Hoover, Ph.D.
> > Computational Biologist
> > High Performance Computing Services,
> > Center for Information Technology,
> > National Institutes of Health
> > 12 South Dr., Rm 2N207
> > Bethesda, MD 20892, USA
> > TEL:
(+1) 301-435-2986
> > Email:
hoov...@hpc.nih.gov <mailto:
hoov...@hpc.nih.gov> <mailto:
hoov...@hpc.nih.gov <mailto:
hoov...@hpc.nih.gov>>
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Mirror-Specific Support" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to
genome-mirro...@soe.ucsc.edu <mailto:
genome-mirror%2Bunsu...@soe.ucsc.edu> <mailto:
genome-mirro...@soe.ucsc.edu <mailto:
genome-mirror%2Bunsu...@soe.ucsc.edu>>.
> > To view this discussion on the web visit
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/CAHm%3D4fxAe8kdOXWy4QUihp1V0LdUuBBBPW4cOiRf0TzuSE4%3D_g%40mail.gmail.com <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/CAHm%3D4fxAe8kdOXWy4QUihp1V0LdUuBBBPW4cOiRf0TzuSE4%3D_g%40mail.gmail.com> <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/CAHm%3D4fxAe8kdOXWy4QUihp1V0LdUuBBBPW4cOiRf0TzuSE4%3D_g%40mail.gmail.com?utm_medium=email&utm_source=footer <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/CAHm%3D4fxAe8kdOXWy4QUihp1V0LdUuBBBPW4cOiRf0TzuSE4%3D_g%40mail.gmail.com?utm_medium=email&utm_source=footer>>.
>
> --
> David Hoover, Ph.D.
> Computational Biologist
> High Performance Computing Services,
> Center for Information Technology,
> National Institutes of Health
> 12 South Dr., Rm 2N207
> Bethesda, MD 20892, USA
> TEL:
(+1) 301-435-2986
> To view this discussion on the web visit
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/CAHm%3D4fy8z%2B%3DmSyDipxamf4P5CFP1YTO_kFyrBW5B-CO%3DHxSQ4A%40mail.gmail.com <
https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/CAHm%3D4fy8z%2B%3DmSyDipxamf4P5CFP1YTO_kFyrBW5B-CO%3DHxSQ4A%40mail.gmail.com?utm_medium=email&utm_source=footer>.
--
David Hoover, Ph.D.
Computational Biologist
CIT SCS HPCS
National Institutes of Health
12 South Dr., Rm 2N207
TEL:
(+1) 301-435-2986
Email:
david....@nih.gov