Compilng Kent source tree: curl -s "http://hgwdev-$USER.soe.ucsc.edu/cgi-bin/hgMirror?reset=1"

53 views
Skip to first unread message

Sven Klages

unread,
Jun 23, 2017, 11:29:28 AM6/23/17
to genome...@soe.ucsc.edu
Hi,

I am trying to set up a local mirror.

Building the kent works just fine (make all), but somewhere in the make output I do an error (but building the source tree succeeds):

​make[4]: Entering directory '/scratch/local2/build/UCSC_GenomeBrowser/v350/kent/src/hg/js/model'
gcc -O -g -o /scratch/local2/UCSC_GenomeBrowser/v350/bin/x86_64/ameme ameme.o fragFind.o   ../lib/x86_64/jkweb.a -L/scratch/local2/UCSC_GenomeBrowser/mysql/lib -lmysqlclient -lpthread -lm -lrt -ldl -lstdc++ -lrt ../htslib/libhts.a -pthread -L/lib -lssl -lcrypto -L/usr/lib -lpng15 -lz -lm -lm
gcc -O -g  -Wall -Wformat -Wimplicit -Wreturn-type -Wuninitialized -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DMACHTYPE_x86_64   -Wall -Wformat -Wimplicit -Wreturn-type -Wuninitialized -I../inc -I../../inc -I../../../inc -I../../../../inc -I../../../../../inc -I../../htslib -I/include -I/usr/include/libpng15  -o swissProtComments.o -c swissProtComments.c
rm -f /scratch/local2/UCSC_GenomeBrowser/v350/cgi-bin-USER/hgMirror
cp -p hgMirror /scratch/local2/UCSC_GenomeBrowser/v350/cgi-bin-USER/
# can only delete temp files as the apache user
cd react && echo react && make doInstall
gcc -O -g  -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DMACHTYPE_x86_64   -Wall -I../inc -I../../inc -I../../../inc -I../../../../inc -I../../../../../inc -I../../htslib -I/include -I/usr/include/libpng15  -o hgGtexTrackSettings.o -c hgGtexTrackSettings.c
react
curl -s "http://hgwdev-USER.soe.ucsc.edu/cgi-bin/hgMirror?reset=1"
make[4]: Entering directory '/scratch/local2/build/UCSC_GenomeBrowser/v350/kent/src/hg/js/react'
rsync -a BackboneExtend.js es5-sham.4.0.3.min.js es5-shim.4.0.3.min.js immutable.3.7.4.min.js jquery.bifrost.1.0.1.min.js lodash.3.10.0.compat.min.js react-with-addons-0.12.2.min.js /scratch/local2/UCSC_GenomeBrowser/v350/htdocs/js/USER/
makefile:16: recipe for target 'my' failed
make[2]: *** [my] Error 6
make[2]: Leaving directory '/scratch/local2/build/UCSC_GenomeBrowser/v350/kent/src/hg/hgMirror'
makefile:257: recipe for target 'hgMirror.bins' failed
make[1]: *** [hgMirror.bins] Error 2
make[1]: *** Waiting for unfinished jobs....
true ajax.js alleles.js autocomplete.js edwPersona.js hgCustom.js hgGateway.js hgGtexTrackSettings.js hgHubConnect.js hgTracks.js hgVarAnnogrator.js hui.js jsHelper.js utils.js ddcl.js subCfg.js encodeProject.js encodeDataSummary.js encodeDataMatrix.js encodeChipMatrix.js encodeMatrix.js d3.dendrograms.js cdwSummaryJs.js spectrum.min.js staticDoc.js dbDbTaxonomy.js
../utils/cpAndLinkToResourceFiles.pl  -exclude='makefile' \
        -versionFile=../inc/versionInfo.h -destDir=/scratch/local2/UCSC_GenomeBrowser/v350/htdocs/js/USER ajax.js alleles.js autocomplete.js edwPersona.js hgCustom.js hgGateway.js hgGtexTrackSettings.js hgHubConnect.js hgTracks.js hgVarAnnogrator.js hui.js jsHelper.js utils.js ddcl.js subCfg.js encodeProject.js encodeDataSummary.js encodeDataMatrix.js encodeChipMatrix.js encodeMatrix.js d3.dendrograms.js cdwSummaryJs.js spectrum.min.js d3pie.min.js jquery.js jquery.imgareaselect.js jquery-ui.js jquery.contextmenu.js jquery.cookie.js jquery.watermark.js jquery.watermarkinput.js jquery.ui.colorPicker.js jquery.floatheader.js jquery.rotateTableCellContent.js jquery.plugins.js jquery.validate.min.js jquery.history.js mousetrap.min.js jquery.tablednd.js ui.dropdownchecklist.js
mv a.out hgTrackUi


Is that an essential part? What happens there and what went wrong?
Is it safe to continue the setup (this way)?

best & thanks,
Sven

Christopher Lee

unread,
Jun 23, 2017, 2:02:38 PM6/23/17
to Sven Klages, genome-mirror
Hi Sven,

Thank you for your question about compiling the kent source. Is there
a particular reason why you are manually installing a mirror yourself?
We have a script now that should take care of all this setup for you
(if you are running some flavor of Debian/RedHat Linux), the "Genome
Browser in the Cloud" install script, which you can obtain from our
genome-store:
https://genome-store.ucsc.edu

The script is free for non-commercial use, and takes care of setting
up MySQL, an Apache webserver, downloading our CGI binaries,
downloading genome data, etc. For more information on using this
script, please see the Genome Browser in the Cloud documentation:
http://genome.ucsc.edu/goldenPath/help/gbic.html

That being said, the curl command is trying to run a script to clean
up some temporary files. As to why this was part of your build, we
will need to know how you got this error message, did you run make
from the top of the source tree? Is there some documentation you are
following in order to install this mirror?

Thank you again for your inquiry and using the UCSC Genome Browser. If
you have any further questions, please reply to genome...@soe.ucsc.edu.
All messages sent to that address are archived on a
publicly-accessible forum. If your question includes sensitive data,
you may send it instead to genom...@soe.ucsc.edu.

Christopher Lee
UCSC Genomics Institute
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "UCSC Genome Browser Mirror-Specific Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to genome-mirro...@soe.ucsc.edu.
> To post to this group, send email to genome...@soe.ucsc.edu.
> Visit this group at
> https://groups.google.com/a/soe.ucsc.edu/group/genome-mirror/.
> To view this discussion on the web visit
> https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/CAAX9WiUQACikaHBDf1n1641nXXYp%3DXecCoA%2BFCEX_jvNwyf-kg%40mail.gmail.com.
> For more options, visit https://groups.google.com/a/soe.ucsc.edu/d/optout.

Sven Klages

unread,
Jun 26, 2017, 4:03:11 PM6/26/17
to Christopher Lee, genome-mirror
Hi Christopher,

thanks for your quick response.

Yes, there some reasons to install genome browser "the hard way" ..
- we are not running a common flavor of linux, instead we are ruuning our own, home made linux since almost two decades :-)
- we have our own package management system, not rpm, not deb
- I have no root access to any of our servers, so I need to install genome browser in custom "project" with user privileges
- our sysadmins won't install genome browser systemwide, again, I need to install it somewhere I want to

We have already a few instances running (version from 2012!), the guy who compiled and installed these instances has left the institute and his documentation is poor. I need to setup one or more genome browser instances with our (custom) genomes, I don't need the public data(bases), no hg19 or mm10.

After googling and reading quite a while, I decided to start this way:

- compiling and installing apache httpd
- compiling and installing mysql
- setting some environment variables, MACHTYPE, BINDIR, CGI_BIN, SCRIPTS, DOCUMENTROOT among others
- compiling and installing samtabix, setting USE_SAMTABIX, SAMTABIXDIR
- getting kent source v350, wget http://hgdownload.soe.ucsc.edu/admin/jksrc.zip
- deflating archive, cd kent/src,
  make all
  works fine, except the error I reported.

The difficulties I encounter are:
* there is more than one place with similar, but not identical information about the manual installation process, mostly incomplete
* I do not find any resource describing completely the process of compiling and installing genome browser as a mirror without public data, but with custom genomes.
  Compiling bins/cgi-bins is one thing, the other things are apache setup (ok, there are some infos, should be manageable), and mysql setup. How is the database setup when I don't want to use/import public data? 

All these infos can be deduced from our ancient installation in combination with heavy googling and personal know-how .. that's the hard way :-)
I wish, you'd point me to some resources I have overseen in my attempts to find the "best" way to install genome browser ...

Let me shortly summarize what I try to accomplish:

* genome browser installation from source, mirror
* no public data, exclusively custom genomes and annotations
* custom linux system (64bit), no root access and no possibilty to install genome browser systemwide by our IT dep

Updating genome browser is hopefully done by recompiling kent source base ...


best,
Sven



--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Mirror-Specific Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome-mirror+unsubscribe@soe.ucsc.edu.

To post to this group, send email to genome...@soe.ucsc.edu.
Visit this group at https://groups.google.com/a/soe.ucsc.edu/group/genome-mirror/.

Matthew Speir

unread,
Jun 30, 2017, 1:21:25 PM6/30/17
to Sven Klages, Christopher Lee, genome-mirror
Hi Sven,

Thanks for providing more information about your specific Genome Browser setup and requirements.

You mentioned that you were planning to use your mirror to display a custom set of genomes, have you considered using an assembly hub: https://genome.ucsc.edu/goldenPath/help/hgTrackHubHelp.html#Assembly? As mentioned on that page, assembly hubs are used to display custom genomes and their annotations using our public site, https://genome.ucsc.edu. All you need to do is convert your genome and annotations to the supported formats, place them in a publicly-accessible web location, and then provide the URL to this location to the UCSC Genome Browser.

However, if you want to continue with your mirror installation, you should read our official mirror documentation:

http://genome.ucsc.edu/goldenPath/help/mirror.html
http://genome.ucsc.edu/goldenPath/help/mirrorManual.html

and here:
http://genome-source.cse.ucsc.edu/gitweb/?p=kent.git;a=tree;f=src/product

For the "curl -s" error you reported initially is in the hgMirror/makefile where it is trying to get the installed hgMirror script at the mirror to clean up temporary files. The hgMirror CGI is primarily used by the Genome Browser in a Box (GBiB) to download track data to speed up the Genome Browser. Since you're not intending to mirror any UCSC databases, it shouldn't be necessary.

I hope this is helpful. If you have any further questions, please reply to genome...@soe.ucsc.edu. All messages sent to that address are archived on a publicly-accessible Google Groups forum. If your question includes sensitive data, you may send it instead to genom...@soe.ucsc.edu.

Matthew Speir
UCSC Genome Bioinformatics Group
To unsubscribe from this group and stop receiving emails from it, send an email to genome-mirro...@soe.ucsc.edu.

To post to this group, send email to genome...@soe.ucsc.edu.
Visit this group at https://groups.google.com/a/soe.ucsc.edu/group/genome-mirror/.
Reply all
Reply to author
Forward
0 new messages