Error While Compiling

[Grady, Cameron (NIH/NHLBI) [F]]

Hi,

My name is Cameron Grady and I work with NHLBI in Bethesda MD and I am trying to set up a mirror site for lab. During the process of setting the kent source tree during the "make compile" step in instructions I realized that I did not have an apache folder. So I performed the following steps like it said to do instructions:

mkdir /usr/local/apache
ln -s /var/www/cgi-bin /usr/local/apache/cgi-bin
ln -s /var/www/html /usr/local/apache/htdocs
ln -s /var/www/cgi-bin /usr/local/apache/cgi-bin-$(LOGNAME)

When I typed in the command "make cgi" I got this error:

make[1]: Entering directory `/home/gradycr/Desktop/kent/src/hg/cartReset'

gcc -O -g  -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DMACHTYPE_x86_64 -DUSE_SSL   -Wall -I../inc -I../../inc -I../../../inc -I../../../../inc -I../../../../../inc -I../../htslib -I/include -I/usr/include/libpng12  -o cartReset.o -c cartReset.c
gcc cartReset.o ../../lib/x86_64/jkhgap.a ../../lib/x86_64/jkweb.a ../../htslib/libhts.a -pthread -L/lib -lssl -lcrypto -L/usr/lib/x86_64-linux-gnu -lpng12 -lm -L/usr/lib/x86_64-linux-gnu -lmysqlclient -lpthread -lz -lm -ldl -lstdc++ -lrt
mv a.out cartReset
chmod a+rx cartReset
rm -f /usr/local/apache/cgi-bin-gradycr/cartReset
mv cartReset /usr/local/apache/cgi-bin-gradycr/cartReset
mv: cannot move ‘cartReset’ to ‘/usr/local/apache/cgi-bin-gradycr/cartReset’: No such file or directory
make[1]: *** [my] Error 1
make[1]: Leaving directory `/home/gradycr/Desktop/kent/src/hg/cartReset'
make: *** [cartReset.bins] Error 2

I looked for answers on google groups (https://groups.google.com/a/soe.ucsc.edu/forum/#!msg/genome/vQ0_zYs33yE/qWUrroWaQVIJ) and found someone who had a similar problem. However when I tried all of the purposed solutions I still received an error. Particularly when running "make alpha" in src directory I received this error:

home/gradycr/Desktop/kent/

src/htslib/hts.c:142: undefined reference to `inflateInit2_'
/home/gradycr/Desktop/kent/src/htslib/hts.c:145: undefined reference to `inflate'
/home/gradycr/Desktop/kent/src/htslib/hts.c:148: undefined reference to `inflateEnd'
../../htslib/libhts.a(cram_io.o): In function `zlib_mem_deflate':
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:782: undefined reference to `deflateInit2_'
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:796: undefined reference to `deflate'
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:803: undefined reference to `deflate'
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:808: undefined reference to `deflateEnd'
../../htslib/libhts.a(cram_io.o): In function `itf8_decode_crc':
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:195: undefined reference to `crc32'
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:201: undefined reference to `crc32'
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:208: undefined reference to `crc32'
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:216: undefined reference to `crc32'
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:225: undefined reference to `crc32'
../../htslib/libhts.a(cram_io.o):/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:537: more undefined references to `crc32' follow
../../htslib/libhts.a(cram_io.o): In function `zlib_mem_inflate':
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:718: undefined reference to `inflateInit2_'
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:731: undefined reference to `inflate'
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:751: undefined reference to `inflateEnd'
../../htslib/libhts.a(cram_io.o): In function `cram_read_block':
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:946: undefined reference to `crc32'
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:948: undefined reference to `crc32'
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:980: undefined reference to `crc32'
../../htslib/libhts.a(cram_io.o): In function `cram_write_block':
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:1049: undefined reference to `crc32'
/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:1052: undefined reference to `crc32'
../../htslib/libhts.a(cram_io.o):/home/gradycr/Desktop/kent/src/htslib/cram/cram_io.c:1054: more undefined references to `crc32' follow
../../htslib/libhts.a(zfio.o): In function `zfgets':
/home/gradycr/Desktop/kent/src/htslib/cram/zfio.c:69: undefined reference to `gzgets'
../../htslib/libhts.a(zfio.o): In function `zfputs':
/home/gradycr/Desktop/kent/src/htslib/cram/zfio.c:80: undefined reference to `gzputs'
../../htslib/libhts.a(zfio.o): In function `zfpeek':
/home/gradycr/Desktop/kent/src/htslib/cram/zfio.c:95: undefined reference to `gzgetc'
/home/gradycr/Desktop/kent/src/htslib/cram/zfio.c:97: undefined reference to `gzungetc'
../../htslib/libhts.a(zfio.o): In function `zfeof':
/home/gradycr/Desktop/kent/src/htslib/cram/zfio.c:105: undefined reference to `gzeof'
../../htslib/libhts.a(zfio.o): In function `zfopen':
/home/gradycr/Desktop/kent/src/htslib/cram/zfio.c:165: undefined reference to `gzopen64'
/home/gradycr/Desktop/kent/src/htslib/cram/zfio.c:169: undefined reference to `gzopen64'
../../htslib/libhts.a(zfio.o): In function `zfclose':
/home/gradycr/Desktop/kent/src/htslib/cram/zfio.c:180: undefined reference to `gzclose'
../../htslib/libhts.a(bgzf.o): In function `bgzf_write_init':
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:166: undefined reference to `deflateInit2_'
../../htslib/libhts.a(bgzf.o): In function `inflate_gzip_block':
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:335: undefined reference to `inflate'
../../htslib/libhts.a(bgzf.o): In function `bgzf_open':
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:174: undefined reference to `compressBound'
../../htslib/libhts.a(bgzf.o): In function `bgzf_dopen':
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:196: undefined reference to `compressBound'
../../htslib/libhts.a(bgzf.o): In function `bgzf_hopen':
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:218: undefined reference to `compressBound'
../../htslib/libhts.a(bgzf.o): In function `bgzf_compress':
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:244: undefined reference to `deflateInit2_'
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:245: undefined reference to `deflate'
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:246: undefined reference to `deflateEnd'
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:252: undefined reference to `crc32'
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:252: undefined reference to `crc32'
../../htslib/libhts.a(bgzf.o): In function `bgzf_gzip_compress':
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:267: undefined reference to `deflate'
../../htslib/libhts.a(bgzf.o): In function `bgzf_read_block':
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:505: undefined reference to `inflateInit2_'
../../htslib/libhts.a(bgzf.o): In function `inflate_block':
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:302: undefined reference to `inflateInit2_'
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:306: undefined reference to `inflate'
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:307: undefined reference to `inflateEnd'
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:311: undefined reference to `inflateEnd'
../../htslib/libhts.a(bgzf.o): In function `bgzf_close':
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:816: undefined reference to `inflateEnd'
/home/gradycr/Desktop/kent/src/htslib/bgzf.c:817: undefined reference to `deflateEnd'
collect2: error: ld returned 1 exit status
make[3]: *** [/home/gradycr/bin/x86_64/bptForTwoBit] Error 1
make[3]: Leaving directory `/home/gradycr/Desktop/kent/src/index/bptForTwoBit'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/home/gradycr/Desktop/kent/src/index'
make[1]: *** [utils-alpha] Error 2
make[1]: Leaving directory `/home/gradycr/Desktop/kent/src'
make: *** [alpha] Error 2

 I have run out of things to try to trouble shoot. Anything hints or clues you guys could give would be helpful. Here is some pertinent information about my Linux system:

$ uname -a 
Linux ubuntu 3.13.0-101-generic #148-Ubuntu SMP Thu Oct 20 22:08:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ mysql_config --version
5.5.53
$ mysql_config --libs
-L/usr/lib/x86_64-linux-gnu -lmysqlclient -lpthread -lz -lm -ldl

Best,

Cameron Grady

[Hiram Clawson]

Good Afternoon Cameron:

The good news is that your build is almost finished. You do want to work
in your home directory and you do *not* want to build this as root.

The work-around you need is that permissions need to be granted for writing
into /var/www/cgi-bin/ and /var/www/html/ from your user account. This
will require assistance from the 'root' account.

You can either chmod the directories to add write permission:
# chmod 777 /var/www/cgi-bin/ /var/www/html/

while you do your 'make' from your user account, then switch them back:
# chmod 755 /var/www/cgi-bin/ /var/www/html/

to maintain the security of those directories.

Or, you could add the 'apache' group to your user account to give your
user account permission to write in group-owned 'apache' directories.
You may need to add write permission for the group:

# chmod 775 /var/www/cgi-bin/ /var/www/html/

This all depends upon your system security preferences. If you are not
allowed permissions to write into these directories, you would need to build
with a DESTDIR of some location where you can write to, then ask system
administrators to copy from your DESTDIR build directory into these apache
directories.

Please do not hesitate to get back to us with any difficulties.

--Hiram

[Grady, Cameron (NIH/NHLBI) [F]]

Thanks for the response to my earlier inquiry. It worked perfectly. I am having another issue this time during the installation phase. When I check localhost on my browser the webpage appears but towards the top where the menu bar should be is there is an error: [an error occurred while processing this directive]. I checked the apache error log and I continuously getting this error: unable to include "cgi-bin/hgMenubar" in parsed file /var/www/html/index.html, subrequest returned 404. I have added the following lines to my apache2.conf file:
XBitHack on

<Directory /var/www/html>
Options +Includes
SSILegacyExprParser on
</Directory>

ScriptAlias /cgi-bin/ "/var/www/html/cgi-bin"

<Directory "/var/www/html/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
AddHandler cgi-script cgi pl
</Directory>
I believe the problem has something to do with this apache conf file. I have run out of things to try to trouble shoot. Anything hints or clues you guys could give would be helpful.

Thanks again,

Cameron Grady
________________________________________
From: Hiram Clawson [hi...@soe.ucsc.edu]
Sent: Tuesday, November 15, 2016 3:30 PM
To: Grady, Cameron (NIH/NHLBI) [F]; genome...@soe.ucsc.edu
Subject: Re: [genome-mirror] Error While Compiling

[Hiram Clawson]

Good Morning Cameron:

Does your index.html have the 'x' execute bits on ? This is used
by the XBitHack option to trigger Apache to do the include statements.

$ ls -og index.html
-rwxrwxr-x 1 6760 Oct 19 13:34 index.html

(other .html files need this too)

And, do you have a cgi-bin/hgMenubar binary ?

Our Apache options for the html directory are:

Options Indexes FollowSymLinks MultiViews Includes
Header set Access-Control-Allow-Origin "*"
Header set Access-Control-Allow-Headers: Range
AllowOverride None

To prevent browsing of the 'trash' directory:

<Directory "/var/www/html/trash">
Options MultiViews
AllowOverride None

Order allow,deny
Allow from all

</Directory>

--Hiram

[Grady, Cameron (NIH/NHLBI) [F]]

It is still giving me the same error. The x execute bits appear to be on and yes I have all the cgi-bin binaries downloaded. I have uploaded a few screen shots to show how I configured the page. I have an image of my apache configuration file, the .htaccess file, a picture of the website when I check localhost, and the folder/path with all the html files and binaries in the documentroot path for apache on my linux machine.

Cameron

________________________________________
From: Hiram Clawson [hi...@soe.ucsc.edu]

Sent: Tuesday, November 29, 2016 1:48 PM

[Hiram Clawson]

You have a good puzzle here Grady.

When you access one of the CGI binaries, for example:
http://localhost/cgi-bin/hgTracks

Does the browser display anything ?

Do you see any unusual errors in your Apache error_log ?

On our command line, if you go to the cgi-bin directory and run
hgGateway directly, does it run and output html code, or emit an error ?

./hgGateway

On 11/30/16 1:47 PM, Grady, Cameron (NIH/NHLBI) [F] wrote:
> Hi,
>

> I was wondering if you were able to take a look at the screenshots I sent you. I was not able to fix the error with your previous suggestions. Thanks again for your help.
>
> Cameron
> ________________________________________
> From: Grady, Cameron (NIH/NHLBI) [F]
> Sent: Tuesday, November 29, 2016 2:35 PM
> To: Hiram Clawson; genome...@soe.ucsc.edu
> Subject: RE: [genome-mirror] Error While Compiling

[Grady, Cameron (NIH/NHLBI) [F]]

I do get an error when I try to access the CGI binaries. I have uploaded pictures of the errors I get on the command line when I type ./hgGateway and when I click on the link http://localhost/cgi-bin/hgTracks. I believe it has something to do with mysql configuration but I followed the mysql.readme setup exactly.

________________________________________
From: Hiram Clawson [hi...@soe.ucsc.edu]

Sent: Wednesday, November 30, 2016 5:07 PM

[Grady, Cameron (NIH/NHLBI) [F]]

Also, here is a screenshot of my apache error log.

________________________________________
From: Grady, Cameron (NIH/NHLBI) [F]

Sent: Wednesday, November 30, 2016 5:26 PM

[Hiram Clawson]

Good Afternoon Grady:

Your apache error_log is highly informative. It indicates various
configuration difficulties.

For your MySQL permissions, I refer you to the mysql setup discussion

http://genome-source.cse.ucsc.edu/gitweb/?p=kent.git;a=blob;f=src/product/README.mysql.setup

There are two access mechanisms involved with the MySQL:
1. using 'kent' command line operations via $HOME/.hg.conf
2. The CGI binaries when operated via Apache via: /var/www/html/cgi-bin/hg.conf

Normally you want your command line operations to have read/write permissions so
you can load and manipulate the databases. The CGI binaries should have read-only
permissions to all databases, and read/write permissions to hgcentral.

To test the CGI binary/apache operation permissions, temporarily change
your $HOME/.hg.conf file to be a single line:

include /var/www/html/cgi-bin/hg.conf

Thus, when you run command line programs, they will have the read-only permissions
that the CGI binaries would normally see. You should be able to run commands
such as:
$ hgsql -e 'show tables;' hgcentral

to see the tables in your hgcentral database. Your Apache error_log currently indicates
that CGI binaries have no access to the hgcentral database.

When you are done testing that read-only access, convert your $HOME/.hg.conf back to
the read/write access. Please note, you can run CGI binaries from the command line,
they will use whatever is in your $HOME/.hg.conf in that case, now what they would
be using /var/www/html/cgi-bin/hg.conf when they run via Apache.

Another error indicated in your error_log seems to indicate that code we have
in index.html is legacy code and newer versions of Apache need a special option
to allow parsing of that code. Please try turning on: SSILegacyExprParser on
in your Apache configuration.
http://httpd.apache.org/docs/2.4/mod/mod_include.html#ssilegacyexprparser

Another puzzle I note with your configuration is that you have cgi-bin under html
I don't know if that works correctly, I am not familiar with the nesting rules
of <Directory> statements in Apache. Usually I find the directory setup:
/var/www/html/
/var/www/cgi-bin/
/var/www/trash/

With a symlink from /var/www/html/trash to ../trash

Perhaps your setup will work with the SSILegacyExprParser option and the MySQL
permissions.

You may find it useful to review the steps in the browser installer script:

http://genome-source.cse.ucsc.edu/gitweb/?p=kent.git;a=blob;f=src/product/installer/browserSetup.sh

to see if it is doing something extra from the steps you have performed.

Let me know how this works out.

--Hiram

[Grady, Cameron (NIH/NHLBI) [F]]

Hiram,

Thanks for the advice. I feel that I am really close but I am still getting one error consistently. When I run the cgi binaries in terminal for example ./hgGateway I get an html file. However when I click on the link on the actual webpage it gives me the error "Forbidden: You don't have permission to access /cgi-bin/hgGateway on this server." When I check the apache log I see the error "AH01630: client denied by server configuration: /var/www/cgi-binhgGateway, referer: http://example.com/." So I did some research and that error is common for a syntax error between Apache 2.2 and Apache 2.4, which is the version I am running. It says that the syntax in the apache2.conf should change from:

Order allow,deny
Allow from all

to:
Require all granted
I did that for all instances but I am still getting the same error. I have also attached a screenshot of my apache2.conf file and apache error log.

Thanks again,

Cameron
________________________________________
From: Hiram Clawson [hi...@soe.ucsc.edu]

Sent: Thursday, December 01, 2016 3:26 PM

To: Grady, Cameron (NIH/NHLBI) [F]; genome...@soe.ucsc.edu
Subject: Re: [genome-mirror] Error While Compiling

Good Afternoon Grady:

[Hiram Clawson]

Good Evening Grady:

Try adding to your ScriptAlias definition:

ScriptAlias /cgi-bin/ "/var/www/html/cgi-bin"

the trailing / after cgi-bin/:

ScriptAlias /cgi-bin/ "/var/www/html/cgi-bin/"

Your error_log appears to indicate it is missing.
It looks like there is a special discussion of why and when
this is necessary.
https://httpd.apache.org/docs/current/mod/mod_alias.html
"Note that if you include a trailing / on the URL-path then
the server will require a trailing / in order to expand the alias."

I'm curious, when you started this procedure, were you aware of
the install script in the source tree ? Are you familiar with
assembly hub setups that allow you to display a novel genome
assembly without a mirror ?

--Hiram

[Hiram Clawson]

How is it working out there Cameron ?