Bugzilla error "end of script output before headers index.cgi"

[Sarah Weinberger]

I got to the point where I was configuring Bugzilla through the GUI and ran into a SSL issue with email. I submitted a support issue on that. One of the steps for that was to redo the ./checksetup.pl command.

Stupid me complied, very annoying. The result is that I now receive

500 Internal Server Error

Troubleshooting this error, I discovered this log file:

/var/www/vhosts/jmr.com/logs/bugzilla.jmr.com/error_log

The contents of that are:

End of script output before headers: index.cgi

After I first installed Bugzilla, I ran into the same problem. I contacted my web host provider, Media Temple, and before they threw a fit saying "we do not support 3rd party software," I got hold of a nice engineer, who said the problem was permissions. He changed a whole lot of permissions and kept talking of sparingly making 755. Sadly, I do not know what else he might have done, but I did save a log for myself, so I know at least a few lines and I recall the root.

As such, I set the permissions as follows:

# cd /var/www/vhosts/jmr.com/logs/bugzilla.jmr.com
# chown jmr-admin:psacln -R *
# chmod 755 -R *

I verified the output and do see (and on all sub-directories too)

-rwxr-xr-x 1 jmr-admin psacln 793 Oct 16 16:55 admin.cgi
-rwxr-xr-x 1 jmr-admin psacln 29702 Oct 16 16:55 attachment.cgi
-rwxr-xr-x 1 jmr-admin psacln 43460 Oct 16 16:55 buglist.cgi
...

I have Selinux disabled:

[root@tkts-wtsc bugzilla.jmr.com]# sestatus
SELinux status: disabled
[root@tkts-wtsc bugzilla.jmr.com]#

I followed the installation guide (again too, and emptied out the damn lib folder before installing again):

https://www.howtoforge.com/tutorial/install-bugzilla-50-on-centos-7/

People said that the problem is with the bugzilla.conf file, which I edited.

[root@tkts-wtsc bugzilla.jmr.com]# vi /etc/httpd/conf.d/bugzilla.conf
#/etc/httpd/conf.d/bugzilla.conf
<VirtualHost *:80>
DocumentRoot /var/www/vhosts/jmr.com/bugzilla.jmr.com
</VirtualHost>
<Directory /var/www/vhosts/jmr.com/bugzilla.jmr.com>
AddHandler cgi-script .cgi
Options +Indexes +ExecCGI
DirectoryIndex index.cgi
AllowOverride Limit FileInfo Indexes Options AuthConfig
</Directory>

Yes, I have AuthConfig there. I even tried as several sites suggested `AllowOverride ALL` not that that helped, but back to proper setting.

I restarted httpd after each attempt

# systemctl restart httpd.service

For whatever it is worth, here is the error_log file:

# vi /var/log/httpd/error_log
[Wed Oct 23 10:50:28.235718 2019] [mpm_event:notice] [pid 9095:tid 140636605827200] AH00492: caught SIGWINCH, shutting down gracefully
[Wed Oct 23 10:50:33.453210 2019] [suexec:notice] [pid 10308:tid 140358398789760] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Oct 23 10:50:33.457272 2019] [ssl:warn] [pid 10308:tid 140358398789760] AH01909: RSA certificate configured for lists:443 does NOT include an ID which matches the server name
[Wed Oct 23 10:50:33.457984 2019] [ssl:warn] [pid 10308:tid 140358398789760] AH01909: RSA certificate configured for default-64_207_152_108:443 does NOT include an ID which matches the server name
[Wed Oct 23 10:50:33.459011 2019] [ssl:warn] [pid 10308:tid 140358398789760] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Wed Oct 23 10:50:33.459066 2019] [:notice] [pid 10308:tid 140358398789760] ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/) configured.
[Wed Oct 23 10:50:33.459077 2019] [:notice] [pid 10308:tid 140358398789760] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
[Wed Oct 23 10:50:33.459084 2019] [:notice] [pid 10308:tid 140358398789760] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Wed Oct 23 10:50:33.459093 2019] [:notice] [pid 10308:tid 140358398789760] ModSecurity: LUA compiled version="Lua 5.1"
[Wed Oct 23 10:50:33.459100 2019] [:notice] [pid 10308:tid 140358398789760] ModSecurity: LIBXML compiled version="2.9.1"
[Wed Oct 23 10:50:33.459106 2019] [:notice] [pid 10308:tid 140358398789760] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Wed Oct 23 10:50:33.551930 2019] [lbmethod_heartbeat:notice] [pid 10308:tid 140358398789760] AH02282: No slotmem from mod_heartmonitor
[Wed Oct 23 10:50:33.556959 2019] [ssl:warn] [pid 10308:tid 140358398789760] AH01909: RSA certificate configured for lists:443 does NOT include an ID which matches the server name
[Wed Oct 23 10:50:33.558234 2019] [ssl:warn] [pid 10308:tid 140358398789760] AH01909: RSA certificate configured for default-64_207_152_108:443 does NOT include an ID which matches the server name
[Wed Oct 23 10:50:33.559572 2019] [ssl:warn] [pid 10308:tid 140358398789760] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Wed Oct 23 10:50:33.561615 2019] [:notice] [pid 10308:tid 140358398789760] mod_bw : Memory Allocated 0 bytes (each conf takes 48 bytes)
[Wed Oct 23 10:50:33.561650 2019] [:notice] [pid 10308:tid 140358398789760] mod_bw : Version 0.92 - Initialized [0 Confs]

I do not see an .htaccess, not that that means there is not one, just I do not see it.

research (I did more than one):
1) https://www.ionos.com/community/server-cloud-infrastructure/apache/how-to-fix-http-error-code-500-internal-server-error/

Server:
* CentOS v7
* Plesk

FYI:
[root@tkts-wtsc bugzilla.jmr.com]# ./checksetup.pl
* This is Bugzilla 5.0 on perl 5.16.3
* Running on Linux 3.10.0-042stab140.1 #1 SMP Thu Aug 15 13:32:22 MSK 2019

Checking perl modules...
Checking for CGI.pm (v3.51) ok: found v3.63
Checking for Digest-SHA (any) ok: found v5.85
Checking for TimeDate (v2.23) ok: found v2.24
Checking for DateTime (v0.75) ok: found v1.04
Checking for DateTime-TimeZone (v1.64) ok: found v1.70
Checking for DBI (v1.614) ok: found v1.627
Checking for Template-Toolkit (v2.24) ok: found v2.24
Checking for Email-Sender (v1.300011) ok: found v1.300031
Checking for Email-MIME (v1.904) ok: found v1.926
Checking for URI (v1.55) ok: found v1.60
Checking for List-MoreUtils (v0.32) ok: found v0.33
Checking for Math-Random-ISAAC (v1.0.1) ok: found v1.004
Checking for File-Slurp (v9999.13) ok: found v9999.19
Checking for JSON-XS (v2.01) ok: found v3.01

Checking available perl DBD modules...
Checking for DBD-Pg (v2.7.0) ok: found v2.19.3
Checking for DBD-mysql (v4.001) ok: found v4.023
Checking for DBD-SQLite (v1.29) ok: found v1.39
Checking for DBD-Oracle (v1.19) not found

The following Perl modules are optional:
Checking for GD (v1.20) ok: found v2.49
Checking for Chart (v2.4.1) ok: found v2.4.10
Checking for Template-GD (any) ok: found v1.56
Checking for GDTextUtil (any) ok: found v0.86
Checking for GDGraph (any) ok: found v1.44
Checking for MIME-tools (v5.406) ok: found v5.505
Checking for libwww-perl (any) ok: found v6.05
Checking for XML-Twig (any) ok: found v3.44
Checking for PatchReader (v0.9.6) ok: found v0.9.6
Checking for perl-ldap (any) ok: found v0.56
Checking for Authen-SASL (any) ok: found v2.15
Checking for Net-SMTP-SSL (v1.01) ok: found v1.01
Checking for RadiusPerl (any) ok: found v0.26
Checking for SOAP-Lite (v0.712) ok: found v1.10
Checking for XMLRPC-Lite (v0.712) ok: found v0.717
Checking for JSON-RPC (any) ok: found v1.03
Checking for Test-Taint (v1.06) ok: found v1.06
Checking for HTML-Parser (v3.67) ok: found v3.71
Checking for HTML-Scrubber (any) ok: found v0.15
Checking for Encode (v2.21) ok: found v2.51
Checking for Encode-Detect (any) ok: found v1.01
Checking for Email-Reply (any) ok: found v1.203
Checking for HTML-FormatText-WithLinks (v0.13) ok: found v0.14
Checking for TheSchwartz (v1.07) ok: found v1.14
Checking for Daemon-Generic (any) ok: found v0.85
Checking for mod_perl (v1.999022) ok: found v2.000010
Checking for Apache-SizeLimit (v0.96) ok: found v0.97
Checking for File-MimeInfo (any) ok: found v0.21
Checking for IO-stringy (any) ok: found v2.110
Checking for Cache-Memcached (any) ok: found v1.30
Checking for File-Copy-Recursive (any) ok: found v0.38
Checking for mod_env (any) ok
Checking for mod_expires (any) ok
Checking for mod_headers (any) ok
Checking for mod_rewrite (any) ok
Checking for mod_version (any) ok
Reading ./localconfig...
Checking for DBD-mysql (v4.001) ok: found v4.023
Checking for MySQL (v5.0.15) ok: found v5.5.64-MariaDB

Removing existing compiled templates...
Precompiling templates...done.
Fixing file permissions...
checksetup.pl complete.
[root@tkts-wtsc bugzilla.jmr.com]#

My guess is that since ./checksetup.pl resets permissions is that I still have a permissions issue, which the setup procedure does not discuss, conveniently leaves this topic out.

Thoughts?

[Sarah Weinberger]

Update:

I ran across this post and discovered the testserver.pl file.

https://support-bugzilla.mozilla.narkive.com/vVX9wUTU/bugzilla-testserver-pl-test-failed-webserver-is-permitting-fetch-of-http-localhost-bugzilla

Result:

TEST-OK Webserver is running under group id in $webservergroup.
TEST-OK Got padlock picture.
Use of uninitialized value $response in pattern match (m//) at ./testserver.pl line 104.
Use of uninitialized value $response in pattern match (m//) at ./testserver.pl line 106.
TEST-FAILED Webserver is not executing CGI files.
TEST-FAILED Webserver is permitting fetch of http://bugzilla.jmr.com/localconfig.
This is a serious security problem.
Check your web server configuration.

The test said that CGI files do not execute and that the test is not permitted to fetch the localconfig.

That is interesting considering permissions for the file are 755:

-rwxr-xr-x 1 jmr-admin psacln 5669 Oct 23 12:20 localconfig

The associated configuration file looks fine and matches the install guide and what was there before. Maybe I missed a write permission somewhere, as I did not get a rundown on every change the web hosting company engineer made to get rid of this problem the first time and the CentOS 7 install document conveniently does not discuss permissions.

[root@tkts-wtsc bugzilla.jmr.com]# vi /etc/httpd/conf.d/bugzilla.conf

#/etc/httpd/conf.d/bugzilla.conf
<VirtualHost *:80>
DocumentRoot /var/www/vhosts/jmr.com/bugzilla.jmr.com
</VirtualHost>
<Directory /var/www/vhosts/jmr.com/bugzilla.jmr.com>
AddHandler cgi-script .cgi
Options +Indexes +ExecCGI
DirectoryIndex index.cgi
AllowOverride Limit FileInfo Indexes Options AuthConfig
</Directory>

I added a Directory entry for Bugzilla, though it was not there before, as follows (after the original ones already there):

[root@tkts-wtsc bugzilla.jmr.com]# vi /etc/httpd/conf/httpd.conf
<Directory "/var/www/vhosts/jmr.com/bugzilla.jmr.com">
AllowOverride All
#
Require all granted
</Directory>

I just tried removing what I added, as it this entry was not there before and restarted httpd.service to no avail, same problem.

[Sarah Weinberger]

I reran the test, after adding back in the <Directory></Directory> in /etc/httpd/conf/httpd.conf, and now get:

TEST-OK Webserver is running under group id in $webservergroup.

TEST-FAILED Fetch of images/padlock.png failed
Your web server could not fetch http://bugzilla.jmr.com/images/padlock.png.
Check your web server configuration and try again.

Permissions are still fine:

drwxr-xr-x 2 jmr-admin psacln 4096 Oct 16 18:24 images

-rwxr-xr-x 1 jmr-admin psacln 1204 Oct 16 16:55 csv.png
-rwxr-xr-x 1 jmr-admin psacln 4150 Oct 16 16:55 favicon.ico
-rwxr-xr-x 1 jmr-admin psacln 241 Oct 16 16:55 ical.png
-rwxr-xr-x 1 jmr-admin psacln 1066 Oct 16 16:55 padlock.png
-rwxr-xr-x 1 jmr-admin psacln 1230 Oct 16 16:55 rss.png
-rwxr-xr-x 1 jmr-admin psacln 764 Oct 16 16:55 rss_small.png
-rwxr-xr-x 1 jmr-admin psacln 1148 Oct 16 16:55 xml.png

I start off with the SSL/TLS issue and keep going further down the rabbit hole, quite agitating.

Is there anything more to configure on my CentOS 7 / Plesk environment than?

[root@tkts-wtsc bugzilla.jmr.com]# vi /etc/httpd/conf.d/bugzilla.conf

[Thorsten Schöning]

Guten Tag Sarah Weinberger,
am Mittwoch, 23. Oktober 2019 um 21:23 schrieben Sie:

> I got to the point where I was configuring Bugzilla through the GUI
> and ran into a SSL issue with email. I submitted a support issue on
> that. One of the steps for that was to redo the ./checksetup.pl command.

Re-running checksetup.pl is considered safe. It is used to install, it
is used during updates/upgrades, it is used in case templates have been
customized and if it simply has nothing to do, it won't do anything.

So if it does things and those break for some reason now and didn't
before, that's because you have changed your setup.

> End of script output before headers: index.cgi

That most likely means that the setup of your web server is broken and
simply doesn't execute CGI anymore, while it most likely did in the
past. The only reason why checksetup.pl could have broken that is by
changing the permissions of those files because of an invalid config.

The root dir of Bugzilla contains the file "localconfig" and that
expects things like the following under Linux:

> $webservergroup

Permissions are set according to that group, so check that and its
group membership. Other than that I can't think of how checksetup.pl
could have broken your setup, so if you changed the config of your web
server on your own, revert those modifications.

> After I first installed Bugzilla, I ran into the same problem. I
> contacted my web host provider, Media Temple, and before they threw
> a fit saying "we do not support 3rd party software," I got hold of a
> nice engineer, who said the problem was permissions. He changed a
> whole lot of permissions and kept talking of sparingly making 755.
> Sadly, I do not know what else he might have done, but I did save a
> log for myself, so I know at least a few lines and I recall the root.

This engineer was correct, but manually changing permissions per file
is the wrong thing to do. checksetup.pl needs to be configured
properly so that it knows under which account your web server is
runnign to set the correct permissions itself. While you can do that
manually, checksetup.pl will reset things whenever executed, leading
most likely to the problem you have now.

You need to get to know the account of your web server and such
stuff.

https://bugzilla.readthedocs.io/en/5.0/installing/linux.html#localconfig

> I followed the installation guide (again too, and emptied out the
> damn lib folder before installing again):

> https://www.howtoforge.com/tutorial/install-bugzilla-50-on-centos-7/

You should prefer using the official docs and only do otherwise as
necessary. Deploying Bugzilla using wget and tar instead of using GIT
is not a good recommendation anymore these days.

> <VirtualHost *:80>
[...]
> </VirtualHost>
> <Directory /var/www/vhosts/jmr.com/bugzilla.jmr.com>
[...]
> </Directory>

From my understanding that means that your "Directory"-config is NOT
for the VirtualHost you have configured before, but global to the web
server instead. That look correct to me, especially because
"DocumentRoot" is only valid for your "VirtualHost". You should
properly nest things and start with doing what the official docs
suggest:

https://bugzilla.readthedocs.io/en/5.0/installing/apache.html#apache-with-mod-cgi

> For whatever it is worth, here is the error_log file:

I don't see any errors in those logs.

> I do not see an .htaccess, not that that means there is not one, just I do not see it.

checksetup.pl creates .htaccess as needed. But why do you think you
need to care? I don't see it mentioned on your log snippet or else.

> My guess is that since ./checksetup.pl resets permissions is that I
> still have a permissions issue, which the setup procedure does not
> discuss, conveniently leaves this topic out.

The official docs do discuss those things:

https://bugzilla.readthedocs.io/en/5.0/installing/linux.html#localconfig

> TEST-FAILED Fetch of images/padlock.png failed
> Your web server could not fetch http://bugzilla.jmr.com/images/padlock.png.

That might be because of your strange/wrongly nested config regarding
"VirtualHost" and "Directory".

> Permissions are still fine:

> -rwxr-xr-x 1 jmr-admin psacln 1204 Oct 16 16:55 csv.png

x-permissions on those files are wrong most likely and I'm somewhat
sure that checksetup.pl would have reset those, if it would be working
properly. Which might be a hint that your "localconfig" is still wrong
and you are customizing things you shouldn't need to care about.

My setup before and after adding x-perms manually and after running
checksetup.pl again:

> 276275 4 -rw-r----- 1 root www-data 1066 Jul 8 2015 padlock.png
vs.
> 276275 4 -rwxr-x--x 1 root www-data 1066 Jul 8 2015 padlock.png
vs.
> 276275 4 -rw-r----- 1 root www-data 1066 Jul 8 2015 padlock.png

> I start off with the SSL/TLS issue and keep going further down the
> rabbit hole, quite agitating.

Sorry to say that, but because your setup was wrong before already.
Your manual tweaks are the wrong way to go, you need to properly
configure "localconfig" and I'm somewhat sure the web server config
itself as well.

Mit freundlichen Grüßen,

Thorsten Schöning

--
Thorsten Schöning E-Mail: Thorsten....@AM-SoFT.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/

Telefon...........05151- 9468- 55
Fax...............05151- 9468- 88
Mobil..............0178-8 9468- 04

AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow