httpd remains in 'starting' state when starting via systemctl

[David Carson]

After following all directions for installing ReviewBoard 2.0 on CentOS 7, I am still seeing one error that bothers me.  When I try to restart httpd, it remains in 'starting' state and will never go to 'active (running)' state.  However, if I remove the following file, httpd starts and stops properly.

   /usr/lib/systemd/system/httpd.service.d/reviewboards-sites.conf

I only discovered this file because there was an error in /var/log/messages regarding the line "type=oneshot" (which should be "Type=oneshot").  After correcting the spelling, I no longer see the error, but I cannot start httpd either.

I have moved this file out of the way, and my ReviewBoard installation is now working, at least to the point of bringing up the interface and allowing me to login as admin.  But I'm wondering what the repercussions of removing the systemd file might be. 

Thanks,

David

[Stephen Gallagher]

On 09/23/2014 03:38 PM, David Carson wrote:
> After following all directions for installing ReviewBoard 2.0 on CentOS
> 7, I am still seeing one error that bothers me. When I try to restart

> httpd, it remains in *'starting'* state and will never go to *'active
> (running)'* state. However, if I remove the following file, httpd

> starts and stops properly.
>
> /usr/lib/systemd/system/httpd.service.d/reviewboards-sites.conf
>
> I only discovered this file because there was an error in
> /var/log/messages regarding the line "type=oneshot" (which should be

> "*T*ype=oneshot"). After correcting the spelling, I no longer see the

> error, but I cannot start httpd either.
>
> I have moved this file out of the way, and my ReviewBoard installation
> is now working, at least to the point of bringing up the interface and
> allowing me to login as admin. But I'm wondering what the repercussions
> of removing the systemd file might be.
>

I'll field this one, since this bit is unique to my packages. The
purpose of that oneshot target is to guarantee that the Review Board
automatically runs 'rb-site upgrade' for all sites when it is started up
(thus guaranteeing that if you perform an RPM upgrade, it will upgrade
Review Board the next time it is started.

I'm curious why it's hanging during the upgrade step, though. Can you
check the journal for what's happening when you see that?

The command 'journalctl -e -u httpd.service' should give you the output.
It's possible that there's a bug in the upgrade routine. Have you
checked for SELinux denials?

[David Carson]

Stephen,
This is an internal-only server, so I turned off SELinux before doing anything else (and rebooted, of course).

The journalctl command seems to indicate that the update finished:

-- Logs begin at Tue 2014-09-23 18:56:22 EDT, end at Tue 2014-09-23 19:50:32 EDT. --
:
:
Sep 23 19:50:13 pinehurst.e.com systemd[1]: Stopping The Apache HTTP Server...
Sep 23 19:50:14 pinehurst.e.com systemd[1]: Starting The Apache HTTP Server...
Sep 23 19:50:14 pinehurst.e.com systemd[1]: Started The Apache HTTP Server.
Sep 23 19:50:27 pinehurst.e.com systemd[1]: Stopping The Apache HTTP Server...
Sep 23 19:50:28 pinehurst.e.com systemd[1]: Starting The Apache HTTP Server...
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: Rebuilding directory structure
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: Updating database. This may take a while.
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: The log output below, including warnings and errors,
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: can be ignored unless upgrade fails.
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: ------------------ <begin log output> ------------------
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: Creating tables ...
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: Installing custom SQL ...
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: Installing indexes ...
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: Installed 0 object(s) from 0 fixture(s)
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: No evolution required.
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: ------------------- <end log output> -------------------
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: Resetting in-database caches.
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: Upgrade complete!

But the systemctl command itself is sitting there, not returning to the command prompt, and the status shows that it is still activating, but never activated:

(0)[root@pinehurst ~] systemctl status httpd.service
httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
Drop-In: /usr/lib/systemd/system/httpd.service.d
└─reviewboard-sites.conf
Active: activating (start) since Tue 2014-09-23 19:50:28 EDT; 11min ago
Process: 3223 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
Process: 3228 ExecStartPre=/usr/bin/rb-site upgrade --all-sites (code=exited, status=0/SUCCESS)
Main PID: 3235 (httpd)
Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /system.slice/httpd.service
├─3235 /usr/sbin/httpd -DFOREGROUND
├─3237 /usr/sbin/httpd -DFOREGROUND
├─3238 /usr/sbin/httpd -DFOREGROUND
├─3239 /usr/sbin/httpd -DFOREGROUND
├─3240 /usr/sbin/httpd -DFOREGROUND
├─3241 /usr/sbin/httpd -DFOREGROUND
└─3242 /usr/sbin/httpd -DFOREGROUND

Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: Rebuilding directory structure
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: Updating database. This may take a while.
Sep 23 19:50:32 pinehurst.e.com rb-site[3228]: The log output below, including warnings and errors,
:
:
:

[Paul Fee]

Hi Stephen,

I don't recall exact details, but I've seen the "rb-site upgrade" command fail on occasion.  I think in my case the sequence was:

* yum update       [Included a reviewboard update]
* reboot
* While httpd service was starting rb-site upgrade was invoked.
* rb-site upgrade failed because mariadb had not started yet.

I manually fixed the issue by starting mariadb and running rb-site upgrade by hand.  However it does appear there's a race condition between httpd startup and rb-site's dependency on the database being available.

I'm not sure if this matches David's problem, but hopefully the above sequence will help you reproduce his issue and enhance the auto upgrade feature you implemented for Fedora packages.

Thanks,
Paul

[David Carson]

Paul,

This is not the issue for me.  I am demonstrating below that (a) the 'restart httpd' succeeds when the hook is not in place, and (b) that even if mariadb is running, the 'restart httpd' hangs when the hook is in place.  The "(0)" at the beginning of my prompt is the return code ($?).

(0)[root@pinehurst httpd.service.d] pwd

/usr/lib/systemd/system/httpd.service.d

(0)[root@pinehurst httpd.service.d] ls

reviewboard-sites.conf.no.go

(0)[root@pinehurst httpd.service.d]

(0)[root@pinehurst httpd.service.d] systemctl restart httpd.service  <<-- OK, no hang

(0)[root@pinehurst httpd.service.d]

(0)[root@pinehurst httpd.service.d]

(0)[root@pinehurst httpd.service.d] mv reviewboard-sites.conf.no.go reviewboard-sites.conf

(0)[root@pinehurst httpd.service.d] systemctl daemon-reload

(0)[root@pinehurst httpd.service.d]

(0)[root@pinehurst httpd.service.d] systemctl restart mariadb.service  <<-- make sure mariadb is running

(0)[root@pinehurst httpd.service.d]

(0)[root@pinehurst httpd.service.d] systemctl restart httpd.service <<-- this one hangs (status 'activating')

[David Carson]

By the way, I can do the 'rb-site upgrade --all-sites' as a stand-alone command, whether or not the hook is in place.  So, the problem has something to do with the rb-site being called by the httpd restart (or start).

[Stephen Gallagher]

On 09/24/2014 10:06 AM, David Carson wrote:
> By the way, I can do the 'rb-site upgrade --all-sites' as a stand-alone
> command, whether or not the hook is in place. So, the problem has
> something to do with the rb-site being called by the httpd restart (or
> start).
>
>
> On Wednesday, September 24, 2014 9:59:59 AM UTC-4, David Carson wrote:
>
> Paul,
> This is not the issue for me. I am demonstrating below that (a) the

> 'restart httpd' succeeds when the hook is *not* in place, and (b)

> that even if mariadb is running, the 'restart httpd' hangs when the
> hook is in place. The "(0)" at the beginning of my prompt is the
> return code ($?).
>
> (0)[root@pinehurst httpd.service.d] pwd
> /usr/lib/systemd/system/httpd.service.d
> (0)[root@pinehurst httpd.service.d] ls

> reviewboard-sites.conf.*no.go*

> (0)[root@pinehurst httpd.service.d]
> (0)[root@pinehurst httpd.service.d] systemctl restart

> httpd.service *<<-- OK, no hang*

> (0)[root@pinehurst httpd.service.d]
> (0)[root@pinehurst httpd.service.d]

> (0)[root@pinehurst httpd.service.d] *mv*
> reviewboard-sites.conf.*no.go* reviewboard-sites.conf

> (0)[root@pinehurst httpd.service.d] systemctl daemon-reload
> (0)[root@pinehurst httpd.service.d]
> (0)[root@pinehurst httpd.service.d] systemctl restart

> mariadb.service *<<-- make sure mariadb is running*

> (0)[root@pinehurst httpd.service.d]
> (0)[root@pinehurst httpd.service.d] systemctl restart

> httpd.service *<<-- this one hangs (status 'activating')*
>


Yeah, I've been looking at it today and discovered that I was doing
something wrong.

tl;dr version: Delete the 'type=oneshot' line.


There were a couple of bugs here. The core problem is that I
misunderstood what the unit files in the drop directories did. I was
actually replacing the Type of httpd.service, not setting one just for
the add-on unit. The effect was that httpd.service was behaving as a
oneshot type and was not returning control until it exited. Oops.

However, because I typoed "Type", systemd was just logging an error and
ignoring it. I hadn't noticed the logged error, but since things were
working (including upgrades) I never had cause to look in the log.

So I need to actually just remove that line entirely and the logged
error will go away and the upgrade script will run as expected.

I'm putting out updated packages for 2.0.8 in a few minutes which will
have the corrected unit file.

[David Carson]

Stephen,

Yep, removing the "...oneshot" line fixed it for me.  Thanks a lot.

~David