wtf just happened to my local staging web server
Gary Dale
My Apache2 file/print/web server is running Bullseye. I had to
restart it yesterday evening to replace a disk drive. Otherwise
the last reboot was a couple of weeks ago - I recall some updates
to Jitsi - but I don't think there were any updates since then.
Today I find that I can't get through to any of the sites on the
server. Instead I get the Apache2 default web page. This happens
with both Firefox and Chromium. This happens for all the staging
sites (that I access as ".loc" through entries in my hosts file).
My jitsi and nextcloud servers simply report failure to get to the
server.
I verified that the site files (-available and -enabled) haven't changed in months.
I tried restarting the apache2 service and got an error so I tried stopping it then starting it again - same error:
root@TheLibrarian:~#
service apache2 start
Job for apache2.service failed because the control process
exited with error code.
See "systemctl status apache2.service" and "journalctl -xe" for
details.
root@TheLibrarian:~# systemctl status apache2.service
●
apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service;
enabled; vendor preset: enabled)
Active: failed (Result:
exit-code) since Wed 2022-05-04 12:16:55 EDT; 5s ago
Docs: https://httpd.apache.org/docs/2.4/
Process: 7932 ExecStart=/usr/sbin/apachectl start (code=exited,
status=1/FAILURE)
CPU: 29ms
May 04 12:16:55 TheLibrarian systemd[1]: Starting The Apache
HTTP Server...
May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address
already in use: AH00072: make_sock: could not bind to addre>
May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address
already in use: AH00072: make_sock: could not bind to addre>
May 04 12:16:55 TheLibrarian apachectl[7935]: no listening
sockets available, shutting down
May 04 12:16:55 TheLibrarian apachectl[7935]: AH00015: Unable to
open logs
May 04 12:16:55 TheLibrarian apachectl[7932]: Action 'start'
failed.
May 04 12:16:55 TheLibrarian apachectl[7932]: The Apache error
log may have more information.
May 04 12:16:55 TheLibrarian systemd[1]: apache2.service:
Control process exited, code=exited, status=1/FAILURE
May 04 12:16:55 TheLibrarian systemd[1]: apache2.service:
Failed with result 'exit-code'.
May 04 12:16:55 TheLibrarian systemd[1]: Failed
to start The Apache HTTP Server.
also
root@TheLibrarian:/var/log#
journalctl -xe
░░ The job
identifier is 4527.
May 04 12:50:49 TheLibrarian apachectl[8232]: (98)Address
already in use: AH00072: make_sock: could not bind to addre>
May 04 12:50:49 TheLibrarian apachectl[8232]: (98)Address
already in use: AH00072: make_sock: could not bind to addre>
May 04 12:50:49 TheLibrarian apachectl[8232]: no listening
sockets available, shutting down
May 04 12:50:49 TheLibrarian apachectl[8232]: AH00015: Unable to
open logs
May 04 12:50:49 TheLibrarian apachectl[8229]: Action 'start'
failed.
May 04 12:50:49 TheLibrarian apachectl[8229]: The Apache error
log may have more information.
May 04 12:50:49 TheLibrarian systemd[1]: apache2.service:
Control process exited, code=exited, status=1/FAILURE
░░ Subject: Unit
process exited
░░ Defined-By:
systemd
░░ Support:
https://www.debian.org/support
░░
░░ An ExecStart=
process belonging to unit apache2.service has exited.
░░
░░ The process'
exit code is 'exited' and its exit status is 1.
May 04 12:50:49 TheLibrarian systemd[1]: apache2.service:
Failed with result 'exit-code'.
░░ Subject: Unit
failed
░░ Defined-By:
systemd
░░ Support:
https://www.debian.org/support
░░
░░ The unit
apache2.service has entered the 'failed' state with result
'exit-code'.
May 04 12:50:49 TheLibrarian systemd[1]: Failed
to start The Apache HTTP Server.
░░ Subject: A
start job for unit apache2.service has failed
░░ Defined-By:
systemd
░░ Support:
https://www.debian.org/support
░░
░░ A start job
for unit apache2.service has finished with a failure.
░░
░░ The job
identifier is 4527 and the job result is failed.
As I said, I do get the default Apache2 page saying "It works"
but that appears to be optimistic. ps aux | grep apache2 fails to
show the service, which confirms the systemctl message that it
isn't running.
There is nothing in /var/log/apache2/error.log. The .1 log ends
yesterday but only contains complaints about php7. Systemctl does
report (above) "unable to open logs" so that would explain the
lack of additional messages. The apache2 directory and its files
are root:adm with only root having write privileges.
I tried giving the adm group write privileges but that didn't
work. Turns out the group is empty. Adding www-data to it didn't
work either.
Any ideas on how to track down the cause of the failure(s)?
Thanks.
Greg Wooledge
> May 04 12:16:55 TheLibrarian systemd[1]: Starting The Apache HTTP Server...
> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in use:
> AH00072: make_sock: could not bind to addre>
> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in use:
> AH00072: make_sock: could not bind to addre>
Assuming those ports are 80 and 443, you could use commands like this
to see what's using them:
lsof -i :80
lsof -i :443
If your configuration is telling Apache to use some other ports, then
substitute your port numbers.
Gary Dale
that because I certainly didn't (doesn't seem likely though because they
both failed).
I guess I should pay more attention to the packages that get installed
when I do apt full-upgrade... Usually I just scan to see if there is
anything that I should reboot over.
Brian
[...]
My young childre read -user. They asked me what "wtf" means. Please,
explain, for the benefit of us civilised and acronymn-challenged
users, what it stands for.
For extra points, knowing whether it was an essential part of your
query would be interesting.
--
Brian.
Greg Wooledge
> My young childre read -user. They asked me what "wtf" means. Please,
> explain, for the benefit of us civilised and acronymn-challenged
> users, what it stands for.
Brian
indictated.
--
Brian.
Claudio Kuenzler
May 04 12:16:55 TheLibrarian systemd[1]: Starting The Apache HTTP Server...
May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in use: AH00072: make_sock: could not bind to addre>
May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in use: AH00072: make_sock: could not bind to addre>
May 04 12:16:55 TheLibrarian apachectl[7935]: no listening sockets available, shutting down
May 04 12:16:55 TheLibrarian apachectl[7935]: AH00015: Unable to open logs
May 04 12:16:55 TheLibrarian apachectl[7932]: Action 'start' failed.
May 04 12:16:55 TheLibrarian apachectl[7932]: The Apache error log may have more information.
May 04 12:16:55 TheLibrarian systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
May 04 12:16:55 TheLibrarian systemd[1]: apache2.service: Failed with result 'exit-code'.
May 04 12:16:55 TheLibrarian systemd[1]: Failed to start The Apache HTTP Server.
As I said, I do get the default Apache2 page saying "It works" but that appears to be optimistic. ps aux | grep apache2 fails to show the service, which confirms the systemctl message that it isn't running.
Erwan David
> My Apache2 file/print/web server is running Bullseye. I had to restart
> it yesterday evening to replace a disk drive. Otherwise the last reboot
> was a couple of weeks ago - I recall some updates to Jitsi - but I don't
> think there were any updates since then.
>
> Today I find that I can't get through to any of the sites on the server.
> Instead I get the Apache2 default web page. This happens with both
> Firefox and Chromium. This happens for all the staging sites (that I
> access as ".loc" through entries in my hosts file). My jitsi and
> nextcloud servers simply report failure to get to the server.
>
> I verified that the site files (-available and -enabled) haven't changed
> in months.
>
> I tried restarting the apache2 service and got an error so I tried
> stopping it then starting it again - same error:
>
> root@TheLibrarian:~# service apache2 start
not killed
[...]
>
> May 04 12:16:55 TheLibrarian systemd[1]: Starting The Apache HTTP Server...
> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in
> use: AH00072: make_sock: could not bind to addre>
> May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in
> use: AH00072: make_sock: could not bind to addre>
using the wanted ports.
Klaus Singvogel
Sometimes its also "what the freak", "what the failure", "world trade
federation", or "world taekwondo federation".
I think there are more.
Best regards,
Klaus.
--
Klaus Singvogel
GnuPG-Key-ID: 1024R/5068792D 1994-06-27
Stephan Seitz
>I think there are more.
Stephan
--
| If your life was a horse, you'd have to shoot it. |
Greg Wooledge
> > root@TheLibrarian:~# service apache2 start
>
> It looks like you started it, not restart, thus the running apache is not
> killed
>
> [...]
>
> >
> > May 04 12:16:55 TheLibrarian systemd[1]: Starting The Apache HTTP Server...
> > May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in
> > use: AH00072: make_sock: could not bind to addre>
> > May 04 12:16:55 TheLibrarian apachectl[7935]: (98)Address already in
> > use: AH00072: make_sock: could not bind to addre>
>
> This is consistent with former apache still running at that time, and using
> the wanted ports.
running, and you issue a "start" for it, it'll do nothing.
unicorn:~$ systemctl status mariadb
● mariadb.service - MariaDB 10.5.15 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor prese>
Active: active (running) since Tue 2022-05-03 07:16:46 EDT; 1 day 23h ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 683 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/>
Process: 691 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_STA>
Process: 694 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && V>
Process: 769 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_ST>
Process: 771 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/S>
Main PID: 749 (mariadbd)
Status: "Taking your SQL requests now..."
Tasks: 8 (limit: 14199)
Memory: 118.7M
CPU: 26.273s
CGroup: /system.slice/mariadb.service
└─749 /usr/sbin/mariadbd
Warning: some journal files were not opened due to insufficient permissions.
unicorn:~$ sudo systemctl start mariadb
[sudo] password for greg:
unicorn:~$ systemctl status mariadb
● mariadb.service - MariaDB 10.5.15 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor prese>
Active: active (running) since Tue 2022-05-03 07:16:46 EDT; 1 day 23h ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 683 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/>
Process: 691 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_STA>
Process: 694 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && V>
Process: 769 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_ST>
Process: 771 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/S>
Main PID: 749 (mariadbd)
Status: "Taking your SQL requests now..."
Tasks: 8 (limit: 14199)
Memory: 118.7M
CPU: 26.283s
CGroup: /system.slice/mariadb.service
└─749 /usr/sbin/mariadbd
Warning: some journal files were not opened due to insufficient permissions.
unicorn:~$ sudo journalctl -u mariadb | tail -n5
May 03 07:16:46 unicorn mariadbd[749]: 2022-05-03 7:16:46 0 [Note] Added new Master_info '' to hash table
May 03 07:16:46 unicorn mariadbd[749]: 2022-05-03 7:16:46 0 [Note] /usr/sbin/mariadbd: ready for connections.
May 03 07:16:46 unicorn mariadbd[749]: Version: '10.5.15-MariaDB-0+deb11u1' socket: '/run/mysqld/mysqld.sock' port: 3306 Debian 11
May 03 07:16:46 unicorn systemd[1]: Started MariaDB 10.5.15 database server.
May 03 07:16:47 unicorn mariadbd[749]: 2022-05-03 7:16:47 0 [Note] InnoDB: Buffer pool(s) load completed at 220503 7:16:47
Note that this "Started MariaDB ..." happened 2 days ago, consistent with
the "1 day 23h ago" from systemctl status. The start command that I issued
had no effect at all. It didn't even add a line to the journal.
Gary Dale
> Am Do, Mai 05, 2022 at 09:30:42 +0200 schrieb Klaus Singvogel:
>> I think there are more.
>
> Yes, I only know wtf as „what the fuck”.
>
> Stephan
>
TV/movie franchise, which uses frack as the expletive of choice.
These days "frack" also refers to a gas extraction process with terrible
environmental consequences, thereby justifying its use as an expletive
in the broader world. Fracking is derived from fracturing, the breaking
of something, which is appropriate in the case of my staging server
suddenly being broken.
Gary Dale
it. Also, in my original e-mail, I identified that Apache2 wasn't
running by running ps aux output through grep. Again, this confirms the
systemctl message - as Greg Wooledge mentions in his reply to you.
Greg Wooledge showed me how to diagnose the problem by identifying the
process (nginx in this case) that was grabbing the ports Apache2 needed.
Claudio Kuenzler also provided an alternative method of diagnosing the
problem.
My problem is I'm not all that conversant in tracking down network
issues, such as ports. I didn't know that lsof even had a port option.
And I'm still getting used to systemctl / journalctl.
Anyway, thanks for your attempt to help.
Richard Hector
> Am Do, Mai 05, 2022 at 09:30:42 +0200 schrieb Klaus Singvogel:
>> I think there are more.
>
Yes, but such language is not permitted on this list.
Richard