/etc/init.d/weewx not stopping weewx

[Graham Eddy]

i am setting up weex on a new raspberry pi (re-aquainting with debian after years away - it has changed!).

at this stage, i am using simulator driver.

weewx runs non-root as wxsys:wxsys quite happily (with pidfile in weewx/run/ not /var/run/), generating graphs etc, no problems in log

"/etc/init.d/weewx start" starts up weewx okay, pidfile is correct

"/etc/init.d/weewx stop" hangs for ~60 secs, then *says* stopped at shell prompt, but weewx is still running, syslog says stop failed, and systemctl reports weewx.service to be in failed state. syslog also says “non-root pidfile is insecure” but i think that’s just a warning

is this (weewx not stopping) a known problem and, if so, is there a fix? couldn't find anything pertinent in weewx-user forum, nor anything helpful asking dr google

i thought maybe a dependency wasn’t being satisfied leading to hanging until timeout, so

in /etc/init.d/weex’s LSB comment i changed Required-Stop label to Should-Stop, which didn’t help, so i tried setting Required-Stop value to $none, which also didn’t help

i noticed that the 60 secs hang after issuing stop instruction corresponds to default timeout, so

i tried adding “—retry TERM/5/KILL/2” to “start-stop-daemon —stop” in case weewx missed first SIGTERM, and we know start-stop-daemon doesn’t check that daemon has actually stopped unless it is told to with —retry. this didn’t help

i tried manually sending SIGTERM to weewxd process during this hang period, and start-stop-daemon immediately noticed and completed successfully, including removing pidfile. in this case, i had killed weewxd process so not running, and systemctl did not report weewx.service to be in failed state, so it was happy.

this has lead to my current workaround, where i replace

start-stop-daemon —stop —pidfile $WEEWX_PID

with

start-stop-daemon —stop —pidfile $WEEWX_PID &

sleep 2 ; kill $(cat $WEEWX_PID) ; wait

which shuts weewx down neatly but i’d prefer a proper solution

cheers

g-eddy

[mwall]

when you do '/etc/init.d/weewx stop' or 'systemctl stop weewx', does weewx eventually stop?

when you do either of these commands, this sends a SIGTERM to weewx, which tells the main thread to stop.  however, weewx will not actually stop until all of its threads have completed.  so if you have a report that takes a long time to run then weewx may not shut down until well after you send the SIGTERM.

this is often noticeable on rpi with a weewx configuration that has many compute-intensive reports

if you have an extension with misbehaving threads or forked processes, then you might also end up with zombie threads/processes that prevent weewx from shutting down properly.

m

[Graham Eddy]

it is a new install with no extensions and unchanged skin.

i banged my head against this problem all day yesterday, and today it works fine and i can’t reproduce it :-/

an example syslog extract from yesterday using ‘/etc/init.d/weewx stop'. note that weewx does not report receiving SIGTERM:

Jul  7 22:48:05 dizzy systemd[1]: Stopping LSB: weewx weather system...

Jul  7 22:48:05 dizzy weewx[3561]: Stopping weewx weather system: weewxstart-stop-daemon: matching only on non-root pidfile /opt/weewx/var/run/weewx.pid is insecure

Jul  7 22:48:57 dizzy systemd[1]: Started Session c5 of user graham.

Jul  7 22:50:06 dizzy weewx[3561]: ........................ failed!

Jul  7 22:50:06 dizzy systemd[1]: weewx.service: Control process exited, code=exited, status=1/FAILURE

Jul  7 22:50:06 dizzy systemd[1]: weewx.service: Failed with result 'exit-code'.

Jul  7 22:50:06 dizzy systemd[1]: Stopped LSB: weewx weather system.

Jul  7 22:50:15 dizzy weewx[605] INFO weewx.manager: Added record 2020-07-07 22:50:00 AEST (1594126200) to database 'weewx.sdb'

Jul  7 22:50:15 dizzy weewx[605] INFO weewx.manager: Added record 2020-07-07 22:50:00 AEST (1594126200) to daily summary in 'weewx.sdb'

Jul  7 22:50:16 dizzy weewx[605] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 0.87 seconds

maybe all my fiddling put it (or me!) into a strange state...

now that i know ‘systemctl stop weewx’, i will give that a try next time

i mount weewx/public_html as tmpfs, so weewx re-generates images etc after a reboot.

maybe some of my tests, where i stopped weewx very soon after reboot, were done while that re-generation was occurring so the reporting threads were busy. but the example above was when system was idle

Q: does weewx log the received SIGTERM when it arrives (as i assume) or when it is ready to deal with it?

will report in when i can reproduce the problem again...

thanks

g-eddy

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/5942bce7-b168-433e-ae2e-71fe10f78f31o%40googlegroups.com.

[Tom Keffer]

There is another possibility: there are two kinds of threads in Python: non-daemon and daemon. The former is as you describe: it will prevent the program from exiting when a termination signal is received. However, the latter will not: the thread dies when the program dies. 

WeeWX generally uses daemon threads. However, it's possible some extension or driver is creating non-daemon threads, thus blocking termination until the thread dies.

-tk

--

[Graham Eddy]

of course i was unable to reproduce problem yesterday - my workaround was still in place! (and working well). doh!

removed my workaround and voilà! problem exhibited again.

to summarise:

 * non-root owner wxsys:wxsys with pidfile in weewx/run/; /etc/init.d/weewx changed to reflect this

 * no extensions added, no skin configs, high end RPi (4B 8GB with U-3 SD)

 * tests run while no reports being generated

 * ’sudo /etc/init.d/weewx stop’ and ’sudo /systemctl stop weewx’ fail to stop weewx; weewx process running happily but systemctl reports it in failed state

Jul  9 14:45:25 dizzy weewx[621] INFO weewx.reportengine: Copied 5 files to /opt/weewx/public_html

Jul  9 14:47:36 dizzy systemd[1]: Stopping LSB: weewx weather system...

Jul  9 14:47:36 dizzy weewx[902]: Stopping weewx weather system: weewxstart-stop-daemon: matching only on non-root pidfile /opt/weewx/run/weewx.pid is insecure

Jul  9 14:49:37 dizzy weewx[902]: ........................ failed!

Jul  9 14:49:37 dizzy systemd[1]: weewx.service: Control process exited, code=exited, status=1/FAILURE

Jul  9 14:49:37 dizzy systemd[1]: weewx.service: Failed with result 'exit-code'.

Jul  9 14:49:37 dizzy systemd[1]: Stopped LSB: weewx weather system.

Jul  9 14:50:16 dizzy weewx[621] INFO weewx.manager: Added record 2020-07-09 14:50:00 AEST (1594270200) to database 'weewx.sdb'

my interpretation of log:

 * after issuing ‘stop’ command, the command logged “non-root pidfile is insecure” then hung for 121 secs

 * weewx carried on and never reported SIGTERM => signal was not sent

 * ‘stop’ command reports failure after timeout and marks state ‘failed’

previous observations:

 * if weewxd process manually killed during command hang, command notices weewxd gone and completes successfully

hypothesis: the “non-root pidfile is insecure” msg indicates start-stop-daemon refused to send signal to pid found in “insecure” pidfile.

corollary: this would work fine if run as root, not as wxsys:wxsys

Q: is anyone else trying non-root? do they see this behaviour or is it just me?

in the meantime, my workaround goes back in place.

cheers

g-eddy

[Graham Eddy]

fixed! for non-root user running weewx.

i recommend this be updated in util/init.d/weewx.debian (after someone tests it for root user…)

start-stop-damon(8) clearly says

-p, --pidfile pid-file

              ...

              Warning:  using  this match option with a world-writable pidfile

              or using it alone with a daemon that writes the  pidfile  as  an

              unprivileged  (non-root)  user  will  be  refused  with an error

              (since version 1.19.3) ...

‘using it alone’ → so i just added another daemon-identifying matching option readily to hand, the username

the ‘stop’ request in weewx script becomes

start-stop-daemon --stop --pidfile $WEEWX_PID --user $(echo $WEEWX_USER | sed 's/:.*$//')

and it works like a charm!