Weatther Station stops same time every week

[Monica Mulholland]

Every week, on Saturday night at 11.50 pm my station stops reporting and needs to be rebooted. I am on a RPi4 (recently upgraded). Other than this little glitch, which is annoying, it is running very well since I upgraded to the RPi4.

Any suggestions of where I might look to try and sort this issue?

Any help greatly appreciated.

[vince]

Your system logs.

[dunbrokin]

Apologies, yes it's always a good idea.

 For the last 3 weeks on Sat night at 11.50 it stopped recording.....but rebooting started it again the next day. However this week after rebooting it stopped at 11.10 am and despite rebooting, it never started recording again.

So I ran the log as per the instructions....but it only ran for about 6 minutes. Attached is all that was recorded in almost 15 minutes.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/68cf6bec-901e-449d-b75a-45b3fc7c75b5n%40googlegroups.com.

[vince]

Sigh - as always, set debug=1 and let it run for more like 35 minutes.  You have an archive period of 10 minutes if it's running off the hardware interval of 600 seconds so we need to see a few cycles.

[dunbrokin]

Oops Sorry, I thought I had hardwired in debug=1.....bottom of the class for me on this one I think....made all the rookie mistakes. Apologies for wasting your time!

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/79829310-aa83-4281-9244-7cc9adbc0037n%40googlegroups.com.

[vince]

Set debug=1 and maybe we can figure it out

[dunbrokin]

OK, lets see if I can get it right this time!

 mylog

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/ef35ee91-7527-4e17-93a4-f3f7d5bf91c8n%40googlegroups.com.

[vince]

Let it run through the weekend and if it fails again this Saturday post whatever is logged please.

[Monica Mulholland]

But, it is not running at all at the moment.....it is not showing on the website. On Wunderground it says offline. Despite repeated reboots, it has not run in nearly a week.

[gjr80]

I have no idea what might be causing your nightly stoppage, but from the most recent log posted you almost certainly have corrupt station memory. The clues are in hardware record generation being used, WeeWX talking to the console/logger but no archive records being dowloaded and reports are generated normally but with no new data. The net effect is that WeeWX can obtain loop packet data from the console but it cannot obtain archive records. You might want to work through the Corrupt station memory section of the Troubleshooting the Davis Vantage station wiki page.

As for the nightly stoppage, once you get your station working again you might want to leave debug = 1 and post a log extract covering say 30 minutes either side of the stoppage time.

Gary

[Monica Mulholland]

Thank you!

[Monica Mulholland]

This is the log from last night from before and after the time it shuts down each week and does not restart unless the Rpi is rebooted.

[Tom Keffer]

These four lines are interesting:

Sep 01 00:00:01 weatherpi CRON[10907]: (weatherpi) CMD (/home/weatherpi/backup_weewx.sh)
Sep 01 00:00:01 weatherpi CRON[10906]: (weatherpi) CMD (/home/weatherpi/camera_image1.sh)

Sep 01 00:00:01 weatherpi sudo[10911]: weatherpi : PWD=/home/weatherpi ; USER=root ; COMMAND=/usr/bin/systemctl stop weewx
Sep 01 00:00:01 weatherpi sudo[10911]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)

First some sort of backup shell script /home/weatherpi/backup_weewx.sh is invoked, then something for a camera image. Then the user identified by uid 1000 (presumably user 'weewx'?) got elevated privileges using "sudo", then used them to terminate weewxd. This is happening at the same time as the backup script.

It feels like the backup script is deliberately terminating weewxd before starting the backup. Take a look in the script /home/weatherpi/backup_weewx.sh

As for why this is happening at midnight Sunday morning, perhaps the backup script, or something else, is invoked by crontab. Look in /etc/crontab and in /etc/cron.weekly.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a90e0cb4-8728-443c-a04c-bd4f21cee9f1n%40googlegroups.com.

[Monica Mulholland]

Thank you for that. Here the screenshot of the files.

[Tom Keffer]

Monica: you're going to have to do a little investigation. Take a look in /home/weatherpi/backup_weewx.sh. Anything in there that could cause a weekly reboot?

Also the path /etc/cron.weekly is a directory. You'll have to look inside for any files. Could any of them cause a weekly boot?

How about the file /home/weatherpi/camera_image1.sh?

I encourage you to do a little investigative forensics on your computer. It's hard to diagnose these remotely. 

I could be wrong, but I really doubt this is a weewx problem. It's something specific to your computer.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/4efe8b66-2352-4707-bae5-e0b01a3976a1n%40googlegroups.com.

[Monica Mulholland]

Thanks for that Tom, I have no doubt that you are correct....as usual. I will dig around as you propose and see what I can find.

[Monica Mulholland]

This is the script in /home/weatherpi/backup_weewx.sh 

And what I am trying to do here is to copy the database every week to a third party location for back up purposes. This does stop weewx....but it is supposed to start it after the database is copied. As I understand it, you cannot copy the db while it is live. So I must have screwed up the commands here somewhere which stops weewx from restarting. This is what  was suggested by Chat GPT (my programming skills leave a little to be desired) as the solution. So it looks like that weewx does not restart for some reason....maybe it needs a time lapse before starting

#!/bin/bash

# Stop the WeeWX service
sudo systemctl stop weewx
if [ $? -ne 0 ]; then
    echo "Failed to stop weewx service"
    exit 1
fi

# Copy the database file (this will block until the copy is complete)
cp /var/lib/weewx/weewx.sdb /home/weatherpi/Public/weewx.sdb
if [ $? -ne 0 ]; then
    echo "Failed to copy the database file"
    sudo systemctl start weewx
    exit 1

fi

---------------------------------------------------------------------------------------------------------------------------------------

And this is what is in   /etc/cron.weekly/man-db    

Not sure where this came from...is it not part of weewx?

                     
#!/bin/sh
#
# man-db cron weekly

set -e

if [ -d /run/systemd/system ]; then
    # Skip in favour of systemd timer.
    exit 0
fi

# This should be set by cron, but apparently isn't always; see
# https://bugs.debian.org/209185.  Add fallbacks so that start-stop-daemon
# can be found.
export PATH="$PATH:/usr/local/sbin:/usr/sbin:/sbin"

iosched_idle=
# Don't try to change I/O priority in a vserver or OpenVZ.
if ! grep -Eq '(envID|VxID):.*[1-9]' /proc/self/status && \
   { [ ! -d /proc/vz ] || [ -d /proc/bc ]; }; then
    iosched_idle='--iosched idle'
fi

if ! [ -d /var/cache/man ]; then
    # Recover from deletion, per FHS.
    install -d -o man -g man -m 0755 /var/cache/man
fi

# regenerate man database
if [ -x /usr/bin/mandb ]; then
    # --pidfile /dev/null so it always starts; mandb isn't really a daemon,
    # but we want to start it like one.
    # shellcheck disable=SC2086
    start-stop-daemon --start --pidfile /dev/null \
                      --startas /usr/bin/mandb --oknodo --chuid man \
                      $iosched_idle \
                      -- --quiet
fi

exit 0

[vince]

There are multiple things in every os that run out of cron.  Don't overthink ones you don't recognize.  The one that rebuilds the 'man' db is normal.

Your /home/weatherpi/backup_weewx.sh script is really not good at all and has a major logic error in it (typical stupid ChatGPT).   As currently written the script stops weewx, copies the db, then only restarts weewx 'if' the copy failed.  If the copy works it never restarts weewx.  This is why it's not restarting.    Move the start command line to the last line in the file and that'll fix it.

If you want another way to do it, I back up mine nightly and this script has been working for maybe 10 years now without failures.  

https://raw.githubusercontent.com/vinceskahan/weewx-odds-and-ends/c278b02597bf29fca80b700b718a31ad26f6c3c5/weewx-backup-copy.sh

(note - I do a gzip of the file to compress it, and it saves nightly copies so once or twice a year I need to delete old versions.  Mine would be named vp2.sdb.2024_08_30.gz or the like.  The script should be very obvious where you'd need to edit things to salt to taste so to speak.)

[vince]

I might add that you can probably find where it's running from if you try "sudo crontab -l -u root" or "sudo crontab -l -u weatherpi" or similarly for whatever user you think it's running as.  It is likely that it's being called from a user-specific crontab under /var/spool/cron/crontabs which is where user-specific crontabs are located.

[Monica Mulholland]

Thank you, so much, Vince.....much obliged indeed!

[Monica Mulholland]

Ah! Yes, Vince, I can see it now...the start command is totally in the wrong place! Thank you for that!