Weewx service stops randomly

[Clyde]

I'm running weewx 4.5.1 with a Belchertown skin on a Raspberry pi. All was working fine, but recently the service randomly stops. There is nothing in the /var/log/messages other than normal reports. Restarting the service (`service weewx restart`) resumes normal operation until the next time. The failures are happening maybe once a week, but not on any regular schedule.

Any ideas on how to troubleshoot this?

[Tom Keffer]

Not much information here. I know you think there is nothing in the log, but humor us, include it anyway. Please see Help! Posting to weewx user.

The log /var/log/syslog is preferred.

On Thu, Nov 25, 2021 at 12:57 PM Clyde <cl...@stubbs.me> wrote:

I'm running weewx 4.5.1 with a Belchertown skin on a Raspberry pi. All was working fine, but recently the service randomly stops. There is nothing in the /var/log/messages other than normal reports. Restarting the service (`service weewx restart`) resumes normal operation until the next time. The failures are happening maybe once a week, but not on any regular schedule.

Any ideas on how to troubleshoot this?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/650a5782-259b-4412-a1ba-a2405ec38e14n%40googlegroups.com.

[gjr80]

Yes. Edit weewx.conf and set debug=1, restart WeeWX and wait for a failure. Once you have a failure take two log extracts and post them here. The first extract needs to show the full WeeWX restart and the first couple of report cycles. The second extract should cover the failure but needs to start a couple of report cycles before the failure.

Sometimes the clue is not what is in the log but what is not.

Gary

[Clyde]

Thanks, that was  helpful - I had been looking in /var/log/messages. The syslog does contain useful information:

Nov 23 07:32:10 weather kernel: [4443467.475752] Out of memory: Killed process 25057 (python3) total-vm:357356kB, anon-rss:255512kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:324kB oom_score_adj:0

So at that time it was using around 255MB resident memory. Checking now (weewx has been running for a day or two) and it's 120MB rss. After a restart and waiting for an update round, it's 44MB resident and 100MB virtual (vs the 357MB virtual at time of OOM.)

So it looks like a memory leak to me. A daily restart would probably work around it, but before upgrading to 4.5.1 and Belchertown I had no such issues (was running 3.9.2 for a looong time.)

It's running on a Pi Zero with 0.5GB RAM.

[vince]

You might want to install my memory extension (link) to get some data and make some minimal graphs to go along with that.

I run on a 128MB system which runs for many many months without issues, FWIW, with Belchertown and several other skins and uploaders enabled.    My experience is that it will likely grow 'slightly' but level off after a while.  You certainly shouldn't see it run you out of memory quickly.

Do you have swap enabled on your pi ?    Have you adjusted your memory split to try to get more RAM for compute ?

[Tom Keffer]

We have seen many instances where memory has grown after a system update. Some underlying library starts to leak. Vice-versa as well. Perhaps in the last two days you did an update that fixed whatever was broken?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d4ec3209-61b8-4c0f-94c0-9a94803e6d6fn%40googlegroups.com.

[vince]

On Thursday, November 25, 2021 at 4:17:57 PM UTC-8 Tom Keffer wrote:

We have seen many instances where memory has grown after a system update. Some underlying library starts to leak. Vice-versa as well. Perhaps in the last two days you did an update that fixed whatever was broken?

Piling on here but... Clyde can you tell us what os and version you are running ?

Did you install anything new recently ? 

Specifically did you update anything related to the imaging libraries like 'pil' or 'pillow' ?

 

[Clyde]

Thanks for the replies - in no particular order:

I'm running "Raspbian GNU/Linux 10 (buster)" on a Pi Zero. I used to run Raspbian Wheezy on an original Pi, with weewx 3.9.2. I don't think this problem was there when I first migrated to 4.5.1 - but I haven't touched the system since then so maybe it was.

vince: I would install the memory extension, but I fell at the first hurdle: "setup.py install --extension extensions/mem" - where is setup.py? I installed weewx using apt, and can't find a setup.py anywhere.

I do not have swap enabled, nor do I particularly want to. 512MB should be plenty (it certainly was with 3.9.2 on the old Pi.)

I have not installed anything new recently (i.e. since first moving to 4.5.1 on Buster.)

> Perhaps in the last two days you did an update that fixed whatever was broken?

Nope, nothing is fixed (it was down again just now), nor did I do any updates. Maybe I should....

[Clyde]

correction - I do apparently have swap enabled, only 100MB. That must have been the default Raspbian setup.

[vince]

Sorry Clyde, this one pre-dates the wee_extension installer.

I've updated the readme in the github repo with today's installation instructions.

Download the zip file, install with wee_extension, restart weewx.

That simple.

[Clyde]

Well, after doing an apt upgrade the problem seems solved. I've set a cron job to capture the RSS of the weewx daemon periodically, and after several days running it seems stable at under 50MB. So the theory about a buggy library looks likely.

[mup...@gmail.com]

Has someone worked out a cron job that will periodically check to see that weewx is running and issue a restart if it is not?

I have been getting notified (email) about once per week letting me know that weewx stopped sending data.

[John Kline]

You really should get to the bottom of why weewx is dying.  Logs are your friend.

I don’t have a watchdog for weewx, but I have such jobs for other software.  Here’s a simplified script that should do the job.  If you have mail set up, you should add a line to mail you that weewx had to be restarted.  Also, replace ‘/home/weewx/bin/weewxd’ with the location of your weewxd.   Assuming weewx is running a root, you’ll need to add the cron entry for root.

#!/bin/sh

# If we just started up, allow some time for the process to start.

if [ `cat /proc/uptime | cut -f1 -d '.'` -le  30 ]; then

  sleep 20

fi

ps aux | grep "/home/weewx/bin/weewxd" | grep -v grep > /dev/null

if [ $? -ne 0 ]; then

        /usr/shin/service weewx restart

fi

On Apr 18, 2022, at 9:12 PM, mup...@gmail.com <mup...@gmail.com> wrote:

Has someone worked out a cron job that will periodically check to see that weewx is running and issue a restart if it is not?

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/9b174589-69e8-4937-a9e2-8eb79de8a024n%40googlegroups.com.

[Graham Eddy]

you will probably get a lot more info from /var/log/syslog than /var/log/messages