Whew as a service on Raspberry Pi

[Mike Revitt]

I am running weewx as a service on my Raspberry Pi and have been having problems with durability.

Having been running weewx for several years on my Mac I know that from time to time the service will stop,  but on my Mac they are always restarted by the daemon service.

But on the Raspberry Pi, whenever it stops it just stays stopped.

I originally followed the directions in the Debian section of WeeWX: Installation using setup.py but when this stopped it gave no useful information. I then found a method for using systems.service at https://github.com/weewx/weewx/wiki/systemd and gave that a goe.

This has also failed to keep the service running but has a little more info to go on.

pi@raspberrypi:/var/www/html/MountWeather $ sudo systemctl status weewx

● weewx.service - weewx weather system

   Loaded: loaded (/etc/systemd/system/weewx.service; enabled; vendor preset: enabled)

   Active: inactive (dead) since Sat 2020-03-14 02:51:14 GMT; 2 days ago

 Main PID: 25401

Mar 14 02:50:14 raspberrypi weewxd[25401]: weewx[25401] INFO weewx.manager: Added record 2020-03-14 02:50:00 GMT (1584154200) to dai

Mar 14 02:50:16 raspberrypi weewxd[25401]: weewx[25401] INFO weewx.cheetahgenerator: Generated 7 files for report SeasonsReport in 1

Mar 14 02:50:17 raspberrypi weewxd[25401]: weewx[25401] INFO weewx.imagegenerator: Generated 9 images for report SeasonsReport in 0.

Mar 14 02:50:17 raspberrypi weewxd[25401]: weewx[25401] INFO weewx.reportengine: Copied 0 files to /var/www/html/MountWeather

Mar 14 02:50:17 raspberrypi weewxd[25401]: weewx[25401] INFO botocore.credentials: Found credentials in shared credentials file: ~/.

Mar 14 02:50:20 raspberrypi weewxd[25401]: weewx[25401] INFO user.s3upload: S3UploadGenerator: AWS-S3 copied 16 files to S3 in 3.32 

Mar 14 02:50:21 raspberrypi weewxd[25401]: weewx[25401] INFO botocore.credentials: Found credentials in shared credentials file: ~/.

Mar 14 02:51:14 raspberrypi systemd[1]: weewx.service: Killing process 25401 (weewxd) with signal SIGKILL.

Mar 14 02:51:14 raspberrypi systemd[1]: weewx.service: Killing process 25401 (weewxd) with signal SIGKILL.

Mar 14 02:51:14 raspberrypi systemd[1]: weewx.service: Succeeded.

The process has died during the boto copy to S3, and I am wondering if this is a network timeout or something, and this does appear to be where it normally stops.

However I am also using the command sudo systemctl enable weewx which is supposed to restart dead processes, but as you can see it isn't.

If I restart it it works perfectly for the next several hours then does this again.

Anyone got any experience of setting this up on a Raspberry Pi

pi@raspberrypi:/var/www/html/MountWeather $ sudo systemctl start weewx
pi@raspberrypi:/var/www/html/MountWeather $ sudo systemctl status weewx

● weewx.service - weewx weather system

   Loaded: loaded (/etc/systemd/system/weewx.service; enabled; vendor preset: enabled)

   Active: active (running) since Mon 2020-03-16 08:01:30 GMT; 1s ago

 Main PID: 18479 (weewxd)

    Tasks: 1 (limit: 2200)

   Memory: 7.5M

   CGroup: /system.slice/weewx.service

           └─18479 /usr/bin/python /home/weewx/bin/weewxd --daemon --pidfile=/run/weewx.pid /home/weewx/weewx.conf

Mar 16 08:01:31 raspberrypi weewxd[18479]: weewx[18479] INFO weewx.restx: Wunderground: Posting not enabled.

Mar 16 08:01:31 raspberrypi weewxd[18479]: weewx[18479] INFO weewx.restx: PWSweather: Posting not enabled.

Mar 16 08:01:31 raspberrypi weewxd[18479]: weewx[18479] INFO weewx.restx: CWOP: Posting not enabled.

Mar 16 08:01:31 raspberrypi weewxd[18479]: weewx[18479] INFO weewx.restx: WOW: Posting not enabled.

Mar 16 08:01:31 raspberrypi weewxd[18479]: weewx[18479] INFO weewx.restx: AWEKAS: Posting not enabled.

Mar 16 08:01:31 raspberrypi weewxd[18479]: weewx[18479] INFO __main__: Starting up weewx version 4.0.0b14

Mar 16 08:01:31 raspberrypi weewxd[18479]: weewx[18479] INFO weewx.engine: Clock error is -0.12 seconds (positive is fast)

Mar 16 08:01:31 raspberrypi weewxd[18479]: weewx[18479] INFO weewx.engine: Using binding 'wx_binding' to database 'weewx.sdb'

Mar 16 08:01:31 raspberrypi weewxd[18479]: weewx[18479] INFO weewx.manager: Starting backfill of daily summaries

Mar 16 08:01:31 raspberrypi weewxd[18479]: weewx[18479] INFO weewx.engine: Starting main packet loop.

[Thomas Keffer]

I would suggest looking directly at the log at /var/log/syslog, rather than rely on status commands. This may give you clues. For example, systemd may be harvesting weewxd for memory. 

'Enabling' a process means it will get started when the system boots, not after a crash.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c6468f7d-83fd-4acf-b8a9-828af09239e9%40googlegroups.com.

[Mike Revitt]

Thanks Thomas,

I found a parameter I can set in the service file which I am currently testing. Should know in a day or 2 if it has worked

[Service]

Restart=always

RestartSec=30

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Thomas Keffer]

You should still try to figure out the root causes of the crashes. WeeWX is capable of going weeks without a reboot. Here's an example where it's been running continuously for well over a year: http://www.threefools.org/weewx/status/index.html

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d25471c2-d201-4f66-a5e1-16245d05f124%40googlegroups.com.

[Mike Revitt]

Will examiner further, but there is nothing I can find in the syslog file, these are the lines either side of the final exection when it last failed

Mar 14 02:40:48 raspberrypi rngd[342]: stats: Lowest ready-buffers level: 2

Mar 14 02:40:48 raspberrypi rngd[342]: stats: Entropy starvations: 0

Mar 14 02:40:48 raspberrypi rngd[342]: stats: Time spent starving for entropy: (min=0; avg=0.000; max=0)us

Mar 14 02:45:14 raspberrypi weewx[25401] INFO weewx.manager: Added record 2020-03-14 02:45:00 GMT (1584153900) to database 'weewx.sdb'

Mar 14 02:45:14 raspberrypi weewx[25401] INFO weewx.manager: Added record 2020-03-14 02:45:00 GMT (1584153900) to daily summary in 'weewx.sdb'

Mar 14 02:45:16 raspberrypi weewx[25401] INFO weewx.cheetahgenerator: Generated 7 files for report SeasonsReport in 1.49 seconds

Mar 14 02:45:17 raspberrypi weewx[25401] INFO weewx.imagegenerator: Generated 9 images for report SeasonsReport in 0.77 seconds

Mar 14 02:45:17 raspberrypi weewx[25401] INFO weewx.reportengine: Copied 0 files to /var/www/html/MountWeather

Mar 14 02:45:17 raspberrypi weewx[25401] INFO botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials

Mar 14 02:45:20 raspberrypi weewx[25401] INFO user.s3upload: S3UploadGenerator: AWS-S3 copied 16 files to S3 in 3.21 seconds

Mar 14 02:50:14 raspberrypi weewx[25401] INFO weewx.manager: Added record 2020-03-14 02:50:00 GMT (1584154200) to database 'weewx.sdb'

Mar 14 02:50:14 raspberrypi weewx[25401] INFO weewx.manager: Added record 2020-03-14 02:50:00 GMT (1584154200) to daily summary in 'weewx.sdb'

Mar 14 02:50:16 raspberrypi weewx[25401] INFO weewx.cheetahgenerator: Generated 7 files for report SeasonsReport in 1.50 seconds

Mar 14 02:50:17 raspberrypi weewx[25401] INFO weewx.imagegenerator: Generated 9 images for report SeasonsReport in 0.76 seconds

Mar 14 02:50:17 raspberrypi weewx[25401] INFO weewx.reportengine: Copied 0 files to /var/www/html/MountWeather

Mar 14 02:50:17 raspberrypi weewx[25401] INFO botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials

Mar 14 02:50:20 raspberrypi weewx[25401] INFO user.s3upload: S3UploadGenerator: AWS-S3 copied 16 files to S3 in 3.32 seconds

Mar 14 02:50:21 raspberrypi weewx[25401] INFO botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials

Mar 14 02:51:14 raspberrypi systemd[1]: weewx.service: Killing process 25401 (weewxd) with signal SIGKILL.

Mar 14 02:51:14 raspberrypi systemd[1]: weewx.service: Killing process 25401 (weewxd) with signal SIGKILL.

Mar 14 02:51:14 raspberrypi systemd[1]: weewx.service: Succeeded.

Mar 14 03:17:01 raspberrypi CRON[8707]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

Mar 14 03:18:04 raspberrypi kernel: [466646.601996] brcmfmac: brcmf_run_escan: error (-52)

Mar 14 03:18:04 raspberrypi kernel: [466646.602012] brcmfmac: brcmf_cfg80211_scan: scan error (-52)

Mar 14 03:40:48 raspberrypi rngd[342]: stats: bits received from HRNG source: 3940064

Mar 14 03:40:48 raspberrypi rngd[342]: stats: bits sent to kernel pool: 3900000

Mar 14 03:40:48 raspberrypi rngd[342]: stats: entropy added to kernel pool: 3900000

Mar 14 03:40:48 raspberrypi rngd[342]: stats: FIPS 140-2 successes: 197

Mar

Are there any parameters I can set to. get more information

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d25471c2-d201-4f66-a5e1-16245d05f124%40googlegroups.com.

[Bob Atchley]

I can't help with your crash reason, but interestingly I missed the documentation on weewx systemd and independently set up my weewx systemd service.  Its very similar but works (at least for me on a raspberry pi 2).  There is one difference (that may or may not be important) which is that I created the weewx service file here:

/lib/systemd/system/weewx.service

so /lib rather than /etc

The content is similar, but I'll copy here in case its important:

[Unit]

Description=weewx weather station data gatherer

After=multi-user.target

[Service]

Type=simple

User=weewx

Group=weewx

WorkingDirectory=/home/weewx/

ExecStart=/home/weewx/bin/weewxd /home/weewx/weewx.conf

Restart=always

RestartSec=60

[Install]

WantedBy=multi-user.target

Hope this helps

Regards

Bob

[Mike Revitt]

Thanks Bob,

I had already found

Restart=always

RestartSec=60

And these appear to have stopped the crashes, and I know they shouldn't but my system has now been running without a restart for nearly 24 hours, the previous best was 11 hours.

If it survives for more than a few days then I am going to cut over to the new version and my Raspberry Pi.

[Mike Revitt]

I have now gotten to the bottom of the crashes.

The copy of the database uptp S3 takes more than 5 minutes and as my archive interval is 5 minutes I was getting a checksum error as the source file was getting modified before completing.

I have fixed that by doing a local copy prior to the copy unto S3, but that is now giving me this error

Mar 17 14:30:19 raspberrypi weewx[8008] INFO botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials

Mar 17 14:30:25 raspberrypi weewx[8008] INFO user.s3upload: S3UploadGenerator: AWS-S3 copied 16 files to S3 in 6.31 seconds

Mar 17 14:30:25 raspberrypi weewx[8008] INFO botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials

Mar 17 14:35:16 raspberrypi weewx[8008] INFO weewx.manager: Added record 2020-03-17 14:35:00 GMT (1584455700) to database 'weewx.sdb'

Mar 17 14:35:16 raspberrypi weewx[8008] INFO weewx.manager: Added record 2020-03-17 14:35:00 GMT (1584455700) to daily summary in 'weewx.sdb'

Mar 17 14:35:16 raspberrypi weewx[8008] INFO weewx.engine: Launch of report thread aborted: existing report thread still running

Mar 17 14:38:09 raspberrypi weewx[8008] INFO user.s3backup: S3BackupGenerator: S3 copied files to S3 in 0 attemps which took in 463.65 seconds

Mar 17 14:40:16 raspberrypi weewx[8008] INFO weewx.manager: Added record 2020-03-17 14:40:00 GMT (1584456000) to database 'weewx.sdb'

Mar 17 14:40:16 raspberrypi weewx[8008] INFO weewx.manager: Added record 2020-03-17 14:40:00 GMT (1584456000) to daily summary in 'weewx.sdb'

Mar 17 14:40:18 raspberrypi weewx[8008] INFO weewx.cheetahgenerator: Generated 7 files for report SeasonsReport in 1.84 seconds

So the upload is currently taking 463 seconds and this is blocking the next loop of the report process, not a huge issue as I plan to only back-up at midnight, but was wondering if the backup thread could be started as a parallel thread rather than a serial one

[p q]

In my backup script i stop weewx until the local copy completes. Then I restart it and then copy to Google drive. Has worked consistently for a couple of years. 

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c4bb69de-024b-4b67-92b8-7e734575d7c9%40googlegroups.com.

[Ξ]

I have my own .sdb back up service which looks like this below. It runs once a day but you can schedule it any way you wish and since the R-Pi's connection is a 3G modem that runs mostly on GPRS speeds it helps compressing the file first and running it independently:

[Unit]
Description=wxdb

[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c 'cp /home/weewx/archive/weewx.sdb /home/weewx/archive/dbkp/wxbkp.sdb.$(date +"%F")'
ExecStart=/usr/bin/bash -c 'tar -czf /home/weewx/archive/dbkp/wxbkp.$(date +"%F").tar.gz /home/weewx/archive/dbkp/wxbkp.sdb.$(date +"%F")'
ExecStart=/usr/bin/bash -c 'rsync /home/weewx/archive/dbkp/wxbkp.$(date +"%F").tar.gz -e ssh us...@hostname.net:/home/db'
ExecStart=/usr/bin/bash -c 'rm /home/weewx/archive/dbkp/wxbkp.sdb.$(date +"%F") /home/weewx/archive/dbkp/wxbkp.$(date +"%F").tar.gz'