Showing message to weather website users when stopped working

[hind...@gmail.com]

I use weewx on a Raspberry pi and post my results to my own public weather station website (www.ashteadweather.com) with the excellent belchertown skin.  It all works very nicely most of the time, but now and again, something goes wrong and the Raspberry pi crashes and weewx stops working - then my weather website shows out of date results until I notice this and restart the pi and weewx - then all fine again.  Has anyone got any tips for tracking these problems proactively so as to minimise the downtime (rather than waiting until I happen to notice the problem) and perhaps getting weewx to show a holding message on the website until the problem is corrected?

Thanks

David.

[Andrew Milner]

The best tip I can give you is to identify why the site is crashing by looking at the weewx and webserver logs.

Weewx should never crash per se.  If it encounters an irrecpoverable situation it should restart.  If it is not restarting Tom would require, I am pretty sure, to see the log of the failure.  The log should also give you a clue as to the cause of the failure (if it is indeed weewx).

A more likely cause would be flakey RPi power supply or flakey SD card - both of which are pretty notorious issues when an RPi is running 24/7.  If you are also hosting the site on the RPi it is even possible that your site is being hacked from outside and brought down!!!

So there you have it, in a nutshell.  Find the cause first then try and find a solution!!!

[David Hindley]

Thanks.  I agree that it is probably the Raspberry Pi, rather than weewx.  My syslog file around the relevant time that it stopped working yesterday is as follows.  Does anyone have any idea how to interpret this? The "^@" is very puzzling to a linux novice such as me!

David.

Feb 27 15:30:28 raspberrypi weewx[9312]: sftp: uploaded file /belchertown/graphs/index.html

Feb 27 15:30:28 raspberrypi weewx[9312]: sftp: create remote directory /belchertown/records

Feb 27 15:30:29 raspberrypi weewx[9312]: sftp: uploaded file /belchertown/records/index.html

Feb 27 15:30:29 raspberrypi weewx[9312]: sftp: create remote directory /belchertown/reports

Feb 27 15:30:29 raspberrypi weewx[9312]: sftp: uploaded file /belchertown/reports/index.html

Feb 27 15:30:29 raspberrypi weewx[9312]: sftp: create remote directory /belchertown/about

Feb 27 15:30:29 raspberrypi weewx[9312]: sftp: uploaded file /belchertown/about/index.html

Feb 27 15:30:29 raspberrypi weewx[9312]: sftp: create remote directory /belchertown/images

Feb 27 15:30:29 raspberrypi weewx[9312]: sftp: create remote directory /reports

Feb 27 15:30:29 raspberrypi weewx[9312]: sftp: uploaded file /reports/index.html

Feb 27 15:30:29 raspberrypi weewx[9312]: sftp: create remote directory /about

Feb 27 15:30:29 raspberrypi weewx[9312]: sftp: uploaded file /about/index.html

Feb 27 15:30:29 raspberrypi weewx[9312]: sftp: create remote directory /images

Feb 27 15:30:30 raspberrypi weewx[9312]: sftpgenerator: transferred 34 files in 7.72 seconds

Feb 27 15:30:30 raspberrypi weewx[9312]: reportengine: Running report 'Highcharts_Belchertown'

Feb 27 15:30:30 raspberrypi weewx[9312]: reportengine: Found configuration file /etc/weewx/skins/Highcharts_Belchertown/skin.conf for report 'Highcharts_Belchertown'

Feb 27 15:30:30 raspberrypi weewx[9312]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras', 'user.belchertown_highchartsSearchX.$

Feb 27 15:30:30 raspberrypi weewx[9312]: manager: Daily summary version is 2.0

Feb 27 15:30:44 raspberrypi weewx[9312]: cheetahgenerator: Generated 4 files for report Highcharts_Belchertown in 14.36 seconds

Feb 27 15:30:44 raspberrypi weewx[9312]: copygenerator: copied 0 files to /var/www/html/weewx/belchertown

Feb 27 15:30:44 raspberrypi weewx[9312]: reportengine: Running report 'Belchertown'

Feb 27 15:30:44 raspberrypi weewx[9312]: reportengine: Found configuration file /etc/weewx/skins/Belchertown/skin.conf for report 'Belchertown'

Feb 27 15:30:44 raspberrypi weewx[9312]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras', 'user.belchertown.getData']

Feb 27 15:30:44 raspberrypi weewx[9312]: manager: Daily summary version is 2.0

Feb 27 15:30:47 raspberrypi weewx[9312]: cheetahgenerator: Generated 12 files for report Belchertown in 3.13 seconds

Feb 27 15:30:47 raspberrypi weewx[9312]: copygenerator: copied 1 files to /var/www/html/weewx/belchertown

Feb 27 15:30:47 raspberrypi weewx[9312]: reportengine: Running report 'FTP'

Feb 27 15:30:47 raspberrypi weewx[9312]: reportengine: Found configuration file /etc/weewx/skins/Ftp/skin.conf for report 'FTP'

Feb 27 15:30:47 raspberrypi weewx[9312]: ftpgenerator: FTP upload not requested. Skipped.

Feb 27 15:30:47 raspberrypi weewx[9312]: reportengine: Running report 'RSYNC'

Feb 27 15:30:47 raspberrypi weewx[9312]: reportengine: Found configuration file /etc/weewx/skins/Rsync/skin.conf for report 'RSYNC'

Feb 27 15:30:47 raspberrypi weewx[9312]: rsyncgenerator: rsync upload not requested. Skipped.

Feb 27 15:31:01 raspberrypi CRON[12528]: (pi) CMD (sudo cp -f /var/www/html/weewx/belchertown/json/day.json /var/www/html/weewx/belchertown/json/week.json /var/www/html/weewx/belchertown/json/month.json /var/www/html/weewx/belchertown/json/year.json /var/www/html/weewx/json)

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@$

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] Booting Linux on physical CPU 0x0

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] Linux version 4.14.79-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1159 SMP Sun Nov 4 17:50:20 GMT 2018

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] CPU: ARMv7 Processor [410fd034] revision 4 (ARMv7), cr=10c5383d

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] CPU: div instructions available: patching division code

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] OF: fdt: Machine model: Raspberry Pi 3 Model B Plus Rev 1.3

Feb 27 15:17:04 raspberrypi fake-hwclock[87]: Wed 27 Feb 15:17:01 UTC 2019

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] Memory policy: Data cache writealloc

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] cma: Reserved 8 MiB at 0x3ac00000

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] On node 0 totalpages: 242688

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] free_area_init_node: node 0, pgdat 80c85280, node_mem_map ba39f000

Feb 27 15:17:04 raspberrypi kernel: [    0.000000]   Normal zone: 2133 pages used for memmap

Feb 27 15:17:04 raspberrypi systemd[1]: Started Apply Kernel Variables.

Feb 27 15:17:04 raspberrypi kernel: [    0.000000]   Normal zone: 0 pages reserved

Feb 27 15:17:04 raspberrypi kernel: [    0.000000]   Normal zone: 242688 pages, LIFO batch:31

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] percpu: Embedded 17 pages/cpu @ba348000 s38720 r8192 d22720 u69632

Feb 27 15:17:04 raspberrypi systemd[1]: Started File System Check Daemon to report status.

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] pcpu-alloc: s38720 r8192 d22720 u69632 alloc=17*4096

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 240555

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] Kernel command line: 8250.nr_uarts=0 bcm2708_fb.fbwidth=640 bcm2708_fb.fbheight=480 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p7 rootfstype=ext4 elevator=deadli$

Feb 27 15:17:04 raspberrypi systemd-fsck[119]: e2fsck 1.43.4 (31-Jan-2017)

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] Memory: 940232K/970752K available (7168K kernel code, 576K rwdata, 2076K rodata, 1024K init, 698K bss, 22328K reserved, 8192K cma-reserved)

Feb 27 15:17:04 raspberrypi systemd[1]: Started Create Static Device Nodes in /dev.

Feb 27 15:17:04 raspberrypi kernel: [    0.000000] Virtual kernel memory layout:

Feb 27 15:17:04 raspberrypi kernel: [    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/P7KTI7Xv1CQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Milner]

1.  I recommend you put weewx log messages in their own log file - see wiki for a) separate logfile and also logrotation.

2.  The ^@ rubbish looks like possible corruption whilst executing a cron task

3.  What cron task was it trying to execute

4.  I see the clock has gone backwards around the rubbish point - implying the rpi has rebooted itself.  If that is the case you need to ensure that weewx does not restart again until it has a sensible time - or fit an RTC to your rpi

All looks like some form of corruption - power supply perhaps or memory card problem which is then causing a pi reboot and should cause a weewx restart - but the log portion you posted does not run long enough to give weewx restart details and ensure it held off until it had a valid time and ntp connection.

[David Hindley]

Many Thanks.

1.  I will do as you suggest re separate log files.

3. The cron task was something I had to add after installing the belchertown skin, just to copy some files from one directory to another, on the Rpi, before then using sftp to upload to my website. It seems to work most of the time, but, as you say, this seemed to be when the corruption occurred.  The CRON job works every five minutes at 1,6,11... minutes past each hour - on the basis that weewx should have updated the archive etc by then - maybe that is something to do with it.

4. Further down the log file, I see:

Feb 27 15:17:04 raspberrypi weewx[379]: Starting weewx weather system: weewxerror: unexpectedly disconnected from boot status daemon

and then...

Feb 27 15:17:05 raspberrypi weewx[423]: engine: Initializing weewx version 3.9.1

Feb 27 15:17:05 raspberrypi weewx[423]: engine: Using Python 2.7.13 (default, Sep 26 2018, 18:42:22) #012[GCC 6.3.0 20170516]

Feb 27 15:17:05 raspberrypi weewx[423]: engine: Platform Linux-4.14.79-v7+-armv7l-with-debian-9.6

Feb 27 15:17:05 raspberrypi weewx[423]: engine: Locale is 'en_GB.UTF-8'

Feb 27 15:17:05 raspberrypi weewx[423]: engine: pid file is /var/run/weewx.pid

Feb 27 15:17:05 raspberrypi weewx[467]: engine: Using configuration file /etc/weewx/weewx.conf

Feb 27 15:17:05 raspberrypi weewx[467]: engine: Debug is 1

Feb 27 15:17:05 raspberrypi weewx[379]: .

Feb 27 15:17:05 raspberrypi systemd[1]: Started LSB: weewx weather system.

Feb 27 15:17:05 raspberrypi weewx[467]: engine: Initializing engine

Feb 27 15:17:05 raspberrypi weewx[467]: engine: Loading station type Vantage (weewx.drivers.vantage)

Feb 27 15:17:06 raspberrypi weewx[467]: vantage: Driver version is 3.1.1

Feb 27 15:17:06 raspberrypi weewx[467]: vantage: Socket error while opening port 22222 to ethernet host 10.0.1.83.

Feb 27 15:17:06 raspberrypi weewx[467]: import of driver failed: [Errno 101] Network is unreachable (<class 'weewx.WeeWxIOError'>)

Feb 27 15:17:06 raspberrypi weewx[467]: engine: Unable to load driver: [Errno 101] Network is unreachable

Feb 27 15:17:06 raspberrypi weewx[467]:     ****  Exiting...

So, looks like the internet connection went down perhaps? (I was away when all this happened yesterday, so not sure)

Good point re making sure weewx doesn't restart until it has a sensible time via valid NTP connection - any idea how I do that?  I will order an RTC anyway I think.

Thanks

David.

[Thomas Keffer]

1. It looks like you're using the WeatherLinkIP logger in your Vantage. These are generally not as reliable as the serial and USB versions and, as you discovered, depend on the network being up before the Vantage is useful.

I am not a boot expert, but try this: go into /etc/init.d/weewx and change this

### BEGIN INIT INFO
# Provides:          weewx
# Required-Start:    $local_fs $remote_fs $syslog $time
# Required-Stop:     $local_fs $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: weewx weather system
# Description:       Manages the weewx weather system
### END INIT INFO

to this

### BEGIN INIT INFO
# Provides:          weewx
# Required-Start:    $local_fs $remote_fs $syslog $networking $time
# Required-Stop:     $local_fs $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: weewx weather system
# Description:       Manages the weewx weather system
### END INIT INFO

This says that the system should not run WeeWX until the network is up. What I'm not sure of is how Debian defines "up." Worth a try.

2. The clock must also be set. Either use a hardware real time clock (RTC), or remove the "fake clock" by following these directions. 

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[David Hindley]

Thanks for your reply, Tom - much appreciated.

I have removed the fake clock (pending purchase of a RTC) and made the change to the file in init.d and restarted weewx with a working network and all seems OK, but I guess I won't know fully until the Pi crashes again.

David.

[Thomas Keffer]

Pull its power supply,  then plug it back in. That's as good a test as any.

[G400]

To enable INIT settings, should I remove # sign at beginning of each line?

[Thomas Keffer]

No. Although they start with '#', and, therefore, look like comment lines, they are actually part of a "magic" block interpreted by init.

All of this has been superseded by systemd, but it still works in a backwards compatibility mode.