Where do I start?

[Robin]

I installed weewx as a newbie, 12 years ago. Since then it has run without any issues. Recently, things have changed. I have forgotten the stuff that I learned to get weewx up and running and my brain is less capable of learning new stuff. Please can somebody point me in the right direction.

Hardware is a Raspberry Pi Model B (512MB). 

The last update I did to any software on the Pi was to upgrade to V3.8.0.

Syslog give me lots of stuff I don't recognise or understand.

May 23 09:01:01 MWS avahi-daemon[208]: New relevant interface eth0.IPv4 for mDNS.
May 23 09:01:01 MWS avahi-daemon[208]: Registering new address record for 192.168.1.25 on eth0.IPv4.
May 23 09:01:01 MWS dhcpcd[371]: eth0: adding route to 192.168.0.0/22
May 23 09:01:01 MWS dhcpcd[371]: eth0: adding default route via 192.168.1.1
May 23 09:01:11 MWS weewx[1099]: cheetahgenerator: Generated 22 files for report StandardReport in 53.11 seconds
May 23 09:01:15 MWS systemd[1]: Started Session c6 of user root.
May 23 09:01:44 MWS weewx[1099]: imagegenerator: Generated 31 images for StandardReport in 32.29 seconds
May 23 09:01:44 MWS weewx[1099]: copygenerator: copied 13 files to /var/www/weewx
May 23 09:01:52 MWS weewx[1099]: historygenerator.pyc: Generated 8 tables in 7.69 seconds
May 23 09:01:55 MWS weewx[1099]: cheetahgenerator: Generated 2 files for report HTMLPages in 10.73 seconds
May 23 09:01:55 MWS weewx[1099]: copygenerator: copied 3 files to /var/www/weewx/Bootstrap
May 23 09:01:57 MWS dhcpcd[371]: eth0: carrier lost
May 23 09:01:57 MWS kernel: [  619.007806] smsc95xx 1-1.1:1.0 eth0: link down
May 23 09:01:57 MWS dhcpcd[371]: eth0: deleting default route via 192.168.1.1
May 23 09:01:57 MWS dhcpcd[371]: eth0: deleting route to 192.168.0.0/22
May 23 09:01:57 MWS avahi-daemon[208]: Withdrawing address record for 192.168.1.25 on eth0.
May 23 09:01:57 MWS avahi-daemon[208]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.168.1.25.
May 23 09:01:57 MWS avahi-daemon[208]: Interface eth0.IPv4 no longer relevant for mDNS.
May 23 09:01:58 MWS kernel: [  620.679675] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
May 23 09:01:58 MWS dhcpcd[371]: eth0: carrier acquired
May 23 09:01:58 MWS dhcpcd[371]: eth0: rebinding lease of 192.168.1.25
May 23 09:01:58 MWS dhcpcd[371]: eth0: probing address 192.168.1.25/22
May 23 09:02:00 MWS weewx[1099]: rsyncupload: [['rsync', '--archive', '--stats', '-e ssh', '/var/www/weewx/', ##################.webspace-data.io:/kunden/homepages/###########/htdocs/weather']] reported errors: ssh: Could not resolve hostname ############webspace-data.io: Temporary failure in name resolution. rsync: connection unexpectedly closed (0 bytes received so far) [sender]. rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.2]
May 23 09:02:00 MWS weewx[1099]: rsyncupload: rsync executed in 5.15 seconds
May 23 09:02:03 MWS dhcpcd[371]: eth0: leased 192.168.1.25 for 7200 seconds
May 23 09:02:03 MWS dhcpcd[371]: eth0: adding route to 192.168.0.0/22
May 23 09:02:03 MWS dhcpcd[371]: eth0: adding default route via 192.168.1.1
May 23 09:02:03 MWS avahi-daemon[208]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.168.1.25.
May 23 09:02:03 MWS avahi-daemon[208]: New relevant interface eth0.IPv4 for mDNS.
May 23 09:02:03 MWS avahi-daemon[208]: Registering new address record for 192.168.1.25 on eth0.IPv4.

Please make an old man happy. Any help would be great, Where do I start?

Robin

[John Smith]

ssh: Could not resolve hostname ############webspace-data.io: 

Seems to be a DNS issue, as I got nothing back from the name servers

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-user/999b4352-63da-4dd7-b019-dfa11aa4a1b5n%40googlegroups.com.

[Robin]

Hi John,

I removed part of the server name as it is hidden by my host for security reasons. The exact same configuration ran perfectly before and ,since I asked for help, is running perfectly again.

I am wondering if I have a failing hardware issue with either the rpi or an issue with my router. Is it possible that avahi-daemon and dhcpcd entries point to local network problems?

[Vince Skahan]

You didn't say whether you had updated the os or weewx (nor to what versions) but regardless if it works now call it good.

The old modelB pi could occasionally get a bit unstable occasionally on the network side. Your log says it temporarily lost connection to whatever you have it plugged into but it recovered ok. I wouldn't worry it too much.

[John Smith]

I removed part of the server name as it is hidden by my host for security reasons.

Security by obscurity isn't security. Even the domain alone is a good place to start.

Then you have Shodan which lets people search for open ports etc.

I am wondering if I have a failing hardware issue with either the rpi or an issue with my router.

Bit hard to tell remotely, but unlikely given there was a ssh timeout, it would be more likely to be something misconfigured.

Is it possible that avahi-daemon and dhcpcd entries point to local network problems?

You shouldn't use DHCP for servers as everything goes down in a heap when the server does, same with Avahi, it's just a really bad idea for servers, however neither of those would cause connection timeouts for remote hosts.

First thing to do is try doing a ping to the host name from the system running weeWX, if that fails try doing a traceroute to try and figure out where it's failing.

If either of those work trying telnet-ing to port 22 to see if you can get a basic connection to the ssh port.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/f1743398-54cb-4c90-a3b9-f45ce46e8abfn%40googlegroups.com.

[Robin]

Vince, thank you for your reply. The os has not been updated since I installed weewx Versio 3.8.0 which is still the version in use. The system worked perfectly for 10 years but has now become very unstable. Sometimes I get a week without issues. Yesterday I had to shutdown and restart on 3 seperate occasions. Hence my "where do I start" question. I tried a new SD card but this has not resolved the instability.

My questions remain the same, where (how) should I start to try and isolate the issue.

[Robin]

John, I think I have not explained my problem correctly. My standard weewx install of v3.8.0 has been running without issues for many years. Everything that is reported in the syslog is there because it is part of the standard weewx install. The error you have focused on is the rpi trying to rsync to my internet host. I am convinced (although I might be wrong) that this is a symptom of  the instabilty and not the cause of the instability. When the system is stable rsync works perfectly. I posted the syslog as I hoped it might point towards the cause of the instability, which is new.

I am trying to track down the source of the instability. A new SD card changed nothing. So where do I look next. As mentioned above, I have done no OS updates since I installed V3.8.0 which was before July 2019.

[Vince Skahan]

You need to tell us what device your pi seems to be plugged into, what station you have, and how the station is connected to weewx.

If your pi loses carrier either the pi or whatever it is plugged into is getting unstable and the pi is disconnected from your home network just as if you unplugged it. It is not obvious which device is the cause.

Possible solutions are (a) replace the pi with a pi3 pi4 or pi5 or another linux box or (b) replace or reboot whatever you are plugged into.

I have seen modelB and zeroW eventually get too unstable to use but that was wifi not ethernet. It is also possible you have bad power to the pi or maybe too many things attached to the pi usb bus.  The modelB architecture somewhat shared things under the hood for usb and ethernet so it was possible to overload the system.

Those issues went away when the pi3 came out as far as I ever saw. I sold/junked all my modelB years ago.

I would start with (b) above. Power down and unplug whatever your pi plugs into. Wait 30 seconds or so. Plug it back in and power it back up.  See if that helps.

This all assumes you have changed nothing physically of course.  If you 'have' touched things physically then you might try unplugging/plugging the ethernet cables in again on both sides making sure they click securely into place.

But I'd power reset whatever your pi is plugged into as the first step....

[Robin]

Vince,

Thank you. 

The pi is connected to a Davis Vantage Pro Plus via a Data Logger so there is no power drain and directly to a TP Link Deco X50.

I am powering the pi with a WaveShare Uninterruptible Power Supply.

I've tried shutdowns, unplugging everything, waiting while I drink my tea then reconnecting everything and switching back on. Things seems more stable for a while, but the instability remains.

Regards

Robin

[John Smith]

I've tried shutdowns, unplugging everything, waiting while I drink my tea then reconnecting everything and switching back on. Things seems more stable for a while, but the instability remains.

In that case it might be heat related, does it become more unstable if it's under sustained load? 

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/028dd4fd-5ce7-4aae-a291-22cf2be5989dn%40googlegroups.com.

[Robin]

The pi is only running weewx and capturing an image from our webcam. The load will be at it's highest when processing the image. I would have said that it does this without problem, but now I'm lost as to where to look for answers.

I'm running out of things to try. Perhaps my spare rpi4 needs to be put into service. Its 10 years plus since I last did an install and then added all my bits and bobs, so it might take me a while to relearn everything.

If anyone is interested, my site is at http://www.weather.molyvos.eu/

Thank you for your thoughts, wish me luck!

Robin

[John Smith]

The pi is only running weewx and capturing an image from our webcam. The load will be at it's highest when processing the image.

weeWX itself chews CPU time when processing reports too as it has to do sqlite/database queries etc so if the CPU heats up doing all that and then fails the next step because of it...

To find out for sure you'd have to manually trigger rsync when it's been idle a while and cooled down.

Have you tried to traceroute from the system running weewx to the server? and everything else I suggeted?

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/c5892c9c-13be-4775-ae6b-2bd858b371e5n%40googlegroups.com.

[Robin]

I've just checked the CPU temperature and it was between 50.8 and 52.5 for the time I was wtching.  vcgencmd get_throttled  also reported 0x0.

[John Smith]

What about traceroute to the server from your rpi?

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/c4fca004-0528-4359-9041-cd64c27f5810n%40googlegroups.com.

[Vince Skahan]

If your pi is dropping carrier you have a physical problem and traceroute will not help. You can’t trace a route if your pi is disconnected. The open question is why it is dropping carrier.

If you can plug the pi4 in as a second computer and see if ‘that’ is also unstable that might help determine if your pi or the router is the culprit. Having a second pi online would also help if you need to migrate to a new pi.

I do like the ‘on this date’ page but perhaps that is causing some load to calculate. You could alternately disable that and the camera stuff to get to less load on the pi. Basically nibble away at it one thing at a time.

Definitely save your current setup offline just in case….

[John Smith]

If your pi is dropping carrier you have a physical problem and traceroute will not help.

Traceroute will indicate where the problem is, even on the LAN itself. 

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/ebba534e-75a8-4763-b841-796796a0e8ban%40googlegroups.com.