Davis Datalogger bug

[Andrea Cecilia]

Hi there,

I'm working with lot of Davis stations connected to weewx, and I noticed that there exists a frequent bug which I'm trying to describe now:

sometimes it is like the datalogger gest freezed, so that tweewx cannot download data from it. The only way I tried which is working is banally giving a

/home/weewx/bin/wee_device --clear-memory

but in this way I lose the data inside the datalogger.

In syslog I can see a string like

page timestamp less than final timestamp

which is what I think the problem is. Is there a way to solve this without clearing the data inside the datalogger?

This must be a bug of weewx, because with Weatherlink I never had a problem like this.

Thank you

[Andrew Milner]

attaching the log not just a line may let people offer more advice.

[Andrea Cecilia]

here it is. As you can see, there is no "added record [...] to weewx.sdb" line, which indicates that it is not downloading data from the datalogger, and the reson (I think) is explained in the highlighted line.

Oct 16 10:30:14 raspberrypi weewx-vp2[511]: vantage: Getting archive packets since 2019-10-16 09:05:00 CEST (1571209500)
Oct 16 10:30:14 raspberrypi weewx-vp2[511]: vantage: Gentle wake up of console successful
Oct 16 10:30:15 raspberrypi weewx-vp2[511]: vantage: Retrieving 5 page(s); starting index= 4
Oct 16 10:30:15 raspberrypi weewx-vp2[511]: vantage: DMPAFT complete: page timestamp 2019-10-15 13:20:00 CEST (1571138400) less than final timestamp 2019-10-16 09:05:00 CEST (1571209500)
Oct 16 10:30:15 raspberrypi weewx-vp2[511]: vantage: Catch up complete.
Oct 16 10:30:15 raspberrypi weewx-vp2[511]: reportengine: Running reports for latest time in the database.
Oct 16 10:30:15 raspberrypi weewx-vp2[511]: vantage: Requesting 200 LOOP packets.
Oct 16 10:30:15 raspberrypi weewx-vp2[511]: reportengine: Running report 'StandardReport'
Oct 16 10:30:15 raspberrypi weewx-vp2[511]: vantage: Gentle wake up of console successful
Oct 16 10:30:15 raspberrypi weewx-vp2[511]: reportengine: Found configuration file /home/weewx/skins/Standard/skin.conf for report 'StandardReport'
Oct 16 10:30:15 raspberrypi weewx-vp2[511]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.ch
eetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras']
Oct 16 10:30:15 raspberrypi weewx-vp2[511]: manager: Daily summary version is 2.0
Oct 16 10:30:17 raspberrypi weewx-vp2[511]: cheetahgenerator: Generated 16 files for report StandardReport in 2.31 seconds
Oct 16 10:30:17 raspberrypi weewx-vp2[511]: manager: Daily summary version is 2.0
Oct 16 10:30:18 raspberrypi weewx-vp2[511]: imagegenerator: Generated 15 images for StandardReport in 1.36 seconds
Oct 16 10:30:18 raspberrypi weewx-vp2[511]: copygenerator: copied 0 files to /home/weewx/public_html/vp2
Oct 16 10:30:18 raspberrypi weewx-vp2[511]: reportengine: Running report 'FTP'
Oct 16 10:30:19 raspberrypi weewx-vp2[511]: reportengine: Found configuration file /home/weewx/skins/Ftp/skin.conf for report 'FTP'
Oct 16 10:30:19 raspberrypi weewx-vp2[511]: ftpupload: Attempting connection to ftp.meteoregionelazio.it
Oct 16 10:30:19 raspberrypi weewx-vp2[511]: ftpupload: Connected to ftp.meteoregionelazio.it
Oct 16 10:30:19 raspberrypi weewx-vp2[511]: ftpupload: Uploaded file /www.meteoregionelazio.it/stazioni/soriano/weewx/vp2/daypond.png

[Andrew Milner]

what came just before the log you posted?  why is weewx attempting to do a catchup? had weewx just been started?  had communications been lost for some reason? We need to know what is causing weewx to attempt a catchup in the first place.  You can never post too much log!!!!

[Thomas Keffer]

This is likely a case of memory corruption, brought on by time not being synced during the reboot, due to the lack of an on-board clock. It was not a problem with Weatherlink because you were not running it on an RPi.

See the section WeeWX generated HTML pages, but does not update them in the User's Guide.

But, as Andrew suggested, we would have to see the log from startup to be sure.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/33dfe43a-f6d8-4131-b7d3-b5d4e8415b4d%40googlegroups.com.

[David VE3STI]

I had somewhat similar issues - WeeWX couldn't read from the Vantage Vue after boot up. I needed to do the "memory clear" command then all was well. However, it was a giant pain if I was away/busy/etc and not available to run the command. I thought that it should be possible to "fix" this issue in software but my feeble efforts failed. I installed a RTC. That didn't fix it either so I gave up and now use a UPS to run the RPi. As long as my RPi never reboots, I'm OK.

I don't have much to contribute but I'll watch from the sidelines. I'd love to see where this leads.

David Beach

On Wednesday, 16 October 2019 07:40:59 UTC-4, Thomas Keffer wrote:

This is likely a case of memory corruption, brought on by time not being synced during the reboot, due to the lack of an on-board clock. It was not a problem with Weatherlink because you were not running it on an RPi.

See the section WeeWX generated HTML pages, but does not update them in the User's Guide.

But, as Andrew suggested, we would have to see the log from startup to be sure.

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Thomas Keffer]

This has been my solution as well: a real-time clock, plus a UPS.

But, if the power outage is severe enough to drain the UPS battery, my setup still recovers. At least, the last time that happened (about a year ago).

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/32bb9d0c-63eb-4610-a49f-7043b5f70375%40googlegroups.com.

[gjr80]

One other thing to keep in mind; the de facto standard mode of operation of a Davis station under WeeWX is to use hardware record generation, in other words WeeWX obtains and stores archive records from from the console/logger. WeeWX can also operate using software record generation where only loop packets are obtained from the logger/console and archive records are synthesised by WeeWX from this data. When the logger experiences corrupted memory/temporal displacement it almost always continues to emit loop packets but emits old archive records. So when operated in software record generation mode WeeWX will essentially be immune from the corrupted station memory issue. Similarly, other PWS software that only use loop packets will not be affected by the corrupted station memory issue.

I don't see this as a WeeWX bug, rather a limitation of operating in hardware record generation mode that can be mitigated by reliable power and clock.

Gary

[Thomas Keffer]

Immune, yes, but you also won't get any archive records stored on the logger while your computer is down!

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/e4475612-bb76-4fd2-8e53-dc00286f878d%40googlegroups.com.

[gjr80]

Yes, the price you pay for using software record generation. Also, I forgot about no_catchup being introduced in 3.9.1. That changes the logic during a startup catchup, now software record generation will also get caught up in the 'timestamp less than final' issue unless you set no_catchup = False. But then, as you say, you will miss out on any catchup records in timea when you do not have corrupt station memory.

Guess that's why I have a UPS as well :)

Gary

On Thursday, 17 October 2019 09:35:21 UTC+10, Thomas Keffer wrote:

Immune, yes, but you also won't get any archive records stored on the logger while your computer is down!

On Wed, Oct 16, 2019 at 4:33 PM gjr80 <gjrod...@gmail.com> wrote:

One other thing to keep in mind; the de facto standard mode of operation of a Davis station under WeeWX is to use hardware record generation, in other words WeeWX obtains and stores archive records from from the console/logger. WeeWX can also operate using software record generation where only loop packets are obtained from the logger/console and archive records are synthesised by WeeWX from this data. When the logger experiences corrupted memory/temporal displacement it almost always continues to emit loop packets but emits old archive records. So when operated in software record generation mode WeeWX will essentially be immune from the corrupted station memory issue. Similarly, other PWS software that only use loop packets will not be affected by the corrupted station memory issue.

I don't see this as a WeeWX bug, rather a limitation of operating in hardware record generation mode that can be mitigated by reliable power and clock.

Gary

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Thomas Keffer]

If you care about data integrity, the $50 for a modest UPS is money well spent!

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d91c9ec6-6c02-481d-ac73-54675c772b20%40googlegroups.com.

[Andrea Cecilia]

I've never seen this section of the User's Guide (it might be relatively new), but it is exactly about the problem we are discussing here.

So, the fixing tries they suggest without data loss are two:

1) unplug and reboot the console by removing current and batteries for about 2 minutes and then plug it again;

2) make a /home/weewx/bin/wee_device --dump.

They also specificate that if these won't work, it is necessary to give a --clera-memory as I did, but in this case you lose the data.

As it is suggested in other answers here, it would "fix" the problem buying an UPS, but I honestly wolud find a software solution, not coming around the problem in this way (which could also be expensive).

So, it remains to try the 2 points up here when the problem will come out again. Did anyone try?

Il giorno mercoledì 16 ottobre 2019 13:40:59 UTC+2, Thomas Keffer ha scritto:

This is likely a case of memory corruption, brought on by time not being synced during the reboot, due to the lack of an on-board clock. It was not a problem with Weatherlink because you were not running it on an RPi.

See the section WeeWX generated HTML pages, but does not update them in the User's Guide.

But, as Andrew suggested, we would have to see the log from startup to be sure.

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Andrew Milner]

regardless I would recommend that all Rpis controlling weather stations have an RTC installed to avoid temporal issues as much as possible.

[Thomas Keffer]

As it is suggested in other answers here, it would "fix" the problem buying an UPS, but I honestly wolud find a software solution, not coming around the problem in this way (which could also be expensive). 

There is a software solution: run with record_generation = software. This essentially bypasses the logger and its tendency to corrupt memory. But, it also means you won't enjoy the benefits of the logger and its offline logging capabilities.

If $5 is too much for a clock, at least be sure to read the section in the Wiki on removing the "fake clock": https://github.com/weewx/weewx/wiki/Raspberry-Pi#b-remove-the-fake-clock

 

[Andrea Cecilia]

The problem of this solution is that if a power outage happens, the RPi ceases acquiring data, and these data are going to be lost.

Consider the specific case in which I'm having most troubles: I installed a Davis station in my farmland house, where I usually go once a month, and where power outages may happen at every thunderstorm. I will lost all the data coming during power outages, and they are lot of data!

[David Beach]

I will have to review that section of the User's Guide. I caved and began using a UPS quite some time ago - but I think it is going to need a new battery soon. At this time, I'm some 1,000 km from home so no fiddling with WeeWX for a while.

When I'm home, I am going to try Thomas Keffer's suggestion to run with "record_generation = software" because I currently don't care about missing some data during a power failure. And at this point, if I do have to restart for any reason, I have to clear the memory anyway because my RTC didn't seem to fix the issue.

David

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/hOj_mdhkl1E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/e827069b-d024-43ac-98fe-e81b2875473a%40googlegroups.com.

[Andrea Cecilia]

I'm sorry, but I didn't understand in which way the clock is related to this problem

[Thomas Keffer]

Yes, that's the problem. 

You cannot expect high-quality data without investing in the hardware. You bought an expensive Davis station, why not spend the other $50 to protect the data?

I have a Davis Envoy, a UPS, and a high-quality embedded PC with a clock. It has run continuously without a single failure for over 11 years. It can be done at modest cost!

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/e39000fe-4ff5-4d9f-b830-0c7b7e9e3811%40googlegroups.com.

[vince]

On Thursday, October 17, 2019 at 5:41:32 AM UTC-7, Thomas Keffer wrote:

Yes, that's the problem.

You cannot expect high-quality data without investing in the hardware. You bought an expensive Davis station, why not spend the other $50 to protect the data?

Or alternately sell all your raspberry pi gear and pick up a good used computer that 'has' a RTC in it (even a 4 year old mini pc would work) so you get correct time through your periodic power outages.

In short, buy a real computer.   The pi are great (I have 'many') but you are limited in what they can do without adding things to them.  You have to remove those limitations by either adding a realtime clock, or a ups (or both), or buying a more capable computer if you want enterprise-level data availability.

The other thing to think about is 'does your computer power up when power is restored'.  That's not always goodness.  I've fried a lot of motherboards over the years in places where the power is very bouncy up and down as power is restored to that location.   UPS is almost certainly the short term answer, plus a RTC if you want to stick with a pi.

[David Beach]

This is veering into the tangential but...

My weather logging needs are modest - I'm usually looking for the weather 'at the moment' or for the last several days. Occasionally, rainfall for the month. My livelihood does not depend on my weather observations. If I miss weather data during a power outage, I'm not too distressed. Not missing it would be better but missing it is not a show-stopper for me.

I have used the RPi because it is a power miser, it is small and it is cheap enough to be bought new and 'dedicated' to a single purpose. My RPi Zero W uses between 50 and 100 mA at 5V. I chose WeeWX because it was free to play with and use, appeared well supported and was written in Python, a language I wanted to become more familiar with.

My 'getting stuck on restarting' issues were annoying, not only because I missed data but because I had to restart things manually, remember (or look up again!) the command to clear the memory, etc. If it quit while I was away from home, it was 'dead' until I got back. The RTC (a cheap DS3231 module which required some desoldering/re-soldering to squeeze into the standard RPi Zero case) didn't do the trick. A few feeble attempts on my part at some coding didn't eliminate the issue either. I resorted to an APS UPS that is many times the size of the RPi and likely uses more power to maintain its battery than the RPi does to run.

I don't know enough about the WeeWX code, the Davis firmware or the logging module to understand the technical details of this problem. The problem may be 'baked into' the Davis hardware/firmware. The WeeWX code may be trying to fix/avoid a problem it didn't create. However, if anyone ever does figure out a software solution to this thorny problem, I'll be happy to get rid of my giant UPS. And my hat will be off to you. (But I'll keep using my RTC - it is now soldered on to my RPi Zero!)

David

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/hOj_mdhkl1E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2674140d-f230-4451-a8b2-8961f3819a14%40googlegroups.com.

[vince]

On Thursday, October 17, 2019 at 11:15:00 AM UTC-7, David VE3STI wrote:

I have used the RPi because it is a power miser, it is small and it is cheap enough to be bought new and 'dedicated' to a single purpose.

Yup.  Most pi users do the same, as do people with similar tiny/low-power non-RTC systems (like me - I run on a Seagate Dockstar, which is essentially a stripped down PogoPlug).

There have been MANY discussions about weewx/pi/etc. and databases needing accurate time here in the past.  Almost too many to recount.

But the short 'software' solution is:

• disable anything that tries to keep track of the software time either periodically, or when you do an orderly shutdown
• run ntpd or equivalent in your boot sequence if your system will be connected to an Internet time source
• 'test' your system's powerout behavior.   Every computer comes up at a predictable date+time when it is powered on, if you do not have a RTC.   See what that value is.

Note: systemd (grrrrrr) 'also' seems to have something like the user-space 'fake-hwclock' that is also in debian(ish) operating systems.  You need to disable systemd helping too much (grrrrrr!).

Again, do a few controlled tests.  See what time is.  Yank the power.  See what date+time it boots to.   Reboot the box.  See what it does 'then' re: date+time.   Basically baseline your system.

Weewx shouldn't start up if the date+time of weewx.conf is newer than the system clock, if I remember the code correctly, so you should be safe from reboots.

FWIW, I do 'not' have a RTC on my Dockstar nor do I have a UPS for several reasons.  I 'do' have the battery in the Davis VP2 console however, so theoretically I should be able to run the console on battery there for a long long time.  I only had one power out for more than a couple minutes in the last 10 years I've had this in place, we were down for about 2 days, but the datalogger stayed ok and weewx recovered just fine for me back then when power came back up.

Of course your mileage might vary.  RTC is good.  RTC+UPS is better.  All comes down to your budget etc.

[Greg Troxel]

vince <vince...@gmail.com> writes:

> - 'test' your system's powerout behavior. Every computer comes up at a

> predictable date+time when it is powered on, if you do not have a RTC.
> See what that value is.

On BSD, the default behavior on poweron is more or less to use time from
the root superblock, so this predictable/very-past notion doesn't hold.
(I can believe that what you say is true on Linux systems.)

What I've done is to have the script that starts weewx loop until ntpd
reports that it is well synced to remote peers.

And yes, I know I should put in an RTC...