Future date/times into database resulting in loss of data

[Ken Waters]

Hello,

Sorry if this seems long but I tried to put everything in here that might help explain the situation.

Here is my setup:

  Weewx version: 3.9.2

  Weather station: Davis Vantage Pro 2 connected thru DB9 serial port

  Hardware: HP Pro 6000 SFF dual Pentium RAM: 4 GB, WD 1 TB drive (currently only 2% usage)

  O/S: Ubuntu 18.04.4 LTS

  Website: http://satwatcher.us/satwatcher.us/wxstation/

I've been running WeeWx for about 9-10 years and previously used WxView.

Here is the problem:

Periodically and randomly the SQLite database gets corrupted with the most recent update showing a future day/time. There seems to be no obvious trigger that causes this and it's been happening to me for several years including with other hardware setups. When this happens it basically "locks" up the database so that current observations can no longer be stored and results in a loss of data after the database corruption. In the past I have had to resort to: 

  a) stopping the service

  b) clearing the memory (sudo /usr/share/weewx/wee_device -y --clear-memory)

  c) renaming the .sdb

  d) restarting the service

in effect, starting with a new database.  This has been happening every 2-4 weeks randomly and not as a result of anything done by me to the server.

Here is a sample snippet from syslog that shows the issue:

  Apr  9 07:38:15 WeewxWeatherServer weewx[28253]: vantage: Getting archive packets since 2020-04-09 07:37:00 MST (1586443020)

  Apr  9 07:38:15 WeewxWeatherServer weewx[28253]: vantage: Gentle wake up of console successful

  Apr  9 07:38:15 WeewxWeatherServer weewx[28253]: vantage: Retrieving 1 page(s); starting index= 0

  Apr  9 07:38:16 WeewxWeatherServer weewx[28253]: manager: Added record 2020-04-22 23:06:00 MST (1587621960) to database 'weewx.sdb'

  Apr  9 07:38:16 WeewxWeatherServer weewx[28253]: manager: Added record 2020-04-22 23:06:00 MST (1587621960) to daily summary in 'weewx.sdb'

  Apr  9 07:38:16 WeewxWeatherServer weewx[28253]: vantage: DMPAFT complete: page timestamp 2020-04-07 12:59:00 MST (1586289540) less than final timestamp 2020-04-22 23:06:00 MST (1587621960)

  Apr  9 07:38:16 WeewxWeatherServer weewx[28253]: vantage: Catch up complete.

  Apr  9 07:38:16 WeewxWeatherServer weewx[28253]: vantage: Requesting 200 LOOP packets.

  Apr  9 07:38:16 WeewxWeatherServer weewx[28253]: reportengine: Running reports for latest time in the database.

  Apr  9 07:38:16 WeewxWeatherServer weewx[28253]: reportengine: Running report 'SeasonsReport'

  Apr  9 07:38:16 WeewxWeatherServer weewx[28253]: reportengine: Found configuration file /etc/weewx/skins/Seasons/skin.conf for report 'SeasonsReport'

Here is output captured this morning from a sqlite3 select command that shows the erroneous timestamp in the database:

  $ sqlite3 /var/lib/weewx/weewx.sdb "select datetime(dateTime,'unixepoch','localtime') from archive where dateTime>1550000;"

  2020-04-09 07:30:00 

  2020-04-09 07:31:00 

  2020-04-09 07:32:00 

  2020-04-09 07:33:00 

  2020-04-09 07:34:00 

  2020-04-09 07:35:00 

  2020-04-09 07:36:00 

  2020-04-09 07:37:00 

  2020-04-22 23:06:00 <<<---- erroneous future entry

These date/times are local MST (Z-7) and the problem actually occurred at 2306 MST last night resulting in loss of data for about 8 hours.  It seems as if WeeWx is skipping recording the observation because it thinks this is an old observation due to the last entry being "newer".

Problem #2:

In addition, after stopping, clearing, removing the .sdb, and restarting I now see that the graphics being produced are not current and are time-stamped 4/22/2029 23:06 although the data on the left-side of the page at http://satwatcher.us/satwatcher.us/wxstation/ are current.  Why are the graphics still keying off the erroneous time in the future?

This has been driving me nuts for several years now.  Does anybody have any idea what may be causing this and how to fix it so it no longer happens?

Thank you ever so much!

(ps...I did post something to the list back in 2015 about this but it was not resolved at the time)

Ken

[Thomas Keffer]

Hi, Ken

Problem #1 is almost surely a hardware problem. Have you put ferrite coils around the cable from your logger to the computer? A UPS also helps. If you are still getting random corruption in the logger memory, then the logger may be defective.

Problem #2: a graphic is regenerated only as often as its aggregation interval. For example, if a month plot uses a 3 hour aggregation interval, then it will be regenerated only after it is 3 hours old. To force regeneration, delete the graphic.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2e5a14d8-a48d-47be-96b4-69625aa65d54%40googlegroups.com.

[Ken Waters]

Hi Tom,

Thanks for responding.

Problem #2 resolved, that was easy.  :-)

For Problem #1, I don't have ferrite coils available but will look for them and try this out.  I note the cable is just simple unshielded telephone phone cable. Perhaps I could try something more shielded, if something like that exists for phone cables.

Ken

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEBP8RSp-b5%3DQk%2BGsu9nkjg7GBcp%3Dfv5DN4Q3SR009k42g%40mail.gmail.com.