Data Issue

[Dave McCreath]

Good Evening All, please excuse this Noob!

In a bit of a pickle, I've got a Vantage Vue that has been working fine over the past two years, data ported through to WU via a Rpi (using the Debian Stretch).  Weewx is up to date (v 3.8.0).

Had a power cut this evening for a couple of hours, when the power came back on weeWx pulled a record from the Vue and published it (the record data was erroneous), after this, nothing.

Having recently upgraded to Stretch from Jessie I then switched back to Jessie (changed SD card) and in the syslog I saw that weeWx pulled and published two day's worth of records from the Vue (switched to Stretch two days ago) up until the time of the erroneous record (that was generated just after the power came back up).

I tried resetting the Vue (logger issue?) to no avail, same issue occurred. So switched back to Stretch and the issue was still there.

Went into the weeWx.conf file and changed the record_generation from 'hardware' to 'software' (archive service generation) and now all is working fine!

Does anyone have an idea of what's gone wrong and what are the implications of switching from hardware to software in the longer term?

Many thanks

Dave

[gjr80]

Hi,

Probably corrupted station memory from the power cut but impossible to say without seeing the logs. Software record generation works as it only uses the loop data from your console which is unaffected by corrupt station memory. On the other hand, hardware generation relies on archive records from your station and they are affected by corrupt station memory.

I would go back to hardware record generation, stop weeWX and clear the station memory using wee_device as per the 'Corrupt station memory' section in the User's Guide (http://weewx.com/docs/usersguide.htm#html_generated_but_not_updated). Note weeWX needs to be stopped before running wee_device. Start weeWX and monitor your logs. If all is not working post a log extract from when you just started weeWX including at least a couple of report cycles (archive intervals) of output. Don't omit anything from the logs.

Gary

[Dave McCreath]

Gary

Many, many thanks for getting back and pointing me in the right direction.

Done as you suggested, stopped weeWx, changed config back to hardware, dumped (for clarity), then cleared the station memory and then restarted weeWx.

All seems to now be OK but I've attached the log below just in case from just as the dump was finishing:

Jan 20 08:56:46 raspberrypi wee_device[2124]: manager: Unable to add record 2018-01-18 17:45:00 GMT (1516297500) to database 'weewx.sdb': attempt to write a readonly database

Jan 20 08:56:46 raspberrypi wee_device[2124]: manager: Unable to add record 2018-01-18 17:50:00 GMT (1516297800) to database 'weewx.sdb': attempt to write a readonly database

Jan 20 08:56:46 raspberrypi wee_device[2124]: manager: Unable to add record 2018-01-18 17:55:00 GMT (1516298100) to database 'weewx.sdb': attempt to write a readonly database

Jan 20 08:56:46 raspberrypi wee_device[2124]: manager: Unable to add record 2018-01-18 18:00:00 GMT (1516298400) to database 'weewx.sdb': attempt to write a readonly database

Jan 20 08:56:46 raspberrypi wee_device[2124]: manager: Unable to add record 2018-01-18 18:05:00 GMT (1516298700) to database 'weewx.sdb': attempt to write a readonly database

Jan 20 08:56:46 raspberrypi wee_device[2124]: manager: Unable to add record 2018-01-18 18:10:00 GMT (1516299000) to database 'weewx.sdb': attempt to write a readonly database

Jan 20 08:56:46 raspberrypi wee_device[2124]: manager: Unable to add record 2018-01-18 18:15:00 GMT (1516299300) to database 'weewx.sdb': attempt to write a readonly database

Jan 20 08:56:46 raspberrypi wee_device[2124]: manager: Unable to add record 2018-01-18 18:20:00 GMT (1516299600) to database 'weewx.sdb': attempt to write a readonly database

Jan 20 08:58:06 raspberrypi wee_device[2355]: vantage: Archive memory cleared.

Jan 20 08:59:50 raspberrypi systemd[1]: Starting LSB: weewx weather system...

Jan 20 08:59:50 raspberrypi weewx[2513]: engine: Initializing weewx version 3.8.0

Jan 20 08:59:50 raspberrypi weewx[2513]: engine: Using Python 2.7.13 (default, Nov 24 2017, 17:33:09) #012[GCC 6.3.0 20170516]

Jan 20 08:59:50 raspberrypi weewx[2513]: engine: Platform Linux-4.1.19-v7+-armv7l-with-debian-9.3

Jan 20 08:59:50 raspberrypi weewx[2513]: engine: Locale is 'en_GB.UTF-8'

Jan 20 08:59:50 raspberrypi weewx[2513]: engine: pid file is /var/run/weewx.pid

Jan 20 08:59:50 raspberrypi weewx[2517]: engine: Using configuration file /etc/weewx/weewx.conf

Jan 20 08:59:50 raspberrypi weewx[2517]: engine: debug is 1

Jan 20 08:59:50 raspberrypi weewx[2517]: engine: Initializing engine

Jan 20 08:59:50 raspberrypi weewx[2517]: engine: Loading station type Vantage (weewx.drivers.vantage)

Jan 20 08:59:50 raspberrypi weewx[2501]: Starting weewx weather system: weewx.

Jan 20 08:59:50 raspberrypi systemd[1]: Started LSB: weewx weather system.

Jan 20 08:59:50 raspberrypi weewx[2517]: vantage: Driver version is 3.0.10

Jan 20 08:59:51 raspberrypi weewx[2517]: vantage: Opened up serial port /dev/ttyUSB0; baud 19200; timeout 4.00

Jan 20 08:59:51 raspberrypi weewx[2517]: vantage: Gentle wake up of console successful

Jan 20 08:59:51 raspberrypi weewx[2517]: vantage: Hardware type is 17

Jan 20 08:59:51 raspberrypi weewx[2517]: vantage: Hardware name: Vantage Vue

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.engine.StdTimeSynch

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.engine.StdTimeSynch

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.engine.StdConvert

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: StdConvert target unit is 0x1

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.engine.StdConvert

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.engine.StdCalibrate

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.engine.StdCalibrate

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.engine.StdQC

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.engine.StdQC

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.wxservices.StdWXCalculate

Jan 20 08:59:51 raspberrypi weewx[2517]: wxcalculate: The following values will be calculated: barometer=prefer_hardware, windchill=prefer_hardware, dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, windrun=prefer_hardware, heatindex=prefer_hardware, maxSolarRad=prefer_hardware, humidex=prefer_hardware, pressure=prefer_hardware, inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, cloudbase=prefer_hardware

Jan 20 08:59:51 raspberrypi weewx[2517]: wxcalculate: The following algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.wxservices.StdWXCalculate

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.engine.StdArchive

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Archive will use data binding wx_binding

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Record generation will be attempted in 'hardware'

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Using archive interval of 300 seconds (specified by hardware)

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Use LOOP data in hi/low calculations: 1

Jan 20 08:59:51 raspberrypi weewx[2517]: manager: Daily summary version is 2.0

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Using binding 'wx_binding' to database 'weewx.sdb'

Jan 20 08:59:51 raspberrypi weewx[2517]: manager: Starting backfill of daily summaries

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.engine.StdArchive

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.restx.StdStationRegistry

Jan 20 08:59:51 raspberrypi weewx[2517]: restx: StationRegistry: Registration not requested.

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.restx.StdStationRegistry

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.restx.StdWunderground

Jan 20 08:59:51 raspberrypi weewx[2517]: restx: Wunderground-PWS: Data for station ICAMBSLE2 will be posted

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.restx.StdWunderground

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.restx.StdPWSweather

Jan 20 08:59:51 raspberrypi weewx[2517]: restx: PWSweather: Posting not enabled.

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.restx.StdPWSweather

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.restx.StdCWOP

Jan 20 08:59:51 raspberrypi weewx[2517]: restx: CWOP: Posting not enabled.

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.restx.StdCWOP

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.restx.StdWOW

Jan 20 08:59:51 raspberrypi weewx[2517]: restx: WOW: Posting not enabled.

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.restx.StdWOW

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.restx.StdAWEKAS

Jan 20 08:59:51 raspberrypi weewx[2517]: restx: AWEKAS: Posting not enabled.

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.restx.StdAWEKAS

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.engine.StdPrint

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.engine.StdPrint

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Loading service weewx.engine.StdReport

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Finished loading service weewx.engine.StdReport

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Starting up weewx version 3.8.0

Jan 20 08:59:51 raspberrypi weewx[2517]: vantage: Gentle wake up of console successful

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Clock error is -0.49 seconds (positive is fast)

Jan 20 08:59:51 raspberrypi weewx[2517]: vantage: Getting archive packets since 2018-01-20 08:50:00 GMT (1516438200)

Jan 20 08:59:51 raspberrypi weewx[2517]: vantage: Gentle wake up of console successful

Jan 20 08:59:51 raspberrypi weewx[2517]: manager: Daily summary version is 2.0

Jan 20 08:59:51 raspberrypi weewx[2517]: vantage: Retrieving 0 page(s); starting index= 0

Jan 20 08:59:51 raspberrypi weewx[2517]: engine: Starting main packet loop.

Jan 20 08:59:51 raspberrypi weewx[2517]: vantage: Gentle wake up of console successful

Jan 20 08:59:51 raspberrypi weewx[2517]: vantage: Requesting 200 LOOP packets.

Jan 20 08:59:51 raspberrypi weewx[2517]: vantage: Gentle wake up of console successful

Jan 20 09:00:15 raspberrypi weewx[2517]: vantage: Getting archive packets since 2018-01-20 08:50:00 GMT (1516438200)

Jan 20 09:00:15 raspberrypi weewx[2517]: vantage: Gentle wake up of console successful

Jan 20 09:00:15 raspberrypi weewx[2517]: vantage: Retrieving 1 page(s); starting index= 0

Jan 20 09:00:15 raspberrypi weewx[2517]: manager: Added record 2018-01-20 09:00:00 GMT (1516438800) to database 'weewx.sdb'

Jan 20 09:00:15 raspberrypi weewx[2517]: manager: Added record 2018-01-20 09:00:00 GMT (1516438800) to daily summary in 'weewx.sdb'

Jan 20 09:00:15 raspberrypi weewx[2517]: vantage: Empty record page 0; index 1

Jan 20 09:00:15 raspberrypi weewx[2517]: reportengine: Running reports for latest time in the database.

Jan 20 09:00:15 raspberrypi weewx[2517]: vantage: Requesting 200 LOOP packets.

Jan 20 09:00:15 raspberrypi weewx[2517]: reportengine: Running report StandardReport

Jan 20 09:00:15 raspberrypi weewx[2517]: reportengine: Found configuration file /etc/weewx/skins/ss/skin.conf for report StandardReport

Jan 20 09:00:15 raspberrypi weewx[2517]: vantage: Gentle wake up of console successful

Jan 20 09:00:15 raspberrypi weewx[2517]: copygenerator: copied 6 files to /public_html/ss

Jan 20 09:00:16 raspberrypi weewx[2517]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras']

Jan 20 09:00:16 raspberrypi weewx[2517]: restx: Wunderground-PWS: Published record 2018-01-20 09:00:00 GMT (1516438800)

Jan 20 09:00:16 raspberrypi weewx[2517]: manager: Daily summary version is 2.0

Jan 20 09:00:18 raspberrypi weewx[2517]: cheetahgenerator: Generated 2 files for report StandardReport in 2.14 seconds

Jan 20 09:00:18 raspberrypi weewx[2517]: manager: Daily summary version is 2.0

Jan 20 09:00:20 raspberrypi weewx[2517]: imagegenerator: Generated 11 images for StandardReport in 1.99 seconds

Jan 20 09:00:20 raspberrypi weewx[2517]: reportengine: Running report SteelSeries

Jan 20 09:00:20 raspberrypi weewx[2517]: reportengine: Found configuration file /etc/weewx/skins/ss/skin.conf for report SteelSeries

Jan 20 09:00:20 raspberrypi weewx[2517]: copygenerator: copied 6 files to /var/www/html/weewx/ss

Jan 20 09:00:20 raspberrypi weewx[2517]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras']

Jan 20 09:00:20 raspberrypi weewx[2517]: manager: Daily summary version is 2.0

Jan 20 09:00:20 raspberrypi weewx[2517]: cheetahgenerator: Generated 2 files for report SteelSeries in 0.29 seconds

Jan 20 09:00:20 raspberrypi weewx[2517]: manager: Daily summary version is 2.0

Jan 20 09:00:22 raspberrypi weewx[2517]: imagegenerator: Generated 11 images for SteelSeries in 1.90 seconds

Jan 20 09:00:22 raspberrypi weewx[2517]: reportengine: Running report FTP

Jan 20 09:00:22 raspberrypi weewx[2517]: reportengine: Found configuration file /etc/weewx/skins/Ftp/skin.conf for report FTP

Jan 20 09:00:22 raspberrypi weewx[2517]: ftpgenerator: FTP upload not requested. Skipped.

Jan 20 09:00:22 raspberrypi weewx[2517]: reportengine: Running report RSYNC

Jan 20 09:00:22 raspberrypi weewx[2517]: reportengine: Found configuration file /etc/weewx/skins/Rsync/skin.conf for report RSYNC

Jan 20 09:00:22 raspberrypi weewx[2517]: rsyncgenerator: rsync upload not requested. Skipped.

Jan 20 09:05:15 raspberrypi weewx[2517]: vantage: Getting archive packets since 2018-01-20 09:00:00 GMT (1516438800)

Jan 20 09:05:15 raspberrypi weewx[2517]: vantage: Gentle wake up of console successful

Jan 20 09:05:15 raspberrypi weewx[2517]: vantage: Retrieving 1 page(s); starting index= 1

Jan 20 09:05:15 raspberrypi weewx[2517]: manager: Added record 2018-01-20 09:05:00 GMT (1516439100) to database 'weewx.sdb'

Jan 20 09:05:15 raspberrypi weewx[2517]: manager: Added record 2018-01-20 09:05:00 GMT (1516439100) to daily summary in 'weewx.sdb'

Jan 20 09:05:15 raspberrypi weewx[2517]: vantage: Empty record page 0; index 2

Jan 20 09:05:15 raspberrypi weewx[2517]: reportengine: Running reports for latest time in the database.

Jan 20 09:05:15 raspberrypi weewx[2517]: reportengine: Running report StandardReport

Jan 20 09:05:15 raspberrypi weewx[2517]: vantage: Requesting 200 LOOP packets.

Jan 20 09:05:15 raspberrypi weewx[2517]: vantage: Gentle wake up of console successful

Jan 20 09:05:15 raspberrypi weewx[2517]: reportengine: Found configuration file /etc/weewx/skins/ss/skin.conf for report StandardReport

Jan 20 09:05:15 raspberrypi weewx[2517]: copygenerator: copied 0 files to /public_html/ss

Jan 20 09:05:15 raspberrypi weewx[2517]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras']

Jan 20 09:05:15 raspberrypi weewx[2517]: manager: Daily summary version is 2.0

Jan 20 09:05:15 raspberrypi weewx[2517]: cheetahgenerator: Generated 2 files for report StandardReport in 0.33 seconds

Jan 20 09:05:15 raspberrypi weewx[2517]: manager: Daily summary version is 2.0

Jan 20 09:05:15 raspberrypi weewx[2517]: restx: Wunderground-PWS: Published record 2018-01-20 09:05:00 GMT (1516439100)

Jan 20 09:05:17 raspberrypi weewx[2517]: imagegenerator: Generated 11 images for StandardReport in 1.97 seconds

Jan 20 09:05:17 raspberrypi weewx[2517]: reportengine: Running report SteelSeries

Jan 20 09:05:17 raspberrypi weewx[2517]: reportengine: Found configuration file /etc/weewx/skins/ss/skin.conf for report SteelSeries

Jan 20 09:05:17 raspberrypi weewx[2517]: copygenerator: copied 0 files to /var/www/html/weewx/ss

Jan 20 09:05:17 raspberrypi weewx[2517]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras']

Jan 20 09:05:17 raspberrypi weewx[2517]: manager: Daily summary version is 2.0

Jan 20 09:05:18 raspberrypi weewx[2517]: cheetahgenerator: Generated 2 files for report SteelSeries in 0.31 seconds

Jan 20 09:05:18 raspberrypi weewx[2517]: manager: Daily summary version is 2.0

Jan 20 09:05:20 raspberrypi weewx[2517]: imagegenerator: Generated 11 images for SteelSeries in 1.94 seconds

Jan 20 09:05:20 raspberrypi weewx[2517]: reportengine: Running report FTP

Jan 20 09:05:20 raspberrypi weewx[2517]: reportengine: Found configuration file /etc/weewx/skins/Ftp/skin.conf for report FTP

Jan 20 09:05:20 raspberrypi weewx[2517]: ftpgenerator: FTP upload not requested. Skipped.

Jan 20 09:05:20 raspberrypi weewx[2517]: reportengine: Running report RSYNC

Jan 20 09:05:20 raspberrypi weewx[2517]: reportengine: Found configuration file /etc/weewx/skins/Rsync/skin.conf for report RSYNC

Jan 20 09:05:20 raspberrypi weewx[2517]: rsyncgenerator: rsync upload not requested. Skipped.

Regards

Dave

 

[gjr80]

Dave,

The lines similar to:

Jan 20 08:56:46 raspberrypi wee_device[2124]: manager: Unable to add record 2018-01-18 17:45:00 GMT (1516297500) to database 'weewx.sdb': attempt to write a readonly database

indicate that data dumped from the logger by wee_device could not be written to the database, that could be because the user account under which wee_device was run did not have permission to write to the databse. Did you use sudo in front of wee_device? If not that could be the reason. No matter, that data was not written to database and the memory logger has been cleared so it is no more.

Everything else looks as it should. Corrupted station memory can occur as a result of a power outage. If you are running hardware record generarion (which most vantage users do) the symptoms you will see is that your web pages will not be updating, ie data will stay as per a given date and time (the last date-time in you archive which will probably be the record saved immediately before the outage) and if you then look at the log you will see very simialr to what you posted just now but the lines

Jan 20 09:05:15 raspberrypi weewx[2517]: manager: Added record 2018-01-20 09:05:00 GMT (1516439100) to database 'weewx.sdb'
Jan 20 09:05:15 raspberrypi weewx[2517]: manager: Added record 2018-01-20 09:05:00 GMT (1516439100) to daily summary in 'weewx.sdb'

will be missing (they should occur every archive interval seconds). Of course if you are using software record generation things will keep ticking along and you will not notice the difference, the price you pay using software record generation is that you loose the ability for the logger to buffer your archive records and allow you to download historical records/catchup in the event of an outage (well maybe not all outages given your power outage experience :) )

Gary

[Dave McCreath]

Gary

I'm afraid that you are right again, no, I didn't use sudo before issueing the 'dump' command, never mind too late now.

I hit Amazon this morning for a UPS, it should be here by the end of the coming week.  Maybe a bit of overkill but better to be safe (and angst free) rather than sorry.

Thank you again, very much appreciated.

Dave

[Thomas Keffer]

I hit Amazon this morning for a UPS, it should be here by the end of the coming week.  Maybe a bit of overkill but better to be safe (and angst free) rather than sorry.

​Not at all overkill. It's one of the keys to a stable system, particularly with an RPi. 

Because the Pi's don't draw much power​, the UPS does not have to be very big or expensive. $50 should do it.

-tk

[Dave McCreath]

Tom

About $90, but it'll also cover a Synology NAS plus the RPi and the Davis Vue console.

Dave