New User - Issues with reading WVIEW sdb during conversion to WeeWX.

[Punch Card]

So I think I have gotten the configuration file correct where it is seeing the weather station.

But it seems that there is some errant data in my wview sdb. Also I tried the " wee_database --string-check" there seem to be only one row in the data that had an issue, but it does not look like it corrected the row.
Is there a way to start weewx in a debug mode so I can find the errant data and either correct the data, or just delete the row.
Or do you have another suggestion?

[root@weewx]# wee_database --string-check
Using configuration file /etc/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Checking archive database for strings...
Timestamp = 1347032100; record['rain']= u'0\n0'; ... ignored.
[root@weewx]#

Last /var/log/messages for weewx attempt

Dec 27 08:43:10 md-centos systemd: Stopping SYSV: start and stop the weewx weather system...
Dec 27 08:43:10 md-centos weewx: Shutting down weewx: [FAILED]
Dec 27 08:43:10 md-centos systemd: Stopped SYSV: start and stop the weewx weather system.
Dec 27 09:32:16 md-centos systemd: Starting SYSV: start and stop the weewx weather system...
Dec 27 09:32:16 md-centos weewx[31695]: engine: Initializing weewx version 3.6.2
Dec 27 09:32:16 md-centos weewx[31695]: engine: Using Python 2.7.5 (default, Nov  6 2016, 00:28:07) #012[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]
Dec 27 09:32:16 md-centos weewx[31695]: engine: Platform Linux-3.10.0-514.2.2.el7.x86_64-x86_64-with-centos-7.3.1611-Core
Dec 27 09:32:16 md-centos weewx[31695]: engine: pid file is /var/run/weewx.pid
Dec 27 09:32:16 md-centos weewx[31699]: engine: Using configuration file /etc/weewx/weewx.conf
Dec 27 09:32:16 md-centos weewx[31699]: engine: Loading station type WMR200 (weewx.drivers.wmr200)
Dec 27 09:32:16 md-centos weewx: Starting weewx: [  OK  ]
Dec 27 09:32:16 md-centos systemd: Started SYSV: start and stop the weewx weather system.
Dec 27 09:32:16 md-centos weewx[31699]: wmr200: MainThread: I Created watchdog thread to poke for live data every 30 seconds
Dec 27 09:32:16 md-centos weewx[31699]: wmr200: MainThread: I Created USB polling thread to read block on device
Dec 27 09:32:16 md-centos weewx[31699]: wmr200: Thread-2: I USB polling device thread for live data launched
Dec 27 09:32:16 md-centos weewx[31699]: wmr200: MainThread: I Reset console device
Dec 27 09:32:17 md-centos weewx[31699]: wmr200: Thread-2: I USB polling device thread signaled to start
Dec 27 09:32:17 md-centos weewx[31699]: engine: StdConvert target unit is 0x1
Dec 27 09:32:17 md-centos weewx[31699]: wxcalculate: The following values will be calculated: barometer=prefer_hardware, windchill=prefer_hardware, dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, windrun=prefer_hardware, heatindex=prefer_hardware, maxSolarRad=prefer_hardware, humidex=prefer_hardware, pressure=prefer_hardware, inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, cloudbase=prefer_hardware
Dec 27 09:32:17 md-centos weewx[31699]: wxcalculate: The following algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS
Dec 27 09:32:17 md-centos weewx[31699]: engine: Archive will use data binding wx_binding
Dec 27 09:32:17 md-centos weewx[31699]: engine: Record generation will be attempted in 'hardware'
Dec 27 09:32:17 md-centos weewx[31699]: engine: Using archive interval of 300 seconds (specified by hardware)
Dec 27 09:32:17 md-centos weewx[31699]: wmr200: Thread-1: I Started watchdog thread live data
Dec 27 09:32:17 md-centos weewx[31699]: engine: Using binding 'wx_binding' to database 'weewx.sdb'
Dec 27 09:32:17 md-centos weewx[31699]: manager: Starting backfill of daily summaries
Dec 27 09:32:18 md-centos weewx[31699]: wmr200: Thread-2: I USB polling device thread exiting
Dec 27 09:32:18 md-centos weewx[31699]: wmr200: MainThread: I USB polling thread expired
Dec 27 09:32:18 md-centos weewx[31699]: wmr200: Thread-1: I Watchdog received shutdown
Dec 27 09:32:18 md-centos weewx[31699]: wmr200: Thread-1: I Watchdog thread exiting
Dec 27 09:32:18 md-centos weewx[31699]: wmr200: MainThread: I Watchdog thread expired
Dec 27 09:32:18 md-centos weewx[31699]: wmr200: MainThread: I Received packet count live:0 archive:0 control:0
Dec 27 09:32:18 md-centos weewx[31699]: wmr200: MainThread: I Received bytes:8 sent bytes:16
Dec 27 09:32:18 md-centos weewx[31699]: wmr200: MainThread: I Packet archive queue len:0 live queue len:0
Dec 27 09:32:18 md-centos weewx[31699]: wmr200: MainThread: I Driver gracefully exiting
Dec 27 09:32:18 md-centos weewx[31699]: engine: Caught unrecoverable exception in engine:
Dec 27 09:32:18 md-centos weewx[31699]:    ****  unsupported operand type(s) for +=: 'float' and 'unicode'
Dec 27 09:32:18 md-centos weewx[31699]:    ****  Traceback (most recent call last):
Dec 27 09:32:18 md-centos weewx[31699]:    ****    File "/usr/share/weewx/weewx/engine.py", line 841, in main
Dec 27 09:32:18 md-centos weewx[31699]:    ****      engine = engine_class(config_dict)
Dec 27 09:32:18 md-centos weewx[31699]:    ****    File "/usr/share/weewx/weewx/engine.py", line 76, in __init__
Dec 27 09:32:18 md-centos weewx[31699]:    ****      self.loadServices(config_dict)
Dec 27 09:32:18 md-centos weewx[31699]:    ****    File "/usr/share/weewx/weewx/engine.py", line 140, in loadServices
Dec 27 09:32:18 md-centos weewx[31699]:    ****      self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))
Dec 27 09:32:18 md-centos weewx[31699]:    ****    File "/usr/share/weewx/weewx/engine.py", line 486, in __init__
Dec 27 09:32:18 md-centos weewx[31699]:    ****      self.setup_database(config_dict)
Dec 27 09:32:18 md-centos weewx[31699]:    ****    File "/usr/share/weewx/weewx/engine.py", line 588, in setup_database
Dec 27 09:32:18 md-centos weewx[31699]:    ****      _nrecs, _ndays = dbmanager.backfill_day_summary() # @UnusedVariable
Dec 27 09:32:18 md-centos weewx[31699]:    ****    File "/usr/share/weewx/weewx/manager.py", line 1374, in backfill_day_summary
Dec 27 09:32:18 md-centos weewx[31699]:    ****      _day_accum.addRecord(_rec)
Dec 27 09:32:18 md-centos weewx[31699]:    ****    File "/usr/share/weewx/weewx/accum.py", line 240, in addRecord
Dec 27 09:32:18 md-centos weewx[31699]:    ****      func(self, record, obs_type, add_hilo)
Dec 27 09:32:18 md-centos weewx[31699]:    ****    File "/usr/share/weewx/weewx/accum.py", line 310, in add_value
Dec 27 09:32:18 md-centos weewx[31699]:    ****      self[obs_type].addSum(val)
Dec 27 09:32:18 md-centos weewx[31699]:    ****    File "/usr/share/weewx/weewx/accum.py", line 87, in addSum
Dec 27 09:32:18 md-centos weewx[31699]:    ****      self.sum     += val
Dec 27 09:32:18 md-centos weewx[31699]:    ****  TypeError: unsupported operand type(s) for +=: 'float' and 'unicode'
Dec 27 09:32:18 md-centos weewx[31699]:    ****  Exiting.
Dec 27 09:32:18 md-centos python: detected unhandled Python exception in '/usr/bin/weewxd'
Dec 27 09:32:55 md-centos systemd: Stopping SYSV: start and stop the weewx weather system...
Dec 27 09:32:55 md-centos weewx: Shutting down weewx: [FAILED]
Dec 27 09:32:55 md-centos systemd: Stopped SYSV: start and stop the weewx weather system.

Thanks for your assistance in advance.
Regards,
Mike

[Thomas Keffer]

Did you try adding the --fix option?

wee_database --string-check --fix

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Punch Card]

Hummm.... I guess I should had done a little more research, the one post I found,  the "--fix" was not in the example.

Looks like it is logging, I am seeing the follow entries about Primary Key in in messages. Do these log items look to be an issue?

Dec 27 21:07:22 md-centos weewx[32547]: manager: added record 2016-12-27 07:17:22 CST (1482844642) to database 'weewx.sdb'
Dec 27 21:07:22 md-centos weewx[32547]: manager: added record 2016-12-27 07:17:22 CST (1482844642) to daily summary in 'weewx.sdb'
Dec 27 21:07:26 md-centos weewx[32547]: wmr200: MainThread: I genStartup() Still receiving archive packets cnt:161 len:1
Dec 27 21:07:26 md-centos weewx[32547]: wmr200: MainThread: I Using pc time adjusting archive record time by 314950642 sec 2007-01-04 01:00:00 CST (1167894000) => 2016-12-27 07:17:22 CST (1482844642)
Dec 27 21:07:26 md-centos weewx[32547]: manager: unable to add record 2016-12-27 07:17:22 CST (1482844642) to database 'weewx.sdb': PRIMARY KEY must be unique
Dec 27 21:07:26 md-centos weewx[32547]: wmr200: MainThread: I genStartup() Still receiving archive packets cnt:162 len:1
Dec 27 21:07:26 md-centos weewx[32547]: wmr200: MainThread: I Using pc time adjusting archive record time by 314950642 sec 2007-01-04 01:04:00 CST (1167894240) => 2016-12-27 07:21:22 CST (1482844882)
Dec 27 21:07:26 md-centos weewx[32547]: manager: added record 2016-12-27 07:21:22 CST (1482844882) to database 'weewx.sdb'
Dec 27 21:07:26 md-centos weewx[32547]: manager: added record 2016-12-27 07:21:22 CST (1482844882) to daily summary in 'weewx.sdb'

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[Andrew Milner]

The manuals and documentation are your best friends ....

http://weewx.com/docs/utilities.htm#wee_database_utility

[Oscar]

Here is the process I use to convert my wview to weewx.  The word None and possibly some special characters are in the wview archive table.  The following process will clean up those columns.

1.  copy wview-archive.sdb to weewx.sdb

2.  run the attached script weewx-wviewCleanup.sql.  It will change those values to NULL. This change the command below to the appropriate paths.

sqlite3 -init PATH_TO_FILE/weewx_wviewCleanup.sql PATH_TO_DATABSE/weewx.sdb.

3.  run wee_database again. 

   If it fails, run wee_database --string-check PATH_TO_CONFIG/weewx.conf

   if that fails, note the error.  If the error identifies a column has None in it, modify the weewx_wviewCleanup.sql script to null out that column and run it again.

That process resolved my issues.

After the database has been converted by wee_database, you may want to run the script zkwx_wviewHiLow.py.  It reads the view-hilow.sdb data base. 

It reads most of the tables and identifies the max(high) and min(low) for each day.  It updates the appropriate archive_day_XXX table in weewx.sdb. 

It doesn't touch rain and rainRate.  I wasn't sure how I should total them, so I'm using the weewx calculation.  The script requires python3.  See disclaimer 2.

to run it

zkwx_wviewHiLow.py -f PATH_TO/wview-hilow.sdb -t PATH_TO/weewx.sdb

Disclaimers:

1.  Use at your own risk

2.  I don't know python.  If you (or anyone) looks at the code and asks themselves why did he do that, the answer is I found an example and made it work.

Good Luck,

Oscar