move from old 3.9 (python2) machine to a new machine running 4.1.1 (python3) with mysql database from 3.9 to 4.1...?

[Christian Peters]

Hi,

is there some howto available to move my weewx mysql database to a new weewx machine (now 4.1.1 on Python3). I'm not sure howto migrate the databse and do the transition as smooth as possible?

I noticed that there are a lot of new and additional entries in wview_extended.py which weren't available on my Python2 machine I run for years now. :-) 

I added some entries there and see they are now available ( in parts) in the new weewx 4 scheme?

I would be pleased if I can move the adatabase or import the main values like temp, hum at first place and as a minimum to my new setup?

I will not change the hardware (still VP2), just move the database and the VP2 to a new machine with weewx 4 and will use a new skin.

Is there an utility to merge the old weewx database (Mysql) in the new one on the new machine from a backup?

Will it insert the data of equal fields in the right way!?

Sorry...im not very familiar with mysql but it would ne nice not to loose data from now 6 years and beginn with a blank database.

Thanks for any help!

Regards,

Christian

my "old" wview_extended.py (3.9.):

schema = [('dateTime',             'INTEGER NOT NULL UNIQUE PRIMARY KEY'),
          ('usUnits',              'INTEGER NOT NULL'),
          ('interval',             'INTEGER NOT NULL'),
          ('barometer',            'REAL'),
          ('pressure',             'REAL'),
          ('altimeter',            'REAL'),
          ('inTemp',               'REAL'),
          ('outTemp',              'REAL'),
          ('inHumidity',           'REAL'),
          ('outHumidity',          'REAL'),
          ('windSpeed',            'REAL'),
          ('windDir',              'REAL'),
          ('windGust',             'REAL'),
          ('windGustDir',          'REAL'),
          ('rainRate',             'REAL'),
          ('rain',                 'REAL'),
          ('dewpoint',             'REAL'),
          ('windchill',            'REAL'),
          ('heatindex',            'REAL'),
          ('ET',                   'REAL'),
          ('radiation',            'REAL'),
          ('UV',                   'REAL'),
          ('extraTemp1',           'REAL'),
          ('extraTemp2',           'REAL'),
          ('extraTemp3',           'REAL'),
          ('soilTemp1',            'REAL'),
          ('soilTemp2',            'REAL'),
          ('soilTemp3',            'REAL'),
          ('soilTemp4',            'REAL'),
          ('leafTemp1',            'REAL'),
          ('leafTemp2',            'REAL'),
          ('extraHumid1',          'REAL'),
          ('extraHumid2',          'REAL'),
          ('soilMoist1',           'REAL'),
          ('soilMoist2',           'REAL'),
          ('soilMoist3',           'REAL'),
          ('soilMoist4',           'REAL'),
          ('leafWet1',             'REAL'),
          ('leafWet2',             'REAL'),
          ('rxCheckPercent',       'REAL'),
          ('txBatteryStatus',      'REAL'),
          ('consBatteryVoltage',   'REAL'),
          ('hail',                 'REAL'),
          ('hailRate',             'REAL'),
          ('heatingTemp',          'REAL'),
          ('heatingVoltage',       'REAL'),
          ('supplyVoltage',        'REAL'),
          ('referenceVoltage',     'REAL'),
          ('windBatteryStatus',    'REAL'),
          ('rainBatteryStatus',    'REAL'),
          ('outTempBatteryStatus', 'REAL'),
          ('inTempBatteryStatus',  'REAL'),
          ('cpm',                  'REAL'),
          ('cps',                  'REAL'),
          ('geiger',               'REAL'),
          ('PM10_1',               'REAL'),
          ('PM10_2',               'REAL'),
          ('PM25_1',               'REAL'),
          ('PM25_2',               'REAL'),
          ('tmp_dt_1',             'REAL'),
          ('tmp_dt_2',             'REAL'),
          ('hum_dt_1',             'REAL'),
          ('hum_dt_2',             'REAL')]

and the wview_extended.py (4.1):

table = [('dateTime',             'INTEGER NOT NULL UNIQUE PRIMARY KEY'),
         ('usUnits',              'INTEGER NOT NULL'),
         ('interval',             'INTEGER NOT NULL'),
         ('altimeter',            'REAL'),
         ('appTemp',              'REAL'),
         ('appTemp1',             'REAL'),
         ('barometer',            'REAL'),
         ('batteryStatus1',       'REAL'),
         ('batteryStatus2',       'REAL'),
         ('batteryStatus3',       'REAL'),
         ('batteryStatus4',       'REAL'),
         ('batteryStatus5',       'REAL'),
         ('batteryStatus6',       'REAL'),
         ('batteryStatus7',       'REAL'),
         ('batteryStatus8',       'REAL'),
         ('cloudbase',            'REAL'),
         ('co',                   'REAL'),
         ('co2',                  'REAL'),
         ('consBatteryVoltage',   'REAL'),
         ('dewpoint',             'REAL'),
         ('dewpoint1',            'REAL'),
         ('ET',                   'REAL'),
         ('extraHumid1',          'REAL'),
         ('extraHumid2',          'REAL'),
         ('extraHumid3',          'REAL'),
         ('extraHumid4',          'REAL'),
         ('extraHumid5',          'REAL'),
         ('extraHumid6',          'REAL'),
         ('extraHumid7',          'REAL'),
         ('extraHumid8',          'REAL'),
         ('extraTemp1',           'REAL'),
         ('extraTemp2',           'REAL'),
         ('extraTemp3',           'REAL'),
         ('extraTemp4',           'REAL'),
         ('extraTemp5',           'REAL'),
         ('extraTemp6',           'REAL'),
         ('extraTemp7',           'REAL'),
         ('extraTemp8',           'REAL'),
         ('forecast',             'REAL'),
         ('hail',                 'REAL'),
         ('hailBatteryStatus',    'REAL'),
         ('hailRate',             'REAL'),
         ('heatindex',            'REAL'),
         ('heatindex1',           'REAL'),
         ('heatingTemp',          'REAL'),
         ('heatingVoltage',       'REAL'),
         ('humidex',              'REAL'),
         ('humidex1',             'REAL'),
         ('inDewpoint',           'REAL'),
         ('inHumidity',           'REAL'),
         ('inTemp',               'REAL'),
         ('inTempBatteryStatus',  'REAL'),
         ('leafTemp1',            'REAL'),
         ('leafTemp2',            'REAL'),
         ('leafWet1',             'REAL'),
         ('leafWet2',             'REAL'),
         ('lightning_distance',        'REAL'),
         ('lightning_disturber_count', 'REAL'),
         ('lightning_energy',          'REAL'),
         ('lightning_noise_count',     'REAL'),
         ('lightning_strike_count',    'REAL'),
         ('luminosity',           'REAL'),
         ('maxSolarRad',          'REAL'),
         ('nh3',                  'REAL'),
         ('no2',                  'REAL'),
         ('noise',                'REAL'),
         ('o3',                   'REAL'),
         ('outHumidity',          'REAL'),
         ('outTemp',              'REAL'),
         ('outTempBatteryStatus', 'REAL'),
         ('pb',                   'REAL'),
         ('pm10_0',               'REAL'),
         ('pm1_0',                'REAL'),
         ('pm2_5',                'REAL'),
         ('pressure',             'REAL'),
         ('radiation',            'REAL'),
         ('rain',                 'REAL'),
         ('rainBatteryStatus',    'REAL'),
         ('rainRate',             'REAL'),
         ('referenceVoltage',     'REAL'),
         ('rxCheckPercent',       'REAL'),
         ('signal1',              'REAL'),
         ('signal2',              'REAL'),
         ('signal3',              'REAL'),
         ('signal4',              'REAL'),
         ('signal5',              'REAL'),
         ('signal6',              'REAL'),
         ('signal7',              'REAL'),
         ('signal8',              'REAL'),
         ('snow',                 'REAL'),
         ('snowBatteryStatus',    'REAL'),
         ('snowDepth',            'REAL'),
         ('snowMoisture',         'REAL'),
         ('snowRate',             'REAL'),
         ('so2',                  'REAL'),
         ('soilMoist1',           'REAL'),
         ('soilMoist2',           'REAL'),
         ('soilMoist3',           'REAL'),
         ('soilMoist4',           'REAL'),
         ('soilTemp1',            'REAL'),
         ('soilTemp2',            'REAL'),
         ('soilTemp3',            'REAL'),
         ('soilTemp4',            'REAL'),
         ('supplyVoltage',        'REAL'),
         ('txBatteryStatus',      'REAL'),
         ('UV',                   'REAL'),
         ('uvBatteryStatus',      'REAL'),
         ('windBatteryStatus',    'REAL'),
         ('windchill',            'REAL'),
         ('windDir',              'REAL'),
         ('windGust',             'REAL'),
         ('windGustDir',          'REAL'),
         ('windrun',              'REAL'),
         ('windSpeed',            'REAL'),
         ]

day_summaries = [(e[0], 'scalar') for e in table
                 if e[0] not in ('dateTime', 'usUnits', 'interval')] + [('wind', 'VECTOR')]

schema = {
    'table': table,
    'day_summaries' : day_summaries
}

[Cameron D]

You do not need to change your database at all - but obviously back it up first.

I found all I had to do was add the bottom few lines (day_summaries) to my custom schema file.

[Christian Peters]

Cameron,

this doesn't work. I copied my database to the new machine, made a new weewx database and replayed it.

If I use the wview_extended.py from the 2.5.-3.9 weewx install, it seems to work.

If I add the new additional lines form the 4.0.0. at the end of the wview_extended.py file...

day_summaries = [(e[0], 'scalar') for e in table
                 if e[0] not in ('dateTime', 'usUnits', 'interval')] + [('wind', 'VECTOR')]

schema = {
    'table': table,
    'day_summaries' : day_summaries
}

I got this error:

Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__: Caught unrecoverable exception:
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****  name 'table' is not defined
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****  Traceback (most recent call last):
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****    File "/usr/share/weewx/weewxd", line 148, in main
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****      engine = weewx.engine.StdEngine(config_dict)
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 75, in __init__
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****      self.loadServices(config_dict)
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 138, in loadServices
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****      obj = weeutil.weeutil.get_object(svc)(self,config_dict)
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/wxservices.py", line 81, in __init__
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****      engine.db_binder)
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/wxservices.py", line 124, in __init__
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****      initialize=True)
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/manager.py", line 529, in get_manager
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****      default_binding_dict=defaults)
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/manager.py", line 660, in get_manager_dict_from_config
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****      manager_dict['schema'] = weeutil.weeutil.get_object(schema_name)
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****    File "/usr/share/weewx/weeutil/weeutil.py", line 1093, in get_object
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****      mod = __import__(module)
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****    File "/usr/share/weewx/schemas/wview_extended.py", line 84, in <module>
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****      day_summaries = [(e[0], 'scalar') for e in table
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****  NameError: name 'table' is not defined
Jun 20 10:30:55 weewx weewx[24866] CRITICAL __main__:     ****  Exiting.

The question is: should I just run 4.x with the old database and old scheme!? But then the new added fields are missing like

'

cloudbase',            'REAL'),
     ('co',                   'REAL'),
    ('co2',                  'REAL'),

...which could be interesting in the future.

So how to move the old database to a new one with old data and the new fields and new syntax?

Regards,

Christian

[Cameron D]

Hi Christian,

sorry, I forgot the bit where you also need to rename your main schema from "schema" to "table".

Don't worry about adding columns just in case you might need them in future - with mysql you can add them any time you need. It is rather more flexible than sqlite about altering table definitions after they have been created.

Cameron.

[Christian Peters]

Hi Caeron,

damn...yes....theat was obvious. It works now. Thank you very much

But Python3 seems not as lazy as Python2, now I got this and weewx stops generating the webpage:

 

Generate failed with exception '<class 'TypeError'>'
Jun 20 11:00:56 weewx weewx[25088] ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/Responsive/index.html.tmpl
Jun 20 11:00:56 weewx weewx[25088] ERROR weewx.cheetahgenerator: **** Reason: '>' not supported between instances of 'NoneType' and 'int'
Jun 20 11:00:56 weewx weewx[25088] ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
Jun 20 11:00:56 weewx weewx[25088] ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 322, in generate
Jun 20 11:00:56 weewx weewx[25088] ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
Jun 20 11:00:56 weewx weewx[25088] ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_Responsive_index_html_tmpl.py", line 2284, in respond
Jun 20 11:00:56 weewx weewx[25088] ERROR weewx.cheetahgenerator: ****  TypeError: '>' not supported between instances of 'NoneType' and 'int'

I fear some value inside the database which was queried and I want to do a graph seems to contain 'no data' or 'NoneType'?

But the errer doen'st really help where it could happened...! :-P

Let's see...

Regards,

Christian

[Cameron D]

Hi Christian,

That error has me lost. I use only a heavily modified Seasons skin, which I migrated by merging my changes into the new skin folder.  I have trimmed out all the references to data I do not have, mainly to make it easier to follow, and perhaps get a tiny improvement in efficiency.

I would suggest you enable only one skin, then check for it trying to use parameters not in the database - mostly the uncommon ones are tested for existence before trying to use them but maybe some aren't.

If you are lucky somebody might come along who understands the issue - I am just guessing.

Cameron.

[Graham Eddy]

suggest you look in index.html.tmpl for an expression of the form #if x > y, where x could produce value None and y is an int.

g-eddy

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/151b98d9-9805-4f3b-8654-7c506718bd36o%40googlegroups.com.

[Cameron D]

One other possibility - I notice the developer of Responsive skin is still running 3.9.2.  Perhaps there is something about the skin that needs updating for Python 3.  

[Christian Peters]

Thank you, Graham and Cameron,

it took a long time but Graham was on the right track.

I check $trend.barometer with an #if condition.

This gives N/A for $trend.barometer ...???

This works in P2 and 3.9.x but now not in P3 4.x.x....?

Changing that to $trend($time_delta=3600).barometer gives a value!?

What a nightmare to debug! :-P

Regards,

Christian

g-eddy

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.