Rasbian 10 - Update failed

112 views
Skip to first unread message

Manfred Maier

unread,
May 1, 2020, 10:34:11 AM5/1/20
to weewx-user
Also pulling my issues while updating in a separate thread.

I was using version 3.9.2 on a RPi4 (Rasbian 10) with the Belchertown skin installed.

I've tried to install the Python 3 version with:
curl -s http://weewx.com/apt/weewx-python3.list | sudo tee /etc/apt/sources.list.d/weewx.list
sudo apt-get update
sudo apt-get install weewx

The system crashed with the following lines in the log-file:

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__: Caught unrecoverable exception:

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****  invalid syntax (mqtt.py, line 197)

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****  Traceback (most recent call last):

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****    File "/usr/share/weewx/weewxd", line 148, in main

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****      engine = weewx.engine.StdEngine(config_dict)

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 75, in __init__

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****      self.loadServices(config_dict)

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 138, in loadServices

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****      obj = weeutil.weeutil.get_object(svc)(self,config_dict)

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****    File "/usr/share/weewx/weeutil/weeutil.py", line 1093, in get_object

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****      mod = __import__(module)

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****    File "/usr/share/weewx/user/mqtt.py", line 197

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****      except KeyError, e:

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****                     ^

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****  SyntaxError: invalid syntax

May  1 10:57:40 raspberrypi weewx[14513] CRITICAL __main__:     ****  Exiting.



I then installed the Python 2 version.
Weewx at least started again. The MQTT is working, too. 

But the system stopped creating most of JSON files for the skins.
The weewx_data.json gets created, but all the files for my belchertown graphs do not get generated.


Ideally I would like to switch to the Python 3 version.
Any thoughts what the problem of my Python 3 installation is?


Manfred. 

Tarmo

unread,
May 1, 2020, 10:37:04 AM5/1/20
to weewx-user
Hi Manfred!

try to update mqtt extension https://github.com/matthewwall/weewx-mqtt

Manfred Maier

unread,
May 1, 2020, 10:56:55 AM5/1/20
to weewx-user
A very stupid question:
How do I update the extension?

I don't find the weewx-mqtt.tgz that I need for the installation.

Tarmo

unread,
May 1, 2020, 11:03:36 AM5/1/20
to weewx-user
grab it from here https://github.com/matthewwall/weewx-mqtt/archive/master.zip

wee_extension supports zip files as well

Manfred Maier

unread,
May 1, 2020, 11:06:44 AM5/1/20
to weewx-user
Thanks for your support!

Tried that already - with not much success after downloading the master.zip

pi@raspberrypi:~ $ sudo wee_extension --install master.zip

Request to install 'master.zip'

Extracting from zip archive master.zip

Traceback (most recent call last):

  File "/usr/share/weewx/wee_extension", line 88, in <module>

    main()

  File "/usr/share/weewx/wee_extension", line 80, in main

    ext.install_extension(options.install)

  File "/usr/share/weewx/weecfg/extension.py", line 113, in install_extension

    self.tmpdir, self.logger)

  File "/usr/share/weewx/weecfg/__init__.py", line 1760, in extract_zip

    zip_archive = zipfile.ZipFile(filename)

  File "/usr/lib/python3.7/zipfile.py", line 1222, in __init__

    self._RealGetContents()

  File "/usr/lib/python3.7/zipfile.py", line 1289, in _RealGetContents

    raise BadZipFile("File is not a zip file")

zipfile.BadZipFile: File is not a zip file

Michael

unread,
May 1, 2020, 11:10:54 AM5/1/20
to weewx-user
it does not appear to be a ZIP. I would reload the file, e.g. with wget https://github.com/matthewwall/weewx-mqtt/archive/master.zip

Manfred Maier

unread,
May 1, 2020, 11:23:37 AM5/1/20
to weewx-user
Yes, I've used the wrong command for the download :/

Still, even after updating the MQTT extension, the python 3 version crashes.

I've followed Matthew's instructions for installing the extension:
mqtt - weewx extension that sends data to an MQTT broker
Copyright 2014-2020 Matthew Wall
Distributed under the terms of the GNU Public License (GPLv3)

Installation instructions:

0) install the MQTT python bindings

sudo pip install paho-mqtt


When installing the MQTT python bindings, I get the following output:

Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple

Requirement already satisfied: paho-mqtt in /usr/local/lib/python2.7/dist-packages (1.4.0)


This looks a lot like Python 2.7

Michael

unread,
May 1, 2020, 11:26:21 AM5/1/20
to weewx-user
try it with 

sudo pip3 install paho-mqtt

Manfred Maier

unread,
May 1, 2020, 11:38:48 AM5/1/20
to weewx-user
Ok. One step done.

Python 3 version still is crashing, but now with a slightly different error message in the log:

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.engine: Finished loading service user.windy.Windy

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.engine: Loading service user.owm.OpenWeatherMap

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.manager: Daily summary version is 2.0

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.manager: Daily summary version is 2.0

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.manager: Daily summary version is 2.0

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.manager: Daily summary version is 2.0

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.restx: Shut down Windy thread.

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.restx: Shut down MQTT thread.

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.restx: Shut down AWEKAS thread.

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.restx: Shut down WOW thread.

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.restx: Shut down CWOP thread.

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.restx: Shut down PWSWeather thread.

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.restx: Shut down Wunderground-PWS thread.

May  1 17:28:48 raspberrypi weewx[19064] DEBUG weewx.restx: Shut down StationRegistry thread.

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__: Caught unrecoverable exception:

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****  invalid syntax (owm.py, line 84)

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****  Traceback (most recent call last):

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****    File "/usr/share/weewx/weewxd", line 148, in main

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****      engine = weewx.engine.StdEngine(config_dict)

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 75, in __init__

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****      self.loadServices(config_dict)

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 138, in loadServices

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****      obj = weeutil.weeutil.get_object(svc)(self,config_dict)

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****    File "/usr/share/weewx/weeutil/weeutil.py", line 1093, in get_object

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****      mod = __import__(module)

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****    File "/usr/share/weewx/user/owm.py", line 84

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****      except KeyError, e:

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****                     ^

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****  SyntaxError: invalid syntax

May  1 17:28:48 raspberrypi weewx[19064] CRITICAL __main__:     ****  Exiting.

Manfred Maier

unread,
May 1, 2020, 11:41:16 AM5/1/20
to weewx-user
Oh. That might be OpenWeatherMap

Give me a minute.

Manfred Maier

unread,
May 1, 2020, 12:00:50 PM5/1/20
to weewx-user
Three more steps done!

Extensions that crash with Python 3:
* OpenWeatherMap
* WCloud
* Wetter

For OWM I've tried to find a newer version (> 0.8), but could find any.
Same for the other two :(


So currently my only issue left:
The JSON files for the Belchertown skin are still not produced.

Manfred Maier

unread,
May 1, 2020, 12:19:48 PM5/1/20
to weewx-user
So, the next issue has to do with MaxSolarRad
I have that in my database, but as 'maxSolarRad'


May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine: Caught unrecoverable exception in generator 'user.belchertown.HighchartsJsonGenerator'

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****  Error trying to use database binding wx_binding to graph observation MaxSolarRad. Error was: MaxSolarRad.

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****  Traceback (most recent call last):

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/user/belchertown.py", line 1821, in get_observation_data

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****      (time_start_vt, time_stop_vt, obs_vt) = archive.getSqlVectors(TimeSpan(start_ts, end_ts), obs_lookup, aggregate_type, aggregate_in$

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/manager.py", line 452, in getSqlVectors

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****      aggregate_type, aggregate_interval)

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/xtypes.py", line 91, in get_series

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****      raise weewx.UnknownType(obs_type)

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****  weewx.UnknownType: MaxSolarRad

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****  During handling of the above exception, another exception occurred:

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****  Traceback (most recent call last):

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 197, in run

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****      obj.start()

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 280, in start

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****      self.run()

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/user/belchertown.py", line 1370, in run

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****      series_data = self.get_observation_data(binding, archive, observation_type, minstamp, maxstamp, aggregate_type, aggregate_interval$

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/user/belchertown.py", line 1823, in get_observation_data

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****      raise Warning( "Error trying to use database binding %s to graph observation %s. Error was: %s." % (binding, obs_lookup, e) )

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****  Warning: Error trying to use database binding wx_binding to graph observation MaxSolarRad. Error was: MaxSolarRad.

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****  Generator terminated

May  1 17:55:22 raspberrypi weewx[21194] DEBUG weewx.reportengine: Running report 'sftp'

May  1 17:55:22 raspberrypi weewx[21194] DEBUG weewx.reportengine: Found configuration file /etc/weewx/skins/sftp/skin.conf for report 'sftp'

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine: Unable to instantiate generator 'user.sftp.SFTPGenerator'

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****  invalid syntax (sftp.py, line 81)

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****  Traceback (most recent call last):

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 180, in run

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****      obj = weeutil.weeutil.get_object(generator)(

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weeutil/weeutil.py", line 1093, in get_object

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****      mod = __import__(module)

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/user/sftp.py", line 81

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****      except pysftp.ConnectionException, e:

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****                                       ^

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****  SyntaxError: invalid syntax

May  1 17:55:22 raspberrypi weewx[21194] ERROR weewx.reportengine:         ****  Generator ignored




Manfred Maier

unread,
May 1, 2020, 12:45:38 PM5/1/20
to weewx-user
Good. Next one solved.
The Belchertown graphs.conf seems to be case sensitive with the new weewx 4.0

The observation "MaxSolarRad" worked in 3.9.2, but does not work in 4.0
After renaming it to "maxSolarRad" this issue was gone.


But there's another issue - and hopefully the last one:
Is that again a python 3 issue with an extension? In this case SFTP?

Any hint welcome!


pi@raspberrypi:~ $ sudo wee_reports

Using configuration file /etc/weewx/weewx.conf

Generating for all time

Traceback (most recent call last):

  File "/usr/share/weewx/weewx/reportengine.py", line 180, in run

    obj = weeutil.weeutil.get_object(generator)(

  File "/usr/share/weewx/weeutil/weeutil.py", line 1093, in get_object

    mod = __import__(module)

  File "/usr/share/weewx/user/sftp.py", line 81

    except pysftp.ConnectionException, e:

                                     ^

SyntaxError: invalid syntax




Manfred Maier

unread,
May 1, 2020, 12:53:00 PM5/1/20
to weewx-user
Apologies for talking to myself here :)

I've now updated the SFTP extension. 
At least the error massaged changed:

pi@raspberrypi:~ $ sudo wee_reports

Using configuration file /etc/weewx/weewx.conf

Generating for all time

Traceback (most recent call last):

  File "/usr/share/weewx/weewx/reportengine.py", line 197, in run

    obj.start()

  File "/usr/share/weewx/weewx/reportengine.py", line 280, in start

    self.run()

  File "/usr/share/weewx/user/sftp.py", line 259, in run

    n = uploader.run()

  File "/usr/share/weewx/user/sftp.py", line 89, in run

    (timestamp, fileset) = self.get_last_upload()

  File "/usr/share/weewx/user/sftp.py", line 158, in get_last_upload

    timestamp = cPickle.load(f)

TypeError: a bytes-like object is required, not 'str'



Now I'm really running out of ideas.



vince

unread,
May 1, 2020, 1:34:49 PM5/1/20
to weewx-user
On Friday, May 1, 2020 at 9:45:38 AM UTC-7, Manfred Maier wrote:

    except pysftp.ConnectionException, e:

                                     ^

SyntaxError: invalid syntax





Any time you see something that looks like that, you are running python3 against code written for python2

All the extensions written for python2 need to be updated to support python3.
You should check to see if there is a more recent version of this extension.


vince

unread,
May 1, 2020, 1:40:05 PM5/1/20
to weewx-user
On Friday, May 1, 2020 at 9:53:00 AM UTC-7, Manfred Maier wrote:

  File "/usr/share/weewx/user/sftp.py", line 158, in get_last_upload

    timestamp = cPickle.load(f)

TypeError: a bytes-like object is required, not 'str'



This is another known python3 thing, which tells me your extension isn't fully updated to support python3 quite yet.

Looking in google for the exact words on your 'TypeError' output above points me to:
and



Manfred Maier

unread,
May 1, 2020, 1:44:40 PM5/1/20
to weewx-user
That's Matthew Wall's SFTP extension and I haven't found an updated version.
Anyway, I'll raise this issue directly in the Github project.

Thanks!

vince

unread,
May 1, 2020, 1:55:10 PM5/1/20
to weewx-user
On Friday, May 1, 2020 at 10:44:40 AM UTC-7, Manfred Maier wrote:
That's Matthew Wall's SFTP extension and I haven't found an updated version.
Anyway, I'll raise this issue directly in the Github project.

If you want to try something, since Matthew's busy I'm sure on more core stuff (plus having a life etc.), run the extension through "2to3 -w" and restart weewx and see if that helps any.  That will try to do an upgrade to python3-compliant and will get pretty close.

If you want to point me to a URL with the file you're using, I can take a look if you want. 

Manfred Maier

unread,
May 1, 2020, 2:00:51 PM5/1/20
to weewx-user
I'll give it a try, once the kids are in bed.

Thanks!


Manfred Maier

unread,
May 1, 2020, 3:37:25 PM5/1/20
to weewx-user
Just FYI

I was able to resolve the python 3 issue of the SFTP extension by making a few changes in the sftp.py file.

Thanks vince for pointing me in the right direction!

Manfred Maier

unread,
May 1, 2020, 4:33:36 PM5/1/20
to weewx-user
I've cheered too soon. 
The new version only worked with wee-Reports, but not in the normal reporting loop.

Anyway, I'm using FTP instead of SFTP now.

John Kline

unread,
May 2, 2020, 11:18:46 PM5/2/20
to weewx...@googlegroups.com
> TypeError: a bytes-like object is required, not 'str'
I do not use this extension, but the error is an indication that the extension was not ported to work with Python 3 (bytes and str are different types in Python 3).

I don’t know if it the author claims WeeWX 4 comparability, but if he does, he may not have tested it with WeeWX 4/Python 3.


On May 1, 2020, at 9:53 AM, Manfred Maier <moni.und...@web.de> wrote:


--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/1e213c8a-579e-44c8-9da6-c73b47071061%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages