New weewx-sftp errors

[Tom Hogland]

I've been using weewx-sftp to upload to my hosted server without error for some time - ssh-rsa keys in place, no password, all great. WIth the 4.10 update I dug into the logs and discovered that my uploads were failing with a publickey/password error. I dug (for days, and back and forth with support, etc.) and discovered that at some point something between my system and the remote host decided rsa wasn't a good cipher to use. Switching to ed25519 or ecdsa both allowed the connections to work fine, until I changed the [sftp] keyfile to use either one and restarted I'm now getting this error in the logs:

Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine: Caught unrecoverable exception in generator 'user.sftp.SFTPGenerator'
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  unpack requires a buffer of 4 bytes
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/paramiko/pkey.py", line 508, in _uint32_cstruct_unpack
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      s_size = struct.unpack(">L", data[idx : idx + 4])[0]
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  struct.error: unpack requires a buffer of 4 bytes
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  During handling of the above exception, another exception occurred:
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/pysftp/__init__.py", line 166, in _set_authentication
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      self._tconnect['pkey'] = RSAKey.from_private_key_file(
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/paramiko/pkey.py", line 232, in from_private_key_file
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      key = cls(filename=filename, password=password)
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/paramiko/rsakey.py", line 63, in __init__
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      self._from_private_key_file(filename, password)
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/paramiko/rsakey.py", line 187, in _from_private_key_file
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      self._decode_key(data)
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/paramiko/rsakey.py", line 203, in _decode_key
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      n, e, d, iqmp, p, q = self._uint32_cstruct_unpack(data, "iiiiii")
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/paramiko/pkey.py", line 528, in _uint32_cstruct_unpack
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      raise SSHException(str(e))
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  paramiko.ssh_exception.SSHException: unpack requires a buffer of 4 bytes
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  During handling of the above exception, another exception occurred:
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/paramiko/pkey.py", line 508, in _uint32_cstruct_unpack
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      s_size = struct.unpack(">L", data[idx : idx + 4])[0]
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  struct.error: unpack requires a buffer of 4 bytes
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  During handling of the above exception, another exception occurred:
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 197, in run
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      obj.start()
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 385, in start
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      self.run()
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/user/sftp.py", line 260, in run
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      n = uploader.run()
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/user/sftp.py", line 98, in run
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      con = pysftp.Connection(host=self.server,
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/pysftp/__init__.py", line 142, in __init__
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      self._set_authentication(password, private_key, private_key_pass)
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/pysftp/__init__.py", line 170, in _set_authentication
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      self._tconnect['pkey'] = DSSKey.from_private_key_file(
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/paramiko/pkey.py", line 232, in from_private_key_file
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      key = cls(filename=filename, password=password)
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/paramiko/dsskey.py", line 65, in __init__
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      self._from_private_key_file(filename, password)
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/paramiko/dsskey.py", line 226, in _from_private_key_file
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      self._decode_key(data)
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/paramiko/dsskey.py", line 242, in _decode_key
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      keylist = self._uint32_cstruct_unpack(data, "iiiii")
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****    File "/usr/local/lib/python3.10/dist-packages/paramiko/pkey.py", line 528, in _uint32_cstruct_unpack
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****      raise SSHException(str(e))
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  paramiko.ssh_exception.SSHException: unpack requires a buffer of 4 bytes
Feb  4 08:45:17 tagon weewx-vp2[1835630] ERROR weewx.reportengine:         ****  Generator terminated

Digging into __init__.py and some Google-fu, it looks like __init__.py is only using the RSAkey method, not ed25519 or ecdsa. Paramiko supports both ecdsa and ed25519, so it should be possible to add those as supported key types. For now, using a password instead of a key works fine, but I'd like to add a more secure key in if possible.

Is this an "in-work" thing perhaps? Or Is it as simple as adding ed25519 and ecdsa in the appropriate spots? My python is almost non-existent, unfortunately... :-(

Thanks,

Tom

[Tom Hogland]

I'd note that it appears that pysftp is mostly abandoned and doesn't support anything but rsa - the suggestions are all to just call paramiko directly, and rewriting weewx-sftp like that is WAY beyond my abilities. 

While this might be a great future-proof thing for weewx-sftp, it appears that it's beyond the quick fix I was hoping for. It may be better to use sshfs to mount the remote sftp directory using the better keys and then publish reports there directly, either via a skins report or rsync. Rsync would be preferred, but I can't rsync directly to my host - ssh connections like that are locked down - and the rsync report needs a servername. I might try just using localhost as the server and a path, just to see if it'll work, otherwise a simple shell script and cron will probably be the answer. 

Is it possible to have multiple [Seasons] sections and have them all run? Or just copy /seasons to /seasons2 and add [Seasons2] into the config?