New Install with weewx-sftp, Raspberry Pi Bookworm, Weewx 5.1

[Burns Fisher]

I just got this working and thought it might be useful to pass on some hints collected together here.

FIrst, REALLY do use a python virtual environment.  It will save much grief later.  The instructions for installing weewx under a venv are in the docs:  https://weewx.com/docs/5.1/quickstarts/pip/?h=virtual+environment

Now install pysftp.  This package is a bit old and lots of net searches tell you to use something else, but it works fine once you get it in.  The key is that on the latest Raspbian you need to install this in a virtual environment.  You can't install stuff system wide (without really knowing what you are doing).  You do this by

- be sure the virtual environment is activated in your terminal process.  (The prompt will start (weewx-venv).

-pip install --user --break-system-packages pysftp

  (Don't be scared of "break system packages" if you also say -user.  If you don't say -user, this overrides the checks and installs it system wide.  It may be fine, but not recommended)

-feel free to let it start and make sure it is working, setting up the database, etc.  If you have an old database from a previous incarnation you can just drop it in the right place, assuming it is not corrupted.

- stop weewx using the command 'systemctl stop weewx'

- Now install weewx-sftp.  The readme in https://github.com/matthewwall/weewx-sftp is a bit out of date, as are a couple other things.  Get the zip file as the readme says, but then the correct command to install it on these newer versions is

              weectl extension install weewx-sftp.zip

-DO NOT start weewx back up yet.  First you will find some stuff in sftp.py that does not work on the latest of everything.  sftp.py is in ~/weew-data/bbin/user assuming you did everything based on your login directory.  Edit it, and everywhere you see cPickle (or maybe cpickle) change it to just pickle.  This requirement must have been fairly recent since Matthew Wall did a checkin that was supposed to make this work for Pip V3.

-Change the [SFTP] username, password, and destination directory to where you want it to be.

- restart weewx using 'systemctl start weewx'

If there is a problem, you can look at error messages using

journalctl -e  (the last page or so of the log)  or

journalctl -n 100 (number of your choice...show the last 100 lines of the log)

==============================

I hope this will help someone.  Honestly, there is nothing here that is not likely to be somewhere else, but I thought it would be nice to put it all together.  BTW, I used ChatGPT to help figure out some of this stuff.  It was not always right, but usually pointed me in the right direction.  And surprisingly it knows a lot about weewx!

[vince]

Sounds incorrect. Once you activate the venv you should not use —user nor —break-system-packages. You should just use pip3 without those options, which will install things into the venv tree. 

You might look into using the rsync uploader which is much faster and easier to set up, assuming your remote host supports passwordless ssh access.

[Tom -KQ5S]

I can vouch for this.  I was having trouble with my ISP and the FTP uploads.  I switched to rsync and now get uploads of 700,000 bytes in 1.5 sec or better.  700,000 bytes and around 50 files is a normal upload for my 5 minute cycle.

Tom 

[Burns Fisher (AMSAT)]

Hmm.  My reply disappeared.  Sorry if there comes to be a duplicate.  Anyway, thanks.  All I can say is that install did not work with venv and with or without --user unless I added --break.  Not a big expert here.  Just a one-time experience.

As to sftp, the server I am using ONLY allows upload via sftp.  I don't blame him.  I had a server on my Pi for a while, and got all sort of unwanted attention from around the world.

[Burns Fisher (AMSAT)]

Thanks.  I can only tell you what worked for me.  When I did install or install --user without the -break it would not install even in a venv.  

As to rsync, unfortunately, the server I use will ONLY accept ssh sftp.  In fact, I don't blame him...I used to have httpd as the web server daemon on my Pi and just opening that one port got be all kinds of unwanted attention from around the world.

[vince]

A quick test says otherwise for debian-12 (bookworm).  Note that I had to install the 'wheel' python module to quiet down some whining when installing pysftp in the next step.  That happens occasionally with some packages.

Just documenting it here to capture it for future google group searches folks might do when trying to do similar things....

# set up the venv

vagrant@bookworm:~$ python3 -m venv weewx-venv

# activate it

vagrant@bookworm:~$ source weewx-venv/bin/activate

# install weewx

(weewx-venv) vagrant@bookworm:~$ pip3 install weewx
Collecting weewx
  Downloading weewx-5.1.0-py3-none-any.whl (1.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 14.7 MB/s eta 0:00:00
Collecting CT3<4.0,>=3.1
  Downloading CT3-3.4.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.whl (911 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 911.4/911.4 kB 63.6 MB/s eta 0:00:00
Collecting Pillow>=5.2
  Downloading pillow-11.3.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 48.3 MB/s eta 0:00:00
Collecting PyMySQL<2.0,>=1.0
  Downloading PyMySQL-1.1.1-py3-none-any.whl (44 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.0/45.0 kB 14.9 MB/s eta 0:00:00
Collecting configobj<6.0,>=5.0
  Downloading configobj-5.0.9-py2.py3-none-any.whl (35 kB)
Collecting ephem<5.0,>=4.1
  Downloading ephem-4.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 55.5 MB/s eta 0:00:00
Collecting pyserial<4.0,>=3.4
  Downloading pyserial-3.5-py2.py3-none-any.whl (90 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.6/90.6 kB 19.7 MB/s eta 0:00:00
Collecting pyusb<2.0.0,>=1.0.2
  Downloading pyusb-1.3.1-py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.5/58.5 kB 22.5 MB/s eta 0:00:00
Installing collected packages: pyserial, ephem, pyusb, PyMySQL, Pillow, CT3, configobj, weewx
Successfully installed CT3-3.4.0 Pillow-11.3.0 PyMySQL-1.1.1 configobj-5.0.9 ephem-4.2 pyserial-3.5 pyusb-1.3.1 weewx-5.1.0

# install wheel

(weewx-venv) vagrant@bookworm:~$ pip3 install wheel
Collecting wheel
  Downloading wheel-0.45.1-py3-none-any.whl (72 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 72.5/72.5 kB 1.5 MB/s eta 0:00:00
Installing collected packages: wheel
Successfully installed wheel-0.45.1

# install pysftp

(weewx-venv) vagrant@bookworm:~$ pip3 install pysftp

Collecting pysftp
  Downloading pysftp-0.2.9.tar.gz (25 kB)
  Preparing metadata (setup.py) ... done
Collecting paramiko>=1.17
  Downloading paramiko-3.5.1-py3-none-any.whl (227 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.3/227.3 kB 4.9 MB/s eta 0:00:00
Collecting bcrypt>=3.2
  Downloading bcrypt-4.3.0-cp39-abi3-manylinux_2_34_x86_64.whl (284 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 284.2/284.2 kB 39.3 MB/s eta 0:00:00
Collecting cryptography>=3.3
  Downloading cryptography-45.0.5-cp311-abi3-manylinux_2_34_x86_64.whl (4.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 51.6 MB/s eta 0:00:00
Collecting pynacl>=1.5
  Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (856 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 856.7/856.7 kB 58.2 MB/s eta 0:00:00
Collecting cffi>=1.14
  Downloading cffi-1.17.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (467 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 467.2/467.2 kB 56.5 MB/s eta 0:00:00
Collecting pycparser
  Downloading pycparser-2.22-py3-none-any.whl (117 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.6/117.6 kB 36.4 MB/s eta 0:00:00
Building wheels for collected packages: pysftp
  Building wheel for pysftp (setup.py) ... done
  Created wheel for pysftp: filename=pysftp-0.2.9-py3-none-any.whl size=15496 sha256=349bc3296f9ec2230a72e66c52fbd3cd8a3af8bc8c5a1d2e842877472319272d
  Stored in directory: /home/vagrant/.cache/pip/wheels/f2/e1/a7/80af9a419febfbac5d549441b79a91583714f4abb5de47f6f3
Successfully built pysftp
Installing collected packages: pycparser, bcrypt, cffi, pynacl, cryptography, paramiko, pysftp
Successfully installed bcrypt-4.3.0 cffi-1.17.1 cryptography-45.0.5 paramiko-3.5.1 pycparser-2.22 pynacl-1.5.0 pysftp-0.2.9

# list the pip modules installed within the venv

(weewx-venv) vagrant@bookworm:~$ pip3 list
Package      Version
------------ -------
bcrypt       4.3.0
cffi         1.17.1
configobj    5.0.9
cryptography 45.0.5
CT3          3.4.0
ephem        4.2
paramiko     3.5.1
pillow       11.3.0
pip          23.0.1
pycparser    2.22
PyMySQL      1.1.1
PyNaCl       1.5.0
pyserial     3.5
pysftp       0.2.9
pyusb        1.3.1
setuptools   66.1.1
weewx        5.1.0
wheel        0.45.1