Weewx SFTP errors

[Invisible Man]

Hello, 
I have recently enable sftp on Weewx, to upload the weewx web pages to a remote site.

Before that, I used ftp, and it worked no problem, but I'm have errors with SFTP.

- I'm using Weewx 4.9.1

- Python 3.7.3

- I can connect by hand to the remote sftp no problem

- Not sure if I should be using passive or non passive for SFTP.

```

Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine: ftpgenerator: (2): caught exception '<class 'socket.timeout'>': timed out
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 437, in run
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****      n = ftp_data.run()
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weeutil/ftpupload.py", line 175, in run
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****      ftp_server.connect(self.server, self.port)
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****    File "/usr/lib/python3.7/ftplib.py", line 155, in connect
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****      self.welcome = self.getresp()
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****    File "/usr/lib/python3.7/ftplib.py", line 236, in getresp
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****      resp = self.getmultiline()
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****    File "/usr/lib/python3.7/ftplib.py", line 226, in getmultiline
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****      nextline = self.getline()
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****    File "/usr/lib/python3.7/ftplib.py", line 204, in getline
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****      line = self.file.readline(self.maxline + 1)
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****    File "/usr/lib/python3.7/socket.py", line 589, in readinto
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****      return self._sock.recv_into(b)
Jan 10 15:52:16 vegan python3[7637]: weewx[7637] ERROR weewx.reportengine:         ****  socket.timeout: timed out
```

this is my weewx.conf file:

```

[[FTP]]
        skin = Ftp
        enable = True
       
        # FTP'ing the results to a webserver is treated as just another report,
        # albeit one with an unusual report generator!
        #
        # If you wish to use FTP, uncomment and fill out the next four lines:
        user = CENSORED
        password = CENSORED
        server = CENSORED
        path = /
       
        # If you wish to upload files from something other than what HTML_ROOT
        # is set to above, then reset it here:
        HTML_ROOT = external_html
       
        # Most FTP servers use port 21, but if yours is different, you can
        # change it here
        port = 22
       
        # Set to 1 to use passive mode, zero for active mode:
        passive = 1
       
        # How many times to try to transfer a file before giving up:
        max_tries = 5
       
        # Set to True for a secure FTP (SFTP) connection. Not all servers
        # support this.
        secure_ftp = True
```

[Invisible Man]

A user suggests to use Rsync instead, as Rsync is running through SSH for weewx.

see http://www.weewx.com/docs/usersguide.htm#config_RSYNC

"Fast, efficient, and secure, it does an incremental update, that is, it only synchronizes those parts of a file that have changed, saving the outgoing bandwidth of your Internet connection.

If you wish to use rsync, you must configure passwordless ssh using public/private key authentication from the user account that WeeWX runs, to the user account on the remote machine where the files will be copied."

Indeed an option to try. I'll switch to that.

[Invisible Man]

As suggested, I used RSync instead of SFTP , and it works just fine :)

[Peter Walker]

I'm interested in this topic.

I've tried and tried to get Weewx to write files to my server, firstly via FTP, with no success, (Ionos helpdesk said, after a few calls that FTP had been disabled on my server - it came back with an error 550 - but different guys said slightly conflicting things, but all agreed that FTP wasn't secure) and then using RSYNC. Again, I had no success by configuring weewx.conf - the error message was concerning the public and private keys (host key verification failed - rsync upload unexplained error (code 255).. I was trying as root, and I can SSH into the server without a password having set up the keys from the Raspberry Pi. I've done a botch to get round it by running rsync as a cron job at 01 and 31 past each hour and that works OK. "rsync -a /var/www/html/weewx/ ro...@nnn.nnn.nn.nnn:/var/www/.../weewx/". I would like to get to grips with this at some stage, but I've got other stuff which is higher priority now that the weather station is actually writing to the server. It's a retirement project so all for my own fun/edification.

[vince]

In the absence of any logs or console transcripts we really can't help you...

[Peter Walker]

No, I'm aware of that. Given that I've got the site working in a very non-purist way, I shall let if get on with it for the time being as I've got other things which are higher priority at the moment. When I come back to tackle it again, I shall certainly provide much more info.

[matthew wall]

On Wednesday, February 1, 2023 at 2:12:39 PM UTC-5 walke...@googlemail.com wrote:

> Again, I had no success by configuring weewx.conf - the error message was concerning the public and 

> private keys (host key verification failed - rsync upload unexplained error (code 255).. I was trying as root, 

> and I can SSH into the server without a password having set up the keys from the Raspberry Pi. I've done a 

> botch to get round it by running rsync as a cron job at 01 and 31 past each hour and that works OK. 

> "rsync -a /var/www/html/weewx/ ro...@nnn.nnn.nn.nnn:/var/www/.../weewx/".

rsync is a bit fiddly.  it depends on ssh for the actual connections, and ssh, though powerful, has so many options that you've got to get it just right.  since rsync uses ssh, it is best to test your connection first using ssh.  if that works, then rsync will work; use ssh manually to test the connection, then use rsync manually to make sure you are syncing the right files/directories.

there are basically two contexts in which you can specify ssh options:

1) the ssh config file for the user who invokes ssh.  this is nominally located at ~/.ssh/config

2) command-line

i find it easiest to test using command-line, then when you have it working, but the options into the config file and remove them from command-line.  the ssh options you specify in the weewx config file are basically ssh command-line (or rsync command-line) options.

for example, the most basic invocation would be:

ssh user...@provider.example.com -i /path/to/id_rsa

you can specify other ssh options using the -o, for example:

ssh user...@provider.example.com -i /path/to/id_rsa -o StrictHostKeyChecking=no

if you put all of this into the ssh config file, then you can simplify:

Host provider

   HostName provider.example.com

   User username

   IdentityFile /path/to/id_rsa

   StrictHostKeyChecking no

so that the actual invocation is then:

ssh provider

and your rsync will just work, assuming that you got the source and destination paths right in your rsync config, but that is a separate topic.  (be sure that you understand the difference between /path/to/dir/ and /path/to/dir - it matters to rsync!)

note that permissions on the keyfile must be readable by owner and not readable by anyone else.

one other thing: be sure to establish the ssh connection manually first, as the user who runs the crontab or weewx.  that way you can be sure that the remote host's key has been accepted.

m

[vince]

On Wednesday, February 1, 2023 at 11:49:24 AM UTC-8 walke...@googlemail.com wrote:

No, I'm aware of that. Given that I've got the site working in a very non-purist way, I shall let if get on with it for the time being as I've got other things which are higher priority at the moment. When I come back to tackle it again, I shall certainly provide much more info.

 

FWIW, the usual mistake people make is using one user (typically 'pi') for setting the keys up, editing weewx.conf to match, then being surprised when it fails.  The nuance there is that weewx runs as a different user (typically 'root').  You have to set up the $HOME/.ssh/config and keys for the user weewx runs as.

Basically the quickest test is to sudo to root (who weewx runs as) and run one command...

sudo bash

ssh remote_username date

(and see the remote system's date returned)

[jterr...@gmail.com]

Another way to go  is to install and use the weewx-SFTP generator : https://github.com/matthewwall/weewx-sftp

It is working very well with IONOS, OVHCloud and others...