Secure FTP from ftpupload call?

642 views
Skip to first unread message

Dan Klein

unread,
Sep 21, 2014, 7:57:10 AM9/21/14
to weewx...@googlegroups.com
My web hosting is locked to only allow secure FTP. I have weewx publishing correctly to PWS and can see the FTP service trying to connect to my web host in an insecure manner, only to fail.
I have then looked to modify wxengine to use FTP_TLS as the inherited class of FTP instead. I have had no luck and the debug logs seem only to tell me that it continues to fail to connect. However I can correctly sftp from my raspberry pi to the web host.
Any ideas? and has anyone built secure FTP service for weewx before and got it to work?
Many thanks.

Thomas Keffer

unread,
Sep 21, 2014, 9:30:50 AM9/21/14
to weewx-user
Hi, Dan

Two things to try:

1. The TLS extension to FTP only comes with Python 2.7. Perhaps you are using an earlier version of Python?

2. Did you remember to call prot_p()?

>>> from ftplib import FTP_TLS
>>> ftps = FTP_TLS('ftp.python.org')
>>> ftps.login()           # login anonymously before securing control channel
>>> ftps.prot_p()          # switch to secure data connection
>>> ftps.retrlines('LIST') # list directory content securely

​This should really be part of weewx. I'll put a ticket in for it.​

​-tk

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Dan Klein

unread,
Sep 21, 2014, 9:42:42 AM9/21/14
to weewx...@googlegroups.com
Tom,
Thanks for this - running 2.7.3 and yes had forgotten to switch to a secure data channel with prot_p(). Will go back and try the code again...
Yes, it would be good if this was included in future builds with a config switch for secure/unsecure. Thanks for your help and raising the ticket.
Dan

Thomas Keffer

unread,
Oct 4, 2014, 11:24:58 AM10/4/14
to weewx-user
Dan,

I'd like to put this patch in the code base, but I wanted to check with you to see if it worked.

-tk

Dan Klein

unread,
Oct 9, 2014, 6:41:02 AM10/9/14
to weewx...@googlegroups.com
Thomas,
Sorry for being slow but been away. Still haven't got it to work properly yet and seem to have some stability issues more generally with the Pi implementation - although it is a clean build. Weewx seems to abort after about a week with limited debug info.
Am spending this weekend sorting out the build.
Best
Dan

--
You received this message because you are subscribed to a topic in the Google Groups "Weewx user's group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/lC5wFBhg2g8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

Rob Cranfill

unread,
Jul 9, 2015, 1:47:03 PM7/9/15
to weewx...@googlegroups.com
I too must use sftp to upload to my server, and cannot get Weewx to work with the current version (3.1.0), which does appear to have code in it to support secure connections - it just doesn't work. The codebase, as it stands, fails on the attempted connection, with an EOFError.

I'm gonna hack on it (locally, that is), if no one else has found a solution.

Yes, I have Python 2.7.

/rob, new Weewx user

Thomas Keffer

unread,
Jul 9, 2015, 1:54:08 PM7/9/15
to weewx-user
Hi, Rob

It would help if you were a bit more specific than "it just doesn't work."

I have gotten to work with Linux FTP servers, but I have had problems with Microsoft. Indeed, it fails right after the user login. Do you know what kind of server you're trying to connect to?

-tk


--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

Rob Cranfill

unread,
Jul 9, 2015, 1:59:17 PM7/9/15
to weewx...@googlegroups.com
I can post my stack trace if you'd like. I guess I was wondering if there is anyone who *has* gotten it to work. Are you saying you have? This is with 'secure_ftp = True', right?

Here's the syslog from a slightly-modified version of the original code (I'm trying to track down the error)


Jul  9 08:50:26 x10pi weewx[16428]: cheetahgenerator: Generated 14 files for report StandardReport in 8.45 seconds
Jul  9 08:50:31 x10pi weewx[16428]: genimages: Generated 12 images for StandardReport in 4.62 seconds
Jul  9 08:50:31 x10pi weewx[16428]: reportengine: copied 0 files to /var/www/weewx
Jul  9 08:50:31 x10pi weewx[16428]: reportengine: Running report FTP
Jul  9 08:50:31 x10pi weewx[16428]: reportengine: Found configuration file /etc/weewx/skins/Ftp/skin.conf for report FTP
Jul  9 08:50:31 x10pi weewx[16428]: ftpupload: Attempting secure connection to robcranfill.net
Jul  9 08:50:31 x10pi weewx[16428]: ftpupload: about to connect to robcranfill.net:22
Jul  9 08:51:01 x10pi weewx[16428]: reportengine: Caught exception <type 'exceptions.EOFError'> in FtpGenerator: ''.
Jul  9 08:51:01 x10pi weewx[16428]: reportengine: exception type: 'EOFError()'
Jul  9 08:51:01 x10pi weewx[16428]: reportengine: local_root: '/var/www/weewx'.
Jul  9 08:51:01 x10pi weewx[16428]:         ****  Traceback (most recent call last):
Jul  9 08:51:01 x10pi weewx[16428]:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 214, in run
Jul  9 08:51:01 x10pi weewx[16428]:         ****      N = ftpData.run()
Jul  9 08:51:01 x10pi weewx[16428]:         ****    File "/usr/share/weewx/weeutil/ftpupload.py", line 99, in run
Jul  9 08:51:01 x10pi weewx[16428]:         ****      ftp_server.connect(self.server, self.port)
Jul  9 08:51:01 x10pi weewx[16428]:         ****    File "/usr/lib/python2.7/ftplib.py", line 135, in connect
Jul  9 08:51:01 x10pi weewx[16428]:         ****      self.welcome = self.getresp()
Jul  9 08:51:01 x10pi weewx[16428]:         ****    File "/usr/lib/python2.7/ftplib.py", line 210, in getresp
Jul  9 08:51:01 x10pi weewx[16428]:         ****      resp = self.getmultiline()
Jul  9 08:51:01 x10pi weewx[16428]:         ****    File "/usr/lib/python2.7/ftplib.py", line 200, in getmultiline
Jul  9 08:51:01 x10pi weewx[16428]:         ****      nextline = self.getline()
Jul  9 08:51:01 x10pi weewx[16428]:         ****    File "/usr/lib/python2.7/ftplib.py", line 186, in getline
Jul  9 08:51:01 x10pi weewx[16428]:         ****      if not line: raise EOFError
Jul  9 08:51:01 x10pi weewx[16428]:         ****  EOFError


I'm now trying to run some python code, based on the Weewx code, to find the right sequence of calls to make sftp work.

/rob

Rob Cranfill

unread,
Jul 9, 2015, 2:00:02 PM7/9/15
to weewx...@googlegroups.com
Oh, and the host is "1and1", a commercial web host with Linux servers of some sort.

 /rob

On Thursday, July 9, 2015 at 10:54:08 AM UTC-7, Tom Keffer wrote:

Thomas Keffer

unread,
Jul 9, 2015, 2:19:23 PM7/9/15
to weewx-user
I have definitely gotten it to work with a Linux host. Have not been able to get it to work with the MSFT host at my disposal (although, when it fails, it fails at the next step: logging in the user and password).

I suspect this is some kind of ssh credential problem.

-tk

Rob Cranfill

unread,
Jul 9, 2015, 8:58:31 PM7/9/15
to weewx...@googlegroups.com
Huh. I am beginning to think that the standard Python 2.7 library "ftplib" does not actually support username/password sftp. I'm a little rusty at Python - I'm a recently-retired Java programmer who's branching out - but in several hours of Googling I am able to find *zero* examples of Python code doing sftp using username/password login. I'd love to be shown to be wrong.

I wonder if you could have unintentionally had the proper certificates in place that facilitated your successful exchange?

 /rob

Rob Cranfill

unread,
Jul 10, 2015, 1:14:18 AM7/10/15
to weewx...@googlegroups.com

Not to flog a dead horse, but here is some code I put together that is basically what looks to me like the pertinent lines of weeutil/ftpupload.py - in the secure=True case - up to the point at which it fails for me, the .connect() call. If it were to work, it would print out "ftpupload: Secure connection to ....", but it doesn't get that far.

 -- asis.py

#
# Being an attempt at summarizing the FTP code run by
https://github.com/weewx/weewx/blob/master/bin/weeutil/ftpupload.py
# to see what's going on.
#
import ftplib

def run():

    server = 'robcranfill.net'
    port = 22
    passive = True
    user = 'xxxx';
    password = 'yyyy'

    try:
        FTPClass = ftplib.FTP_TLS
    except AttributeError:
        FTPClass = ftplib.FTP
        syslog.syslog(syslog.LOG_DEBUG, "ftpupload: Your version of Python does not support SFTP. Quitting.")
        return
       
    print("ftpupload: Attempting secure connection to {}...".format(server))

    try:
        ftp_server = FTPClass()
        ftp_server.connect(server, port)

#       if self.debug:
#           ftp_server.set_debuglevel(self.debug)
       
        ftp_server.login(user, password)
        ftp_server.set_pasv(passive)
   
        ftp_server.prot_p()
        print("ftpupload: Secure connection to {}".format(server))

    except ftplib.all_errors, e:
        print("ftpupload: Unable to connect or log into server: exception is '{}'".format(e))
        print("ftpupload: Unable to connect or log into server:      that is '{}'".format(repr(e)))


if __name__ == "__main__":
    run()

And here's what happens when I run it:

pi@x10pi ~/proj/weewx/sftp $ python asis.py
ftpupload: Attempting secure connection to robcranfill.net...
ftpupload: Unable to connect or log into server: exception is ''
ftpupload: Unable to connect or log into server:      that is 'EOFError()'


If you run the code, does it work for you? If so, then there's something funky up with my server, I guess. (But I can sftp to it from the command line, no problem).

Thanks for any thoughts,
/rob

Les Niles

unread,
Jul 10, 2015, 3:49:04 AM7/10/15
to weewx...@googlegroups.com
Hmm… I poked around at this a little...  sftp is not "secure ftp,” it’s actually “ssh file transfer protocol” — it runs over an ssh connection, on port 22 by default.  The python ftplib FTP_TLS runs ftp over an SSL socket, on the default ftp-control port 21.  This is ftps.  These are different protocols, two different approaches to securing file transfers.  If your server is just supporting sftp, the python library isn’t going to work, AFAICT.  

I use RSYNC in weewx.  It runs over an ssh connection, so is secure.  My ISP supports it.  And it’s potentially more efficient.

  -Les

Andy

unread,
Jul 10, 2015, 10:16:01 AM7/10/15
to weewx...@googlegroups.com
I am using rsync and a shared key to update my pages hosted at 1and1.
Andy

Rob Cranfill

unread,
Jul 10, 2015, 6:38:51 PM7/10/15
to weewx...@googlegroups.com
Les,

Yes, thanks for looking into it. It does indeed look like Python 2.7, and thus Weewx, only supports "ftps", and not "sftp"; that would explain why I couldn't find much/any examples of how to do it!

I wonder if the Weewx docs should be amended to show this? The docs at http://www.weewx.com/docs/usersguide.htm#[[FTP]] say

secure_ftp

Set to True to use a secure FTP (SFTP) connection.

and I think we're agreed now that that's not the case (or, at best, is misleading)?

 /rob

Thomas Keffer

unread,
Jul 11, 2015, 9:24:43 AM7/11/15
to weewx-user
Les's analysis seems to be what's going on.

Still, I can get Rob's program to work by running it over port 21. Using

ftp_server.set_debuglevel(1)

and adding to the end

ftp_server.dir()
ftp_server.quit()

this is what I get:

ftpupload: Attempting secure connection to www.threefools.org...
*cmd* 'AUTH TLS'
*resp* '234 AUTH command ok. Expecting TLS Negotiation.'
*cmd* 'USER xxxx'
*resp* '331 Password required'
*cmd* 'PASS **********'
*resp* '230 User logged in.'
*cmd* 'PBSZ 0'
*resp* '200 PBSZ command successful.'
*cmd* 'PROT P'
*resp* '200 PROT command successful.'
ftpupload: Secure connection to www.threefools.org
*cmd* 'TYPE A'
*resp* '200 Type set to A.'
*cmd* 'PASV'
*resp* '227 Entering Passive Mode (50,62,160,16,196,92).'
*cmd* 'LIST'
*resp* '125 Data connection already open; Transfer starting.'
05-15-15  01:40PM       <DIR>          cgi-bin
05-15-15  01:40PM       <DIR>          error_docs
06-11-15  03:06PM       <DIR>          httpdocs
05-15-15  01:41PM       <DIR>          logs
*resp* '226 Transfer complete.'
None
*cmd* 'QUIT'
*resp* '221 Goodbye.'
221 Goodbye.

Rob Cranfill

unread,
Jul 17, 2015, 12:00:51 AM7/17/15
to weewx...@googlegroups.com
I think (perhaps) we all agree that the issue, if we can call it that, is that Weewx supports FTPS, and not SFTP. (I'm using, I hope, standard terminology here, as found at http://www.jscape.com/blog/bid/75602/Understanding-Key-Differences-Between-FTP-FTPS-and-SFTP or several other references what can be found by Googling "ftps vs sftp" or the like).

I have hacked some code that now allows Weewx to communicate with my 1and1 sftp-only server. (Well, 1and1 may support other protocols, too, but I want to use sftp).

My question: can I contribute this code to the greater good? I am new here, and don't mean to be presumptuous. And it does introduce a new dependency: I use 'pysftp' (http://pysftp.readthedocs.org/en/release_0.2.8/index.html), which in turn depends on 'paramiko'. And perhaps this is undesirable.

Suggestions welcome.

 /rob

Thomas Keffer

unread,
Jul 17, 2015, 12:12:00 AM7/17/15
to weewx-user
Hi, Rob

I originally wrote the ftpupload code because I did not want to depend on paramiko. It's a fine package, but I hated to introduce yet another dependency, especially for such a low-level utility. 

The easy answer to adding pysftp would be "no problem," but I really hesitate at the thought of introducing two new dependencies. We could make it conditional --- install the two if you want SFTP, don't if you don't --- but I know from experience that there will be a lot of support questions.

It would be best if ftpupload could be extended to support SFTP. But, if that's not possible, let's go with a conditional dependency.

The best way to contribute code is to get a GitHub account, fork weewx, make your additions, then request a pull.

We also need to clean up the terminology and stop calling what ftpupload does 'SFTP'.

Thanks in advance!

-tk

Rob Cranfill

unread,
Jul 17, 2015, 12:22:01 AM7/17/15
to weewx...@googlegroups.com
Cool, yes, I totally understand not wanting new dependencies. I will do as you suggest, and fork it to my GitHub account. This may take me a little while -  I told myself I wouldn't spend too many sunny summer days hacking code! ;-)

 /rob


</rob>

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/lC5wFBhg2g8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

Geoff Cusick

unread,
Oct 6, 2017, 6:18:29 PM10/6/17
to weewx-user
Andy,

I know this was a while back, but can you give me any pointers to setting up rsync between weewx and a 1and1 server - they’ve recently disabled ‘ordinary’ ftp, and my website is stuck on 24 September.

Any hints much appreciated.

Thanks
Geoff

Robert Isbell

unread,
Oct 22, 2017, 3:42:12 PM10/22/17
to weewx-user
Andy and Geoff,
I too am in the same position as Geoff. Is there a "Howto" somewhere for setting up rsync to work with 1&1.

Andy

unread,
Oct 23, 2017, 9:05:56 AM10/23/17
to weewx-user
Rsync is same credentials as ssh for 1&1

Geoff Cusick

unread,
Oct 23, 2017, 9:13:14 AM10/23/17
to weewx-user
Apologies that (a) I didn't post the solution immediately I found it, and (b) it's taken me a while to respond.  But I did sort this out.

The critical step is to set up passwordless SSH access to your FTP account on 1and1.  To do this, you need to know:

  • The server address for your FTP account.  In my case, this is something like:
    homexxxxxxxx.1and1-data.host
  • The user name for this account, something like uXXXXXX
  • The password for the account.

The next step is to generate an SSH key pair for your local machine, and upload the public part to the 1and1 server.  There's a helpful article at https://linuxconfig.org/passwordless-ssh that sets out the series of steps.  The location for the ssh-copy-id command is uXXX...@homexxxxxxxx.1and1-data.host


Once you've completed this bit, and verified that you can sss to uXXX...@homexxxxxxxx.1and1-data.host with no password request, you can set up weewx to use rsync to transfer the relevant files from weewx to 1and1.


The relevant part of weewx.conf is:


******************


[[RSYNC]]
        # rsync'ing to a webserver is treated as just another report
        skin = Rsync
       
        # If you wish to use rsync, you must configure passwordless ssh using
        # public/private key authentication from the user account that weewx
        # runs as to the user account on the remote machine where the files
        # will be copied.
        #
        # The following three lines determine where files will be sent.
        #server = replace with the rsync server name, e.g, www.threefools.org
         server = homexxxxxxxx.1and1-data.host
        #path = replace with the rsync destination directory (e.g., /weather)
         path = webroot/weather/
        #user = replace with the rsync username
         user = uXXXXXXX
       
        # Rsync can be configured to remove files from the remote server if
        # they don't exist under HTML_ROOT locally. USE WITH CAUTION: if you
        # make a mistake in the remote path, you could could unintentionally
        # cause unrelated files to be deleted. Set to 1 to enable remote file
        # deletion, zero to allow files to accumulate remotely.
        delete = 0



***********

Just to explain, the root for my web server is at /webroot on the1and1 server, and all the weewx files are under weather, below that.  I found that I had to omit the leading '/' on /webroot.


Then you need to stop and restart weewx to implement the change to weewx.conf.


Good luck!!


Geoff

dhin...@djhindley.com

unread,
Oct 4, 2018, 1:03:44 PM10/4/18
to weewx-user
I have recently signed up to 1&1 to host my weather website and want to get weewx to upload files to it, allowing for the fact that they now use SFTP, not plain FTP, so I can't use the FTP weewx facility. I tried to use Matthew Wall's weew-sftp extension, but couldn't manage to get the python pysftp module installed, so I thought I would try Rsync.   I followed the steps below and can successfully ssh to my 1&1 FTP account without needing to type in a password.

However, when I amend Rysc configuration in weewx.conf as per below, Rsync seems to fail.  My syslog is below.  The Rsync error is at the end, but I included earlier errors in the syslog, but not sure if they are related to the Rsync issue or not.

Any ideas on what is going wrong?

Many Thanks

David.

Oct  4 17:51:17 WeatherPi weewx[10224]: manager: Daily summary version is 2.0
Oct  4 17:51:17 WeatherPi weewx[10224]: reportengine: Caught unrecoverable exception in generator weewx.cheetahgenerator.CheetahGenerator
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****  No module named forecast
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****  Traceback (most recent call last):
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 239, in run
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      obj.start()
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 273, in start
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      self.run()
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 155, in run
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      self.initExtensions(gen_dict[section_name])
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 197, in initExtensions
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      class_ = weeutil.weeutil._get_object(x)
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/share/weewx/weeutil/weeutil.py", line 1132, in _get_object
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      mod = __import__(module)
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****  ImportError: No module named forecast
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****  Generator terminated
Oct  4 17:51:17 WeatherPi weewx[10224]: reportengine: Caught unrecoverable exception in generator weewx.imagegenerator.ImageGenerator
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****  'skin'
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****  Traceback (most recent call last):
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 239, in run
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      obj.start()
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 273, in start
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      self.run()
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/share/weewx/weewx/imagegenerator.py", line 31, in run
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      self.setup()
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/share/weewx/weewx/imagegenerator.py", line 44, in setup
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      self.skin_dict['skin']))
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/lib/python2.7/dist-packages/configobj.py", line 567, in __getitem__
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      val = dict.__getitem__(self, key)
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****  KeyError: 'skin'
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****  Generator terminated
Oct  4 17:51:17 WeatherPi weewx[10224]: reportengine: Caught unrecoverable exception in generator weewx.reportengine.CopyGenerator
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****  'skin'
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****  Traceback (most recent call last):
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 239, in run
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      obj.start()
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 273, in start
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      self.run()
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 417, in run
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      self.skin_dict['skin']))
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****    File "/usr/lib/python2.7/dist-packages/configobj.py", line 567, in __getitem__
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****      val = dict.__getitem__(self, key)
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****  KeyError: 'skin'
Oct  4 17:51:17 WeatherPi weewx[10224]:         ****  Generator terminated
Oct  4 17:51:17 WeatherPi weewx[10224]: reportengine: Running report RSYNC
Oct  4 17:51:17 WeatherPi weewx[10224]: reportengine: Found configuration file /etc/weewx/skins/Rsync/skin.conf for report RSYNC
Oct  4 17:51:17 WeatherPi weewx[10224]: rsyncupload: [['rsync', '--archive', '--stats', '-e ssh', '/var/www/html/weewx/', 'xxxx...@homexxxxxxxxxx.1and1-data.host:.']] reported errors: Host key verification failed.. rsync: connection unexpectedly closed (0 bytes received so far) [sender]. rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
Oct  4 17:51:17 WeatherPi weewx[10224]: rsyncupload: rsync executed in 0.41 seconds

gjr80

unread,
Oct 5, 2018, 2:09:58 AM10/5/18
to weewx-user
I presume that your working passwordless ssh is from your WeatherPi machine. Which user on WeatherPi has passwordless ssh? If it is, for example, user pi and WeeWX is being run as root then I expect WeeWX rsync will fail with the error shown. You need to make sure that the account that runs WeeWX has passwordless ssh access. So you may need to change the user that runs WeeWX or give the account that runs WeeWX (likely root unless you explicitly changed it) passwordless ssh access.

Regarding pysftp, what did you do? what error did you come across? I just tried installing with pip on my RPI running raspbian stretch:

$ sudo pip install pysftp

and received a dependency error on package libffi. Installed libffi using:

$ sudo ap-get install libffi-dev

then ran the pysftp install again and it installed without error, it was slow but eventually it got there. If that does not solve your pysftp issue post the command you used to install it and the output you receive.

Gary
Oct  4 17:51:17 WeatherPi weewx[10224]: rsyncupload: [['rsync', '--archive', '--stats', '-e ssh', '/var/www/html/weewx/', 'xxxxxxxx@homexxxxxxxxxx.1and1-data.host:.']] reported errors: Host key verification failed.. rsync: connection unexpectedly closed (0 bytes received so far) [sender]. rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]

David Hindley

unread,
Oct 5, 2018, 6:20:14 AM10/5/18
to weewx...@googlegroups.com
Thanks, Gary.

I think my problems with pysftp are something to do with using Wheezy on a RPi and ver 2.7 of Python. That seems to have an old version of pip, so I managed to update that to pip ver 18.0, but then pip shows several nasty looking errors when I try to install pysftp.  The syslog is below, but I suspect I am going to have to upgrade my Pi to raspbian stretch  - any tips on making that process quick, allowing for a weewx install would be appreciated.

David.

pi@WeatherPi ~ $ sudo pip install pysftp
Collecting pysftp
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:369: SNIMissingWarning: An HTTPS request has been made, but the SNI (Se                                                                                              rver Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, wh                                                                                              ich can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readth                                                                                              edocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:160: InsecurePlatformWarning: A true SSLContext object is not available                                                                                              . This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of                                                                                               Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:160: InsecurePlatformWarning: A true SSLContext object is not available                                                                                              . This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of                                                                                               Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Collecting paramiko>=1.17 (from pysftp)
  Downloading https://files.pythonhosted.org/packages/cf/ae/94e70d49044ccc234bfdba20114fa947d7ba6eb68a2e452d89b920e62227/paramiko-2.4.2-py2.py3                                                                                              -none-any.whl (193kB)
    100% |████████████████████████████████| 194kB 2.1MB/s
Collecting cryptography>=1.5 (from paramiko>=1.17->pysftp)
  Downloading https://files.pythonhosted.org/packages/22/21/233e38f74188db94e8451ef6385754a98f3cad9b59bedf3a8e8b14988be4/cryptography-2.3.1.tar                                                                                              .gz (449kB)
    100% |████████████████████████████████| 450kB 2.0MB/s
Collecting pynacl>=1.0.1 (from paramiko>=1.17->pysftp)
  Downloading https://files.pythonhosted.org/packages/61/ab/2ac6dea8489fa713e2b4c6c5b549cc962dd4a842b5998d9e80cf8440b7cd/PyNaCl-1.3.0.tar.gz (3                                                                                              .4MB)
    100% |████████████████████████████████| 3.4MB 953kB/s
  Installing build dependencies ... error
  Complete output from command /usr/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-FXioj2 --no-warn-script-                                                                                              location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel "cffi>=1.4.1; python_implementation != 'PyPy'":
  Collecting setuptools
  /usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:369: SNIMissingWarning: An HTTPS request has been made, but the SNI (                                                                                              Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate,                                                                                               which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.read                                                                                              thedocs.io/en/latest/advanced-usage.html#ssl-warnings
    SNIMissingWarning
  /usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:160: InsecurePlatformWarning: A true SSLContext object is not availab                                                                                              le. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version                                                                                               of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
    InsecurePlatformWarning
  /usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:160: InsecurePlatformWarning: A true SSLContext object is not availab                                                                                              le. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version                                                                                               of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
    InsecurePlatformWarning
    Downloading https://files.pythonhosted.org/packages/96/06/c8ee69628191285ddddffb277bd5abdf769166e7a14b867c2a172f0175b1/setuptools-40.4.3-py                                                                                              2.py3-none-any.whl (569kB)
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/fc/e9/05316a1eec70c2bfc1c823a259546475bd7636ba6d27ec80575da523bc34/wheel-0.32.1-py2.py                                                                                              3-none-any.whl
  Collecting cffi>=1.4.1
    Downloading https://files.pythonhosted.org/packages/e7/a7/4cd50e57cc6f436f1cc3a7e8fa700ff9b8b4d471620629074913e3735fb2/cffi-1.11.5.tar.gz (                                                                                              438kB)
  Collecting pycparser (from cffi>=1.4.1)
    Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.g                                                                                              z (158kB)
  Building wheels for collected packages: cffi, pycparser
    Running setup.py bdist_wheel for cffi: started
    Running setup.py bdist_wheel for cffi: finished with status 'error'
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-vvzHKF/cffi/setup.py';f=getattr(                                                                                              tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-                                                                                              wheel-pjsewI --python-tag cp27:
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    running bdist_wheel
    running build
    running build_py
    creating build
    creating build/lib.linux-armv7l-2.7
    creating build/lib.linux-armv7l-2.7/cffi
    copying cffi/lock.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/cffi_opcode.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/setuptools_ext.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/__init__.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/model.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/backend_ctypes.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/cparser.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/vengine_cpy.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/verifier.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/commontypes.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/error.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/vengine_gen.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/api.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/recompiler.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/ffiplatform.py -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/_cffi_include.h -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/parse_c_type.h -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/_embedding.h -> build/lib.linux-armv7l-2.7/cffi
    copying cffi/_cffi_errors.h -> build/lib.linux-armv7l-2.7/cffi
    running build_ext
    building '_cffi_backend' extension
    creating build/temp.linux-armv7l-2.7
    creating build/temp.linux-armv7l-2.7/c
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/incl                                                                                              ude/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-armv7l-2.7/c/_cffi_backend.o
    c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory
    compilation terminated.
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
    Failed building wheel for cffi
    Running setup.py clean for cffi
    Running setup.py bdist_wheel for pycparser: started
    Running setup.py bdist_wheel for pycparser: finished with status 'done'
    Stored in directory: /root/.cache/pip/wheels/f2/9a/90/de94f8556265ddc9d9c8b271b0f63e57b26fb1d67a45564511
  Successfully built pycparser
  Failed to build cffi
  Installing collected packages: setuptools, wheel, pycparser, cffi
    Running setup.py install for cffi: started
      Running setup.py install for cffi: finished with status 'error'
      Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-vvzHKF/cffi/setup.py';f=getatt                                                                                              r(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/                                                                                              pip-record-3KgNZC/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-FXioj2 --compile:
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-armv7l-2.7
      creating build/lib.linux-armv7l-2.7/cffi
      copying cffi/lock.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/cffi_opcode.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/setuptools_ext.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/__init__.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/model.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/backend_ctypes.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/cparser.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/vengine_cpy.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/verifier.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/commontypes.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/error.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/vengine_gen.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/api.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/recompiler.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/ffiplatform.py -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/_cffi_include.h -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/parse_c_type.h -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/_embedding.h -> build/lib.linux-armv7l-2.7/cffi
      copying cffi/_cffi_errors.h -> build/lib.linux-armv7l-2.7/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-armv7l-2.7
      creating build/temp.linux-armv7l-2.7/c
      gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/in                                                                                              clude/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-armv7l-2.7/c/_cffi_backend.o
      c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory
      compilation terminated.
      error: command 'gcc' failed with exit status 1

      ----------------------------------------
  Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-vvzHKF/cffi/setup.py';f=getattr(tokenize, 'open', open                                                                                              )(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-3KgNZC/instal                                                                                              l-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-FXioj2 --compile" failed with error code 1 in /tmp/pip-install-vvz                                                                                              HKF/cffi/

  ----------------------------------------
Command "/usr/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-FXioj2 --no-warn-script-location --no-binary :                                                                                              none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel "cffi>=1.4.1; python_implementation != 'PyPy'"" failed with error cod                                                                                              e 1 in None


Oct  4 17:51:17 WeatherPi weewx[10224]: rsyncupload: [['rsync', '--archive', '--stats', '-e ssh', '/var/www/html/weewx/', 'xxxx...@homexxxxxxxxxx.1and1-data.host:.']] reported errors: Host key verification failed.. rsync: connection unexpectedly closed (0 bytes received so far) [sender]. rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/lC5wFBhg2g8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

David Hindley

unread,
Oct 5, 2018, 6:25:20 AM10/5/18
to weewx...@googlegroups.com
Gary - regarding the ssh point - which option would you recommend - change the user that runs weewx to pi, or create ssh keys for the root user (which I believe currently runs weewx)?

David.

On Fri, 5 Oct 2018 at 07:10, gjr80 <gjrod...@gmail.com> wrote:
Oct  4 17:51:17 WeatherPi weewx[10224]: rsyncupload: [['rsync', '--archive', '--stats', '-e ssh', '/var/www/html/weewx/', 'xxxx...@homexxxxxxxxxx.1and1-data.host:.']] reported errors: Host key verification failed.. rsync: connection unexpectedly closed (0 bytes received so far) [sender]. rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/lC5wFBhg2g8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

gjr80

unread,
Oct 5, 2018, 9:01:34 AM10/5/18
to weewx-user
On Friday, 5 October 2018 20:20:14 UTC+10, David Hindley wrote:
Thanks, Gary.

I think my problems with pysftp are something to do with using Wheezy on a RPi and ver 2.7 of Python. That seems to have an old version of pip, so I managed to update that to pip ver 18.0, but then pip shows several nasty looking errors when I try to install pysftp.  The syslog is below, but I suspect I am going to have to upgrade my Pi to raspbian stretch  - any tips on making that process quick, allowing for a weewx install would be appreciated.


Am no great expert but the SSL warnings are exactly that, warnings. You do have the libffi error:

Package libffi was not found in the pkg-config search path.

You could try to installing libffi-dev as I suggested, though maybe a bit mute if upgrading to Stretch.

Regarding upgrading to Stretch. Its fairly straightforward, backing up your WeeWX config and data should make re-installing WeeWX fairly painless. The basic steps are:
  1. stop WeeWX
  2. backup you WeeWX data, assuming you are using SQLite you just need to backup any .sdb files in /var/lib/weewx
  3. backup up you WeeWX confg and skins, ie everything in /etc/weewx
  4. backup your bin/user directory, /usr/share/weewx/user in your case
  5. download your preferred raspbian stretch image, burn it to a SD card and boot your RPi from the new image
  6. setup RPi as required
  7. install WeeWX and get a minimal install running with the simulator
  8. stop WeeWX and copy your old database over the new one, restart WeeWX
  9. start adding in any extensions/customisations etc, refer back to your backup /etc/weewx/weewx.conf and skin.conf files for any config details/customisations
Some folks might prefer to take a shorter approach, ie just copy all the WeeWX directories from the old to the new, that may work but it can also tangle you in knots if it goes awry.

Gary

gjr80

unread,
Oct 5, 2018, 9:11:18 AM10/5/18
to weewx-user
Really up to you, I am sure some folks will say you are best having a daemon such as WeeWX run as a user other than root. I run WeeWX as root and I suspect most folks do too. I am happy that the risks are fairly minimal. The steps to change your system such that WeeWX is run as a user other than root are fairly straightforward and are covered here in the wiki. Of course if you continue to run WeeWX as root you probably need to setup root with passwordless ssh, arguably that is more involved than changing the WeeWX user, but then again you have already done that once so I am sure it can be easily done again :)

If I was doing things again I would do the same as I have now and leave WeeWX run as root and give root passwordless ssh. Not going to argue with anyone who disagrees though.

Gary

David Hindley

unread,
Oct 5, 2018, 11:27:06 AM10/5/18
to weewx...@googlegroups.com
Thanks, Gary.  Will explore both options and decide which looks easier. 

--

David Hindley

unread,
Oct 5, 2018, 11:33:44 AM10/5/18
to weewx...@googlegroups.com
Thanks again Gary, for your response. Very helpful as always. Upgrade looks a bit scary, but will give it a go. 

Reply all
Reply to author
Forward
0 new messages