sftp on 3.9.2, RPI3B -- not working with several methods

[Messy Potamia]

--

In summary, the Secure FTP capability is indicated in weewx.conf (& the caveat comment seems satisfied with testing of Filezilla's success), however in weewx's native install it does not work.

--

First I tried the method suggested in weewx.conf and the weewx guide: I enabled ssh/sftp with my provider (one.com), followed their instructions, tested it with filezilla, and I connect and upload/dnload just fine. Here's one.com's instructions on the matter:

SSH & SFTP Administration

SSH increases the security on your web space by encrypting your web space connections. With SSH enabled, you can transfer files securely via SFTP or connect to your web space directly from a terminal.

Allow SSH & SFTP access Off On

To log in with SSH & SFTP, use this info:

Host: ssh.<redacted>.net

Username: <redacted>.net

Port: 22

Then their faq on the subject has this:

Open your SFTP client, and enter the following details, replacing yourdomain.com with your own domain:

• Host: sftp.yourdomain.com
• Username: yourdomain.com
• Password: the password you have chosen for SFTP
• Port: 22

Click the connect button in your client to log in. If it is the first time you login using SFTP on this computer, you need to approve the connection.

Note: I have verified that with filezilla, both sftp.mydomain.net and ssh.mydomain.net, using my ssh/sftp password, connect fine.

When I make the appropriate changes to weewx.conf, it is unable to connect. I tried a couple variations just to cross-validate disparities in their guide. No connect.

SO THEN I google "weewx 3.9.2 sftp" and come up with, among other things, MWall's https://github.com/matthewwall/weewx-sftp/blob/master/bin/user/sftp.py

to wit:

sudo pip install pysftp

which results in:

pi@RPI3:/home/weewx $ sudo pip install pysftp
Downloading/unpacking pysftp
  Downloading pysftp-0.2.9.tar.gz
  Running setup.py (path:/tmp/pip-build-6MvFgx/pysftp/setup.py) egg_info for package pysftp

    no previously-included directories found matching 'docs/_build'
Downloading/unpacking paramiko>=1.17 (from pysftp)
  Downloading paramiko-2.7.1-py2.py3-none-any.whl (206kB): 206kB downloaded
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 290, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1266, in prepare_files
    req_to_install.extras):
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2401, in requires
    dm = self._dep_map
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2597, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2630, in _compute_dependencies
    common = frozenset(reqs_for_extra(None))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2627, in reqs_for_extra
    if req.marker_fn(override={'extra':extra}):
  File "/usr/lib/python2.7/dist-packages/_markerlib/markers.py", line 113, in marker_fn
    return eval(compiled_marker, environment)
  File "<environment marker>", line 1, in <module>
NameError: name 'platform_system' is not defined

Storing debug log for failure in /root/.pip/pip.log

I tried a sudo apt-get install but that failed also.

pi@RPI3:/home/weewx $ sudo apt-get install pysftp
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package pysftp

So couldn't get to 1st base this inning.

In summary, the Secure FTP capability as indicated in weewx.conf (& the caveat comment seems satisfied with testing of Filezilla's success), in weewx's native install does not work. Not for me. And I believe I've double checked everything.

If it requires a hack to package(s) (.py) in weewx I'm comfortable doing that if the instructions are clear, concise, well written. Note, I'm not comfortable writing my own driver.

Thank you for helping.

Phil

[Messy Potamia]

I should amend the subj line from "sftp" to "secure_ftp" because I understand that sftp doesn't work, and I was intending to use the native secure_ftp feature.

[Messy Potamia]

Note, per another post the relevance of which I'm not certain, i added the following to the [[FTP]] section

   secure_ftp = True

   secure_data = False

Makes no difference, still won't connect. Filezilla using sftp connects fine, and the native weewx ftp connects fine. I'm trying to use secure ftp.

May 25 12:05:21 RPI3 weewx[26625]: copygenerator: copied 14 files to /home/weewx/public_html
May 25 12:05:41 RPI3 weewx[26625]: ftpupload: Unable to connect or log into server : timed out
May 25 12:06:01 RPI3 weewx[26625]: ftpupload: Unable to connect or log into server : timed out
May 25 12:06:21 RPI3 weewx[26625]: ftpupload: Unable to connect or log into server : timed out
May 25 12:06:21 RPI3 weewx[26625]: ftpupload: Attempted 3 times to connect to server sftp.<redacted>. Giving up.
May 25 12:06:21 RPI3 weewx[26625]: ftpgenerator: ftp'd 0 files in 60.69 seconds

On Monday, May 25, 2020 at 11:33:06 AM UTC+2, Messy Potamia wrote:

[Tom Keffer]

WeeWX supports FTPS, not SFTP. Two questions:

Which one does your server support?

If it supports FTPS, do you know what kind of FTP server it is? The PureFTP server tickles a bug in the Python FTP library, but we have a work around.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2e01a5ba-3dcc-4d8a-bf29-f4dc4fbda05f%40googlegroups.com.

[Messy Potamia]

They haven't replied what kind of server they use. They're a pretty
big hosting entity (one.com). I've been using their service for about
7 years, it includes my host name and about 20Gig of server space (I
only host two weewx wx systems on it, so use less than 2%) and a bunch
of other services which I'll never use. I enabled ssh/sftp per their
instructions, and my Filezilla now connects using sftp protocol. They
issue you a slightly different login (i.e. "ssh.webhostname.dom" or
"sftp.webhostname.dom" and you pick a different password from your FTP
password, other than that it's the same. I doubt this answers your
question.
I wish the mwall github resources was documented so I could see if it
even applied to me.
Thanks --

> You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/GdlSK3vTvYE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zECPae3wT8JPVg7SptswHa-b4_5LwN2%2BYRuG%3D3KnDACrVg%40mail.gmail.com.

[Tom Keffer]

If you connect via the command line command "ftp", it will usually tell you the server make.

BTW, let's switch to weewx-user for these questions. That way, others can contribute and learn. 

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAB7-S74uOEQa_4x54AgM3d6m9K6mp6MjsR9wpn%3Dz9PRbxADdYA%40mail.gmail.com.

[Messy Potamia]

Here's what I get when I connect via cmd line from my pi:

Password:
230 OK. Current restricted directory is /
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

Remote system type is UNIX. Duh.

----

(Aren't we already in weewx-user?)

>> To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

>> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2e01a5ba-3dcc-4d8a-bf29-f4dc4fbda05f%40googlegroups.com.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/GdlSK3vTvYE/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to weewx...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zECPae3wT8JPVg7SptswHa-b4_5LwN2%2BYRuG%3D3KnDACrVg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Tom Keffer]

(Aren't we already in weewx-user?)

So, we are! For some reason, gmail didn't tag it 'weewx' like it usually does.

From your response, I take it you are using SFTP? If so, weewx does not support that. Only FTPS.

-tk

[Messy Potamia]

one.com's robotic tech support said they only do ftp, sftp, and ssh, and don't support ftps. I don't think that offshore robot even knew what ftps was.

So can somebody tell me about the thing that MWALL has on github?

My pi's all have ssh enabled and can ssh & scp amongst each other. Not sure that helps.

Thanks.

[mwall]

On Monday, May 25, 2020 at 3:56:16 PM UTC-4, Messy Potamia wrote:

one.com's robotic tech support said they only do ftp, sftp, and ssh, and don't support ftps. I don't think that offshore robot even knew what ftps was.

So can somebody tell me about the thing that MWALL has on github?

as it says in the readme, it is a client that uses sftp (not ftps, not ftp)

if you can connect to your sftp server using some other sftp client, then the weewx-sftp extension should work using the same credentials/parameters

you can verify by invoking the weewx-sftp extension directly, for example:

PYTHONPATH=/home/weewx/bin python /home/weewx/bin/user/sftp.py /home/weewx/weewx.conf

change the paths if you are using a deb or rpm installation

m

[Messy Potamia]

Aha. "ReadMe". So that's what that's     for. 😂  jk

  I probably missed it. Thanks. 

--

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/GdlSK3vTvYE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c3075a6c-1c68-4098-99e7-378d96b384ee%40googlegroups.com.

[gjr80]

I think the weewx-sftp issue was the inability to install pysftp via pip. It installed fine for me on raspbian Stretch last night though I needed to use pip2 not pip. A bit of googling the OPs pip error message suggested pip may need to be upgraded, wouldn’t be the first time that has come up.

Gary

[Messy Potamia]

Gary, yes, I did try to install pysftp via pip, it failed, and I went onto other things. Will read more about this (including Matt's *readme*) and try this tomorrow.

Meanwhile I'm having problems related to plain ftp that defy logical consistency but that's for another subject.

[Messy Potamia]

Matt, I get the same result whether I invoke pip, pip2, pip3:

pi@RPI3:~ $ sudo pip2 install pysftp
Downloading/unpacking pysftp
  Downloading pysftp-0.2.9.tar.gz
  Running setup.py (path:/tmp/pip-build-QfuB69/pysftp/setup.py) egg_info for package pysftp

    no previously-included directories found matching 'docs/_build'

I also tried apt-get install and it can't find the pkg.

The /root/.pip/pip.log is large

-rw-r--r-- 1 root root 84870 May 26 08:48 pip.log

and I more'd it but couldn't see anything that made sense to me.

Any ideas? btw I did download the rest of the pkg from github including wget the generator, but probably shouldn't proceed until I can get pysftp on my Pi.

Thanks!

---------------------------

[Messy Potamia]

Here's what I did to allow pysftp module to install via pip. I found this on a RaspberryPi forum. You search on this problem and hits come from all over the place with no solutions, until I found this:

sudo apt-get install python-pip
pip install --user -U setuptools
sudo pip install --upgrade pip
sudo apt-get install libffi-dev
sudo pip install pysftp   #errors during pysftp install, "  Building wheel for pynacl (PEP 517) ... \" still working...

This is on my Pi3 running Jessie.

[Messy Potamia]

And let me tell you there are still problems regarding python version,
setuptools version,; this is turning into a dog's breakfast. My pip is
the latest version I have ascertained. But cryptography won't install.
sudo pip -V
pip 20.1.1 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)
--- but ---
Successfully built pysftp pynacl bcrypt
Failed to build cryptography
ERROR: Could not build wheels for cryptography which use PEP 517 and
cannot be installed directly

--MP

> --
> You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/GdlSK3vTvYE/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/94d1c767-7972-4c5f-92ab-e69a349eab1e%40googlegroups.com.

[Messy Potamia]

Messy Potamia <messyp...@gmail.com>

9:53 AM (2 minutes ago)

to weewx-user

And let me tell you there are still problems regarding python version,

setuptools version,; this is turning into a dog's breakfast.

My pip isthe latest version I have ascertained. But cryptography won't install.

 

sudo pip -V
      pip 20.1.1 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)ode here...

     --- but ---

Successfully built pysftp pynacl bcrypt
Failed to build cryptography
ERROR: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly

This is a PI3B (a02082) on Jessie lite.

[Messy Potamia]

Well I knew that was too good to be true.

May 26 10:30:21 RPI3 weewx[21915]: sftpgenerator: upload not possible: No module named pysftp

although the problematic pip install of pysftp "suggested" that module was installed, to wit:

Successfully built pysftp pynacl bcrypt
Failed to build cryptography
ERROR: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly

so I thought since it built the pysftp it would be available for this.

Definitely a #DOG'sBREAKFAST.

Thanks for your help so far.

Anyone else gone / going through this?

Thx/MP

[mwall]

On Tuesday, May 26, 2020 at 4:35:50 AM UTC-4, Messy Potamia wrote:

Well I knew that was too good to be true.

the first error you posted, regarding 'platform_system' NameError, is probably due to an old version of setuptools.  to fix that, update the setuptools for the python installation associated with the 'pip' version that you are invoking.

 

May 26 10:30:21 RPI3 weewx[21915]: sftpgenerator: upload not possible: No module named pysftp

that indicates the python with which you are invoking weewx does not have pysftp installed.  perhaps you installed pysftp for python3 but you are using python2 to invoke weewx?

dog's breakfast indeed.  welcome to the world of python dependencies!  (this is why we try really hard to keep weewx dependencies to a minimum)

one way to avoid this is to install your own python, completely independent of the system's python.  then use the 'pip' from that python installation to install all of the python packages you need.  you may still have to install the build environment and the occasional dev package for your operating system.

the you can use that python by setting it in /etc/default/weewx (if you have a deb/apt or rpm/yum/dnf install) or by invoking it when you do 'setup.py install' (if you have a setup.py install of weewx).

m

[Tom Keffer]

One pip quirk that consistently bites me over and over again, is trying to use pip from the WeeWX home directory, /home/weewx. It appears that is what you're trying to do.

The problem is the presence of the file setup.cfg. Pip sees it, and tries to install in the directories it specifies, in this case /home/weewx. Take a look in /home/weewx/bin and see if your missing pysftp is in there.

The solution is to clean up /home/weewx/bin, then cd to some other directory, then use pip.

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/82433072-10b1-400c-bf00-3cd94191cc5e%40googlegroups.com.

[Rob Cranfill]

Dear Mr or Ms Potamia, ;-)

I have code that I hacked into the standard Weewx base that implements proper sftp. If you would like a copy I can send it to you. (And anyone else who’d like a copy.) Just one modified file.

/rob

[Dr. Thomas Tuch]

Dear Rob

After moving to a different server i have been trying to get sftp to work with weewx and never managed to get pysftp compiled, probably because my pi is running Ubuntu 16. This I cannot change because my home automation is running on the same pi3b - I would not like to mess with heating etc in German winter :-) I would therefore be happy to get a copy of your hack. Thank you in advance. Thomas

[Messy Potamia]

Please be advised I have been using scp in conjunction with ssh in my
scripts to move things around every five minutes (weather station
stuff & wx photos).

> --
> You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/GdlSK3vTvYE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f0444643-d438-47ab-b11c-5a9bf624222en%40googlegroups.com.

[Rob Cranfill]

Funny you should bring this up now. I recently upgraded to WeeWX 2.6, and my modified code no longer works. I was always able to adapt to each new version, but the new codebase is quite different than the previous ones. I have kind of given up on trying to adapt it to 2.6. Sorry. 

Instead, I use a cron job to “rsynch” my local files to the remote. It sounds like maybe you are doing something similar.

If you are using some version of WeeWX less than 2.6 I could still send you my code. Although even that code requires the pysftp library, so maybe none of this will help you. :-/

Let me know if you’d like to try it. 

--

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/GdlSK3vTvYE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f0444643-d438-47ab-b11c-5a9bf624222en%40googlegroups.com.

--

</rob>