Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

pysftp / paramiko problem

1,217 views
Skip to first unread message

Robin Becker

unread,
Jun 11, 2019, 10:24:16 AM6/11/19
to
I am trying to convert older code that uses ftplib as the endpoint has switched to sftp only.

I am using the pysftp wrapper around paramiko.

The following script fails

def main():
import pysftp
with pysftp.Connection('ftp.remote.com', username='me', password='xxxxxx') as sftp:
print('top level')
print(sftp.listdir())
print(sftp.normalize(u'XXXX'))
print('direct list of XXXX')
print(sftp.listdir(u'XXXX'))
with sftp.cd(u'XXXX'):
print(sftp.listdir())

if __name__ == '__main__':
main()


when run the program prints [u'XXXX'] and then fails at the normalize command.

> $ python tsftp.py
> top level
> [u'XXXX']
> Traceback (most recent call last):
> File "tsftp.py", line 13, in <module>
> main()
> File "tsftp.py", line 6, in main
> print(sftp.normalize(u'XXXX'))
> File "/home/rptlab/devel/env/lib/python2.7/site-packages/pysftp/__init__.py", line 640, in normalize
> return self._sftp.normalize(remotepath)
> File "/home/rptlab/devel/env/lib/python2.7/site-packages/paramiko/sftp_client.py", line 632, in normalize
> t, msg = self._request(CMD_REALPATH, path)
> File "/home/rptlab/devel/env/lib/python2.7/site-packages/paramiko/sftp_client.py", line 813, in _request
> return self._read_response(num)
> File "/home/rptlab/devel/env/lib/python2.7/site-packages/paramiko/sftp_client.py", line 865, in _read_response
> self._convert_status(msg)
> File "/home/rptlab/devel/env/lib/python2.7/site-packages/paramiko/sftp_client.py", line 894, in _convert_status
> raise IOError(errno.ENOENT, text)
> IOError: [Errno 2] No such file.

I tried other commands, but it seems any attempt to cd to the XXXX directory fails.

Using sftp in the shell directly I needed to add HostKeyAlgorithms=+ssh-dss for this host.

Any pointers to what the problem could be?
--
Robin Becker

dieter

unread,
Jun 12, 2019, 12:59:42 AM6/12/19
to
Robin Becker <ro...@reportlab.com> writes:
> I am trying to convert older code that uses ftplib as the endpoint has switched to sftp only.
>
> I am using the pysftp wrapper around paramiko.
>
> The following script fails
>
> def main():
> import pysftp
> with pysftp.Connection('ftp.remote.com', username='me', password='xxxxxx') as sftp:
> print('top level')
> print(sftp.listdir())
> print(sftp.normalize(u'XXXX'))

>From the "sftp" documentation:

| normalize(self, remotepath)
| Return the expanded path, w.r.t the server, of a given path. This
| can be used to resolve symlinks or determine what the server believes
| to be the :attr:`.pwd`, by passing '.' as remotepath.

This suggests that your observation could be explained
by "u'XXXX'" being a broken symlink.

Robin Becker

unread,
Jun 12, 2019, 3:05:30 AM6/12/19
to
Well with real sftp I can cd to that path so if it is a symlink it goes somewhere.

With pysftp I am unable to chdir or cd into it. With a bit of difficulty I can use subprocess + sshpass + sftp to do the required
transfer.
--
Robin Becker

dieter

unread,
Jun 13, 2019, 12:57:18 AM6/13/19
to
Robin Becker <ro...@reportlab.com> writes:
> On 12/06/2019 05:59, dieter wrote:
>> Robin Becker <ro...@reportlab.com> writes:
>>> I am trying to convert older code that uses ftplib as the endpoint has switched to sftp only.
> ...
> Well with real sftp I can cd to that path so if it is a symlink it goes somewhere.
>
> With pysftp I am unable to chdir or cd into it. With a bit of
> difficulty I can use subprocess + sshpass + sftp to do the required
> transfer.

Maybe, the problem is the "u" prefix.
Can you try your script with Python 3 or encode the unicode
into a native ``str``?

Robin Becker

unread,
Jun 13, 2019, 9:57:50 AM6/13/19
to
On 13/06/2019 05:56, dieter wrote:
> Robin Becker <ro...@reportlab.com> writes:
>> On 12/06/2019 05:59, dieter wrote:
>>> Robin Becker <ro...@reportlab.com> writes:
>>>> I am trying to convert older code that uses ftplib as the endpoint has switched to sftp only.
>> ...
>> Well with real sftp I can cd to that path so if it is a symlink it goes somewhere.
>>
>> With pysftp I am unable to chdir or cd into it. With a bit of
>> difficulty I can use subprocess + sshpass + sftp to do the required
>> transfer.
>
> Maybe, the problem is the "u" prefix.
> Can you try your script with Python 3 or encode the unicode
> into a native ``str``?
>
no same happens in a fresh python 3.6 environment

> $ cat - > tsftp.py
> def main():
> import pysftp
> with pysftp.Connection('ftp.remote.com', username='me', password='ucl20 11') as sftp:
> print('top level')
> print(sftp.listdir())
> print(sftp.normalize('XXXX'))
>
> if __name__ == '__main__':
> main()
> (tpy3) rptlab@app1:~/tmp/tpy3
> $ python tsftp.py
> top level
> ['XXXX']
> Traceback (most recent call last):
> File "tsftp.py", line 9, in <module>
> main()
> File "tsftp.py", line 6, in main
> print(sftp.normalize('XXXX'))
> File "/home/rptlab/tmp/tpy3/lib/python3.6/site-packages/pysftp/__init__.py", line 640, in normalize
> return self._sftp.normalize(remotepath)
> File "/home/rptlab/tmp/tpy3/lib/python3.6/site-packages/paramiko/sftp_client.py", line 632, in normalize
> t, msg = self._request(CMD_REALPATH, path)
> File "/home/rptlab/tmp/tpy3/lib/python3.6/site-packages/paramiko/sftp_client.py", line 813, in _request
> return self._read_response(num)
> File "/home/rptlab/tmp/tpy3/lib/python3.6/site-packages/paramiko/sftp_client.py", line 865, in _read_response
> self._convert_status(msg)
> File "/home/rptlab/tmp/tpy3/lib/python3.6/site-packages/paramiko/sftp_client.py", line 894, in _convert_status
> raise IOError(errno.ENOENT, text)
> FileNotFoundError: [Errno 2] No such file.

this is what real sftp does

> (tpy3) rptlab@app1:~/tmp/tpy3
> $ sshpass -p ucl2011 sftp m...@ftp.remote.com
> Connected to ftp.remote.com.
> sftp> cd XXXX
> sftp> pwd
> Remote working directory: /XXXX
> sftp> ls
> OLD GR ........
> ZZZZZ.pdf
> sftp> ^D
> (tpy3) rptlab@app1:~/tmp/tpy3



--
Robin Becker

MRAB

unread,
Jun 13, 2019, 1:24:19 PM6/13/19
to
What does:

sftp.normalize('.')

return?

Robin Becker

unread,
Jun 14, 2019, 10:19:12 AM6/14/19
to
On 13/06/2019 18:23, MRAB wrote:
.........
>>
> What does:
>
> sftp.normalize('.')
>
> return?

It returns '/'.

sftp.chdir('XXXX') and that also fails in paramiko as it seems to use CMD_REALPATH to do that.

> File "tsftp.py", line 7, in main
> print(sftp.chdir('XXXX'))
> File "/home/rptlab/tmp/tpy3/lib/python3.6/site-packages/pysftp/__init__.py", line 524, in chdir
> self._sftp.chdir(remotepath)
> File "/home/rptlab/tmp/tpy3/lib/python3.6/site-packages/paramiko/sftp_client.py", line 662, in chdir
> self._cwd = b(self.normalize(path))
> File "/home/rptlab/tmp/tpy3/lib/python3.6/site-packages/paramiko/sftp_client.py", line 632, in normalize
> t, msg = self._request(CMD_REALPATH, path)
> File "/home/rptlab/tmp/tpy3/lib/python3.6/site-packages/paramiko/sftp_client.py", line 813, in _request
> return self._read_response(num)
> File "/home/rptlab/tmp/tpy3/lib/python3.6/site-packages/paramiko/sftp_client.py", line 865, in _read_response
> self._convert_status(msg)
> File "/home/rptlab/tmp/tpy3/lib/python3.6/site-packages/paramiko/sftp_client.py", line 894, in _convert_status
> raise IOError(errno.ENOENT, text)
> FileNotFoundError: [Errno 2] No such file.

--
Robin Becker

Robin Becker

unread,
Jun 14, 2019, 10:48:58 AM6/14/19
to
.......

I tried an experiment with a remote server that I control and pysftp works perfectly there. A difference that I know of is that
this server is using ubuntu 18.04 and we don't use passwords, but a private_key. Also this server is using the openssh internal sftp.

I believe the failing server is using an earlier version of openssh or OS as it wants to use ssh-dss which is now considered
unsafe (I believe).
--
Robin Becker

0 new messages