SFTP With Private Key Configuration

[Lee Pedley]

Hi,

I hope your are all well.  

As mentioned previously, I have a problem with Paramiko when using SFTP with Private Key.

The Private Key is RSA formatted in OpenSSH format, but when using this is seems to hang and then comes up with the following;

Kex Engine KexNistp256 specified hash_algo <built-in function openssl_sha256>

Switch to new Keys ...

[chan 0] Max Packet in:32768 bytes

Oops, unhandled type 3 ('unimplemented').

EOF in transport thread

I have tried taking the same coding out fro the bots script and run it manually and it works to connect up.  So this seems very strange as it would suggest Paramiko is working.

I can also connect direct to the SFTP server using the same private key so it kind of suggests the key is working and no issues.

So I am at a loss, and was wondering if anyone could help me on this.

I have checked with Paramiko but can't find anything about this error.  

I have seen on boards, that it could be permissions issues, and even try different versions of Paramiko which I have done.

Currently I am running Paramio 2.12.0 which is working for all other SFTP functionality just not this, however like I said if I write a python script to use this direct it works, so I am at a total loss.

Any help on this one, or anyone who has experienced the same thing I would be very grateful to hear from you.  If I can't find a fix, I will need to write something outside of bots to load the file locally, but ideally I want to keep it inside of bots.

Many Thanks for your help

Lee

[Henk-Jan Ebbers]

think that is latest paramiko version.

what is error in bots? (with debug turned on) (so you get python trace)

than turn on 'ftpdebug' in bots.ini (1 is ok)

SFTP logging goes to botssys/logging/sftp.log

maybe that helps finding clues.

kind regards, henk-jan ebbers

--
You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/botsmail/ab894eee-ed98-4237-b056-aace619b9b0cn%40googlegroups.com.

[Lee Pedley]

Hi Henk,

Thanks for coming back to me on this;

I am getting the following error within bots;

Traceback (most recent call last): File "/gw/app/app/bots/botslib.py", line 333, in wrapper terug = func(*args,**argv) File "/gw/app/app/bots/communication.py", line 84, in run comclass.run() File "/gw/app/app/bots/communication.py", line 117, in run self.connect() File "/gw/app/app/bots/communication.py", line 1418, in connect self.session = paramiko.SFTPClient.from_transport(self.transport) File "/root/.local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 133, in from_transport File "/root/.local/lib/python2.7/site-packages/paramiko/transport.py", line 772, in open_session File "/root/.local/lib/python2.7/site-packages/paramiko/transport.py", line 896, in open_channel EOFError

and on the SFTP log I have this

DEB [20240804-21:45:02.192] thr=1   paramiko.transport: starting thread (client mode): 0x65f9d1d0L
DEB [20240804-21:45:02.192] thr=1   paramiko.transport: Local version/idstring: SSH-2.0-paramiko_2.8.10
DEB [20240804-21:45:02.228] thr=1   paramiko.transport: Remote version/idstring: u'SSH-2.0-MFT'
INF [20240804-21:45:02.228] thr=1   paramiko.transport: Connected (version 2.0, client MFT)
DEB [20240804-21:45:02.229] thr=1   paramiko.transport: kex follows? False
kex algos: [u'diffie-hellman-group1-sha1', u'diffie-hellman-group14-sha1', u'diffie-hellman-group14-sha256', u'diffie-hellman-group15-sha512', u'diffie-hellman-group16-sha512', u'diffie-hellman-group17-sha512', u'diffie-hellman-group18-sha512', u'diffie-hellman-group-exchange-sha1', u'diffie-hellman-group-exchange-sha256', u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'ecdh-sha2-1.3.132.0.1', u'ecdh-sha2-1.2.840.10045.3.1.1', u'ecdh-sha2-1.3.132.0.33', u'ecdh-sha2-1.3.132.0.26', u'ecdh-sha2-1.3.132.0.27', u'ecdh-sha2-1.3.132.0.16', u'ecdh-sha2-1.3.132.0.36', u'ecdh-sha2-1.3.132.0.37', u'ecdh-sha2-1.3.132.0.38']
server key: [u'ssh-rsa', u'rsa-sha2-256', u'rsa-sha2-512']
client encrypt: [u'3des-cbc', u'3des-ctr', u'blowfish-cbc', u'blowfish-ctr', u'twofish128-cbc', u'twofish128-ctr', u'twofish192-cbc', u'twofish192-ctr', u'twofish-cbc', u'twofish256-cbc', u'twofish256-ctr', u'aes128-cbc', u'aes128-ctr', u'aes192-cbc', u'aes192-ctr', u'aes256-cbc', u'aes256-ctr', u'serpent128-cbc', u'serpent128-ctr', u'serpent192-cbc', u'serpent192-ctr', u'serpent256-cbc', u'serpent256-ctr', u'idea-cbc', u'idea-ctr', u'cast128-cbc', u'cast128-ctr', u'arcfour', u'arcfour128', u'arcfour256']
server encrypt: [u'3des-cbc', u'3des-ctr', u'blowfish-cbc', u'blowfish-ctr', u'twofish128-cbc', u'twofish128-ctr', u'twofish192-cbc', u'twofish192-ctr', u'twofish-cbc', u'twofish256-cbc', u'twofish256-ctr', u'aes128-cbc', u'aes128-ctr', u'aes192-cbc', u'aes192-ctr', u'aes256-cbc', u'aes256-ctr', u'serpent128-cbc', u'serpent128-ctr', u'serpent192-cbc', u'serpent192-ctr', u'serpent256-cbc', u'serpent256-ctr', u'idea-cbc', u'idea-ctr', u'cast128-cbc', u'cast128-ctr', u'arcfour', u'arcfour128', u'arcfour256']
client mac: [u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5', u'hmac-md5-96', u'hmac-sha2-256', u'hmac-sha2-256-96', u'hmac-sha2-512', u'hmac-sha2-512-96']
server mac: [u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5', u'hmac-md5-96', u'hmac-sha2-256', u'hmac-sha2-256-96', u'hmac-sha2-512', u'hmac-sha2-512-96']
client lang: ''
server lang: ''
client compress: [u'none', u'zlib']
server compress: [u'none', u'zlib']
DEB [20240804-21:45:02.229] thr=1   paramiko.transport: Kex agreed: ecdh-sha2-nistp256
DEB [20240804-21:45:02.230] thr=1   paramiko.transport: HostKey agreed: ssh-rsa
DEB [20240804-21:45:02.230] thr=1   paramiko.transport: Cipher agreed: aes128-ctr
DEB [20240804-21:45:02.230] thr=1   paramiko.transport: MAC agreed: hmac-sha2-256
DEB [20240804-21:45:02.230] thr=1   paramiko.transport: Compression agreed: none
DEB [20240804-21:45:02.315] thr=1   paramiko.transport: kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>
DEB [20240804-21:45:02.316] thr=1   paramiko.transport: Switch to new keys ...
DEB [20240804-21:45:02.324] thr=2   paramiko.transport: [chan 0] Max packet in: 32768 bytes
WAR [20240804-21:45:02.394] thr=1   paramiko.transport: Oops, unhandled type 3 ('unimplemented')
DEB [20240804-21:46:02.425] thr=1   paramiko.transport: EOF in transport thread

The Oops part hangs for about a minute before it comes up with the EOF error.

Much appreciated for your response on this Henk.

Thanks

Lee
                                                                                                                                                                                       

[Eppye Bots]

that is deep in the paramiko package.

so think it is better to look for help in that direction.

think I would try to re-do all the steps you did (with certificate etc)?

about permissions: directory where key is needs certain permissions, key file needs certain permissions.  (easy to google). AFAIK you persmissions are not allowed to be too open.

kind regards, Henk-Jan Ebbers

To view this discussion on the web visit https://groups.google.com/d/msgid/botsmail/904d184e-b9c0-482d-9261-a97c8eccdebcn%40googlegroups.com.

[Lee Pedley]

Thanks Henk, and all your time looking into this, its much appreciated.

I had a feeling it was quite deep so will see if I can raise it on the Paramiko boards.

I will try to recreate also to see if this will resolve the issue, very odd as like I said I can access it direct using a basic script to connect to Paramiko, hence why this seems very odd.  

I have checked permissions also, as I had that initially and couldn't connect to it until I corrected that.  

Anyway, once again thank you for your help, have a great evening.

Lee

[Henk-Jan Ebbers]

like I said I can access it direct using a basic script to hence why this seems very odd.  

yeah thats odd.

what is the script like?

what parameters are being filled for this in the channel?

kind regards, henk-jan ebbers

To view this discussion on the web visit https://groups.google.com/d/msgid/botsmail/bcbc35eb-2ed1-416f-a977-05c4b627b234n%40googlegroups.com.

[Lee Pedley]

Hi,

I have started to change this at the moment but was just a copy of the bots script originally.

But this script I am working on 

import paramiko

privatekeyfile = '/srv/sftp/certificates/ada-private-key.ppk'
mykey = paramiko.RSAKey.from_private_key_file(privatekeyfile)
ssh_client = paramiko.SSHClient()
ssh_client.load_system_host_keys()
ssh_client.connect(hostname='mft.ada.uk', username='user_dsv', allow_agent=True, pkey=mykey)

ftp_client = ssh_client.open_sftp()
ftp_client.chdir('/PO6/inbox')
for i in ftp_client.listdir():
    fileout=str(ftp_client.lstat(i)).split()[0]
    if 'd' not in fileout: print i, 'is a file'

ftp_client.close()
ssh_client.close()

This connects straight away and gives a listing of the files on the server.  

So I know that parmiko is working.

Thanks

Lee

[Eppye Bots]

ok, I looked how this works.

you need to make an small communication-script to make this work.

example content communicationscript is below:

def privatekey(channeldict,*args,**kwargs):
    privatekeyfile = channeldict['keyfile']
    pkeytype = 'RSA'
    pkeypassword = None
    return privatekeyfile,pkeytype,pkeypassword

hope this helps

kind regards, Henk-Jan Ebbers

To view this discussion on the web visit https://groups.google.com/d/msgid/botsmail/44b49787-1944-40b6-a09d-19e0b5a5a8dcn%40googlegroups.com.

[Lee Pedley]

Hi Henk,

Thank yo use much for this as it has worked and now uses the private key.

Very much appreciated on this as I was about to write a separate script around it. 

Thank you and have a great day.

Lee

[Lee Pedley]

Hi,

this works great for reading files and I no longer have problems with this part. However I know have problems writing files, as every time I do it closes the connection.

INF [20240805-17:04:30.470] thr=1   paramiko.transport: Authentication (publickey) successful!
DEB [20240805-17:04:30.488] thr=2   paramiko.transport: [chan 0] Max packet in: 32768 bytes
DEB [20240805-17:04:30.506] thr=1   paramiko.transport: [chan 0] Max packet out: 32768 bytes
DEB [20240805-17:04:30.506] thr=1   paramiko.transport: Secsh channel 0 opened.
DEB [20240805-17:04:30.538] thr=1   paramiko.transport: [chan 0] Sesch channel 0 request ok
INF [20240805-17:04:30.556] thr=2   paramiko.transport.sftp: [chan 0] Opened sftp connection (server version 3)
DEB [20240805-17:04:30.556] thr=2   paramiko.transport.sftp: [chan 0] stat('.')
DEB [20240805-17:04:30.588] thr=2   paramiko.transport.sftp: [chan 0] normalize('.')
DEB [20240805-17:04:30.621] thr=2   paramiko.transport.sftp: [chan 0] stat('/S3/inbox')
DEB [20240805-17:04:30.653] thr=2   paramiko.transport.sftp: [chan 0] normalize('/S3/inbox')
DEB [20240805-17:04:30.690] thr=2   paramiko.transport.sftp: [chan 0] open('/S3/inbox/041104_STK_PM_20240805_396.xml', 'a')
INF [20240805-17:04:30.725] thr=2   paramiko.transport.sftp: [chan 0] sftp session closed.
DEB [20240805-17:04:30.725] thr=2   paramiko.transport: [chan 0] EOF sent (0)
DEB [20240805-17:04:30.743] thr=1   paramiko.transport: EOF in transport thread

No errors on the SFTP side, but I do get one in Bots;

Traceback (most recent call last): File "/Comfy-Gateway/app/app/bots/communication.py", line 1511, in outcommunicate tofile = self.session.open(tofilename, mode) # SSH treats all files as binary File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 372, in open t, msg = self._request(CMD_OPEN, filename, imode, attrblock) File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 822, in _request return self._read_response(num) File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 874, in _read_response self._convert_status(msg) File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 907, in _convert_status raise IOError(text) IOError: Internal server error.

Sorry to be a pain on this, but I am hoping it would be an easy fix.

Again I have tried to use a manual script and this works, so I am guessing this is something that bots is doing, or needs slightly tweaking.

Thank you for all your time on this, you will have to send me an invoice for all your time on this as I don't expect you to waste your time for nothing.

Thank you

Lee

[Lee Pedley]

Hi,

I have tested a direct upload via Paramiko to see if it's this that is causing the issue.  

I have used the following code;

import paramiko

privatekeyfile = '/srv/sftp/certificates/private-key.ppk'

mykey = paramiko.RSAKey.from_private_key_file(privatekeyfile)
ssh_client = paramiko.SSHClient()
ssh_client.load_system_host_keys()

ssh_client.connect(hostname='mft.ada.uk', username='user_dsv', allow_agent=True, pkey=mykey)

ftp_client = ssh_client.open_sftp()

localFilePath  = "order.xml"
remoteFilePath = "/S3/testfile.txt"
try:
  ftp_client.put(localFilePath, remoteFilePath)
except FileNotFoundError as err:
  print("File {localFilePath} was not found on the local system")

ftp_client.close()
ssh_client.close()

From running this the file gets uploaded to the SFTP server, so it looks like something in bots is preventing the file to upload.  I will keep you updated with progress on this, but seems very odd behaviour as no errors in the SFTP.log file as it appears to work, but bots reports an IOError.

Thanks

Lee

[Lee Pedley]

Hi,

I have found the solution to this, and was down to their SFTP not supporting APPE and only STOR.  This was fixed with the following line 

{overwrite}OUT_{datetime:%Y%m%d%H%M%S}_*.edi

Once again thanks Hank for al your help on this.

Have a great day all.

Lee

[Eppye Bots]

great!

kind regards, Henk-Jan Ebbers

To view this discussion on the web visit https://groups.google.com/d/msgid/botsmail/6f285865-0fc8-4afc-9c36-2272e15fbd9dn%40googlegroups.com.