Paramiko Connection Problem

Peter Gillespie

unread,

May 30, 2022, 9:34:07 AM5/30/22

to aiidausers

Dear Community,

I recently upgraded to aiida-core v2. The installation went ahead without issues and I can access all of my old data and run processes with no problems (thus far). However, when I came to run calculations on my local HPC cluster I find that AiIDA cannot connect to the system via ssh. I tested the computer node (through verdi computer test) and found the following (with some personal information omitted for obvious reasons):

Opening connection... Error: Error connecting to <hostname> through SSH: [SshTransport] Private key file is encrypted, connect_args were: {'username': <username>, 'port': 22, 'look_for_keys': True, 'key_filename': <path_to_private_key>, 'timeout': 60, 'allow_agent': True, 'proxy_command': '', 'compress': True, 'gss_auth': False, 'gss_kex': False, 'gss_deleg_creds': False, 'gss_host': <hostname>}
[FAILED]: Error while trying to connect to the computer

The traceback from which is:

Full traceback:
Traceback (most recent call last):
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/aiida/cmdline/commands/cmd_computer.py", line 478, in computer_test
     with transport:
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/aiida/transports/transport.py", line 133, in __enter__
     self.open()
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/aiida/transports/plugins/ssh.py", line 522, in open
     self._client.connect(self._machine, **connection_arguments)
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/paramiko/client.py", line 435, in connect
     self._auth(
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/paramiko/client.py", line 766, in _auth
     raise saved_exception
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/paramiko/client.py", line 736, in _auth
     key = self._key_from_filepath(
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/paramiko/client.py", line 588, in _key_from_filepath
     key = klass.from_private_key_file(key_path, password)
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/paramiko/pkey.py", line 249, in from_private_key_file
     key = cls(filename=filename, password=password)
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/paramiko/rsakey.py", line 64, in __init__
     self._from_private_key_file(filename, password)
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/paramiko/rsakey.py", line 194, in _from_private_key_file
     data = self._read_private_key_file("RSA", filename, password)
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/paramiko/pkey.py", line 322, in _read_private_key_file
     data = self._read_private_key(tag, f, password)
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/paramiko/pkey.py", line 348, in _read_private_key
     data = self._read_private_key_pem(lines, end, password)
   File "/home/hyphenater/envs/aiida-v2/lib/python3.8/site-packages/paramiko/pkey.py", line 400, in _read_private_key_pem
     raise PasswordRequiredException("Private key file is encrypted")
paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted
Warning: 1 out of 0 tests failed

For the national facilities that I also use, this same test works without issue - as they did before the upgrade. After making sure that there wasn't a problem with my private key, I checked Paramiko itself and was able to re-create the issue by simply trying to start an ssh client session with the local cluster in IPython:

In [1]: from paramiko.client import SSHClient

In [2]: client = SSHClient()

In [3]: SSH_USER = <username>
  ...: SSH_HOST = <hostname>
  ...: SSH_PORT = 22
  ...: SSH_KEY = <path_to_key_file>
  ...: SSH_KEY_PASSWORD = ""

In [4]: client.load_system_host_keys()
  ...: client.connect(SSH_HOST, port=SSH_PORT, username=SSH_USER,look_for_keys=True,key_filename=SSH_KEY,passphrase=SSH_KEY_PASSWORD)

For the national facilities, this executes with no errors. For my local cluster I get different errors depending on whether the passphrase is included explicitly or not (note that the key is password-less, so this shouldn't make a difference). With the passphrase variable ommitted, the error is the same as before:(paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted)

However, if the passphrase variable is included in the command, I get this instead (after a long traceback):

ValueError: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).'

Again, this works both with and without the passphrase when connecting to the national facilities. I also tested previous versions of Paramiko and found that only v2.8 of Paramiko works with our local cluster - anything newer reproduces the same problem.

What I would like to know is whether aiida-core v2 (2.0.1 in this case) would work at all with an older version of Paramiko, or whether there is some other potential fix that I'm missing here.

All the best and thanks in advance

-Peter

Zhu, Bonan

unread,

May 30, 2022, 10:40:33 AM5/30/22

to aiida...@googlegroups.com

Hi,

I think, swapping paramiko back to 2.8 might break other things. You can try checking if the ssh key is indeed unencrypted and whether your local cluster and national computer are indeed both using same key (ssh -vv get more information).

The other thing to try is to use ssh-agent for serving the keys, then you don’t won’t any problem with encrypted keys (which are now handled by ssh-agent).

If all fails, you can locate the code change between paramiko 2.8 and the latest version, and then try to implement a custom SSHTransport plugin that reproduces the old behaviour in 2.8. This can get quite complicated, so best to try simple solutions first.

Best wishes,

Bonan

From: aiida...@googlegroups.com <aiida...@googlegroups.com> On Behalf Of Peter Gillespie
Sent: 30 May 2022 14:34
To: aiidausers <aiida...@googlegroups.com>
Subject: [aiidausers] Paramiko Connection Problem

⚠ Caution: External sender

--
AiiDA is supported by the NCCR MARVEL (http://nccr-marvel.ch/), funded by the Swiss National Science Foundation, and by the European H2020 MaX Centre of Excellence (http://www.max-centre.eu/).

Before posting your first question, please see the posting guidelines at http://www.aiida.net/?page_id=356 .
---
You received this message because you are subscribed to the Google Groups "aiidausers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aiidausers+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/764fcd62-9dc8-49bb-8e06-a2b51823cde2n%40googlegroups.com.

Giovanni Pizzi

unread,

May 30, 2022, 11:02:33 AM5/30/22

to AiiDA users mailing list

Hi!

Indeed, AiiDA supports asking Paramiko to use an SSH agent. Can you try that?

Also, since you managed to isolate the problem to Paramiko, I suggest you open an issue with Paramiko (on their GitHub) so they can fix the bug (if it's confirmed).

Best,

Giovanni

To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/DB9PR01MB9175280568311ABEE4363366B9DD9%40DB9PR01MB9175.eurprd01.prod.exchangelabs.com.

Peter Gillespie

unread,

May 31, 2022, 5:18:47 AM5/31/22

to aiidausers

Hi Giovanni and Bonan,

Many thanks for your helpful suggestions. I went back and configured the SSH agent correctly (since it seems that I misunderstood how to do this in the first place) and set it to use an encrypted key for my local and national clusters, ensuring that the ssh config file (~/.ssh/config) is also set correctly so that both systems are using the same key. Through the terminal, all the connections work fine and the agent only requires the password when the keys are added to the agent.

If I use this setup with the version of Paramiko installed with aiida-core (paramiko 2.11.0) and repeat the procedure as before for the national cluster (this time including the SSH agent):

In [1]: from paramiko.client import SSHClient

In [2]: client = SSHClient()

In [3]: SSH_USER = <username>
  ...: SSH_HOST = <hostname>
  ...: SSH_PORT = 22
  ...: SSH_KEY = <path_to_key_file>

...: SSH_KEY_PASSWORD = <passphrase>

In [4]: client.load_system_host_keys()
...: client.connect(SSH_HOST, port=SSH_PORT, allow_agent=True, username=SSH_USER,look_for_keys=True,key_filename=SSH_KEY,passphrase=SSH_KEY_PASSWORD)

The client connects with no issues - which works both with and without the passphrase variable (as it should, I assume, since the agent is handling the passphrase for the key by itself). If I do the same for our local cluster, the client still fails to connect. Without the passphrase included, I get:

PasswordRequiredException: Private key file is encrypted

If instead I include the passphrase, I get a different error:

~/envs/aiida-v2/lib/python3.8/site-packages/paramiko/client.py in connect(self, hostname, port, username, password, pkey, key_filename, timeout, allow_agent, look_for_keys, compress, sock, gss_auth, gss_kex, gss_deleg_creds, gss_host, banner_timeout, auth_timeout, gss_trust_dns, passphrase, disabled_algorithms)
   433             key_filenames = key_filename
   434
--> 435         self._auth(
   436             username,
   437             password,

~/envs/aiida-v2/lib/python3.8/site-packages/paramiko/client.py in _auth(self, username, password, pkey, key_filenames, allow_agent, look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host, passphrase)
   764         # if we got an auth-failed exception earlier, re-raise it
   765         if saved_exception is not None:
--> 766             raise saved_exception
   767         raise SSHException("No authentication methods available")
   768

~/envs/aiida-v2/lib/python3.8/site-packages/paramiko/client.py in _auth(self, username, password, pkey, key_filenames, allow_agent, look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host, passphrase)
   740                     # in ['password']
   741                     allowed_types = set(
--> 742                         self._transport.auth_publickey(username, key)
   743                     )
   744                     two_factor = allowed_types & two_factor_types

~/envs/aiida-v2/lib/python3.8/site-packages/paramiko/transport.py in auth_publickey(self, username, key, event)
  1633             # caller wants to wait for event themselves
  1634             return []
-> 1635         return self.auth_handler.wait_for_response(my_event)
  1636
  1637     def auth_interactive(self, username, handler, submethods=""):

~/envs/aiida-v2/lib/python3.8/site-packages/paramiko/auth_handler.py in wait_for_response(self, event)
   257             if issubclass(e.__class__, PartialAuthentication):
   258                 return e.allowed_types
--> 259             raise e
   260         return []
   261

AuthenticationException: Authentication failed.

I must admit, I was scratching my head over this. I repeated the steps with Paramiko v2.8 and, as before, there are no issues with the older version at all. On a whim, I tried generating a new key using a different algorithm to see if that makes a difference - previously I had used an RSA algorithm, so I tried ECDSA instead. If I repeat the same steps with the new ecdsa key, the paramiko v2.11 client connects to the local cluster and AiiDA can submit calculations with no issues. I tried generating a new RSA key as well and tested that just to be sure, but this just re-creates the same problem as before.

I'm going to consider the problem solved at this stage. I was preparing to send a bug report to the Paramiko GitHub page, but I'm fairly certain that this is specifically an issue with our local HPC cluster and not Paramiko itself.