Resize - <type 'exceptions.TypeError'>

[Maiken Pedersen]

Hi,

I am for the first time trying the resize option in elasticluster.

I got a new elasticluster from github today.

This is the only template I have

(elasticluster_07052019)[centos@elasticluster-final ~]$ elasticluster list-templates
1 cluster templates found in configuration file.
2019-05-07 09:42:19 elasticluster-final.novalocal gc3.elasticluster[23614] WARNING ImportWarning: Not importing directory '/usr/lib/python2.7/site-packages/oslo_i18n/locale': missing __init__.py
2019-05-07 09:42:19 elasticluster-final.novalocal gc3.elasticluster[23614] WARNING ImportWarning: Not importing directory '/usr/lib/python2.7/site-packages/babel/localedata': missing __init__.py
2019-05-07 09:42:19 elasticluster-final.novalocal gc3.elasticluster[23614] WARNING ImportWarning: Not importing directory '/usr/lib/python2.7/site-packages/requests/packages': missing __init__.py

name:     slurm
frontend nodes: 1
compute nodes: 7

And this is what I get if I try to expand with 2 compute nodes

1) is it correct to use "compute" to name the group - I am not sure if I should chose compute_node which is what I have in my configuration file, or compute or slurm_worker that I see in my inventory file

2) What is the error below due to?

(elasticluster_07052019)[centos@elasticluster-final ~]$ elasticluster resize -a 2:compute arc6_singularity -t slurm
2019-05-07 09:39:47 elasticluster-final.novalocal gc3.elasticluster[23590] WARNING ImportWarning: Not importing directory '/usr/lib/python2.7/site-packages/oslo_i18n/locale': missing __init__.py
2019-05-07 09:39:47 elasticluster-final.novalocal gc3.elasticluster[23590] WARNING ImportWarning: Not importing directory '/usr/lib/python2.7/site-packages/babel/localedata': missing __init__.py
2019-05-07 09:39:47 elasticluster-final.novalocal gc3.elasticluster[23590] WARNING ImportWarning: Not importing directory '/usr/lib/python2.7/site-packages/requests/packages': missing __init__.py
Adding 2 compute node(s) to the cluster
2019-05-07 09:39:48 elasticluster-final.novalocal gc3.elasticluster[23590] ERROR Could not start node `compute003`: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object -- <type 'exceptions.TypeError'>
Traceback (most recent call last):
  File "/home/centos/elasticluster_07052019/src/elasticluster/cluster.py", line 580, in _start_node
    node.start()
  File "/home/centos/elasticluster_07052019/src/elasticluster/cluster.py", line 1318, in start
    **self.extra)
  File "/home/centos/elasticluster_07052019/src/elasticluster/providers/openstack.py", line 471, in start_instance
    self._check_keypair(key_name, public_key_path, private_key_path)
  File "/home/centos/elasticluster_07052019/src/elasticluster/providers/openstack.py", line 679, in _check_keypair
    pkey = RSAKey.from_private_key_file(private_key_path)
  File "/usr/lib/python2.7/site-packages/paramiko/pkey.py", line 196, in from_private_key_file
    key = cls(filename=filename, password=password)
  File "/usr/lib/python2.7/site-packages/paramiko/rsakey.py", line 45, in __init__
    self._from_private_key_file(filename, password)
  File "/usr/lib/python2.7/site-packages/paramiko/rsakey.py", line 164, in _from_private_key_file
    self._decode_key(data)
  File "/usr/lib/python2.7/site-packages/paramiko/rsakey.py", line 173, in _decode_key
    data, password=None, backend=default_backend()
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/primitives/serialization/base.py", line 28, in load_der_private_key
    return backend.load_der_private_key(data, password)
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1089, in load_der_private_key
    bio_data = self._bytes_to_bio(data)
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 462, in _bytes_to_bio
    data_ptr = self._ffi.from_buffer(data)
TypeError: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object
2019-05-07 09:39:48 elasticluster-final.novalocal gc3.elasticluster[23590] ERROR Could not start node `compute004`: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object -- <type 'exceptions.TypeError'>
Traceback (most recent call last):
  File "/home/centos/elasticluster_07052019/src/elasticluster/cluster.py", line 580, in _start_node
    node.start()
  File "/home/centos/elasticluster_07052019/src/elasticluster/cluster.py", line 1318, in start
    **self.extra)
  File "/home/centos/elasticluster_07052019/src/elasticluster/providers/openstack.py", line 471, in start_instance
    self._check_keypair(key_name, public_key_path, private_key_path)
  File "/home/centos/elasticluster_07052019/src/elasticluster/providers/openstack.py", line 679, in _check_keypair
    pkey = RSAKey.from_private_key_file(private_key_path)
  File "/usr/lib/python2.7/site-packages/paramiko/pkey.py", line 196, in from_private_key_file
    key = cls(filename=filename, password=password)
  File "/usr/lib/python2.7/site-packages/paramiko/rsakey.py", line 45, in __init__
    self._from_private_key_file(filename, password)
  File "/usr/lib/python2.7/site-packages/paramiko/rsakey.py", line 164, in _from_private_key_file
    self._decode_key(data)
  File "/usr/lib/python2.7/site-packages/paramiko/rsakey.py", line 173, in _decode_key
    data, password=None, backend=default_backend()
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/primitives/serialization/base.py", line 28, in load_der_private_key
    return backend.load_der_private_key(data, password)
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1089, in load_der_private_key
    bio_data = self._bytes_to_bio(data)
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 462, in _bytes_to_bio
    data_ptr = self._ffi.from_buffer(data)
TypeError: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object
2019-05-07 09:39:51 elasticluster-final.novalocal gc3.elasticluster[23590] WARNING CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
No handlers could be found for logger "paramiko.transport"
2019-05-07 09:39:51 elasticluster-final.novalocal gc3.elasticluster[23590] ERROR Error: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object
Aborting because of errors: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object.

Thanks!

[Maiken Pedersen]

Hi again,

I tried

sudo pip install --upgrade pip

sudo pip uninstall cryptography
sudo pip install paramiko
sudo pip install cryptography -U




But to no avail unfortunately.

(elasticluster_07052019)[centos@elasticluster-final ~]$ sudo pip freeze | grep paramiko
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
paramiko==2.4.2
(elasticluster_07052019)[centos@elasticluster-final ~]$ sudo pip freeze | grep cryptography
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
cryptography==2.6.1

Still see

Could not start node `compute003`: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object -- <type 'exceptions.TypeError'>
Traceback (most recent call last):
  File "/home/centos/elasticluster_07052019/src/elasticluster/cluster.py", line 580, in _start_node
    node.start()
  File "/home/centos/elasticluster_07052019/src/elasticluster/cluster.py", line 1318, in start
    **self.extra)
  File "/home/centos/elasticluster_07052019/src/elasticluster/providers/openstack.py", line 471, in start_instance
    self._check_keypair(key_name, public_key_path, private_key_path)
  File "/home/centos/elasticluster_07052019/src/elasticluster/providers/openstack.py", line 679, in _check_keypair
    pkey = RSAKey.from_private_key_file(private_key_path)

  File "/usr/lib/python2.7/site-packages/paramiko/pkey.py", line 206, in from_private_key_file
    key = cls(filename=filename, password=password)
  File "/usr/lib/python2.7/site-packages/paramiko/rsakey.py", line 55, in __init__
    self._from_private_key_file(filename, password)
  File "/usr/lib/python2.7/site-packages/paramiko/rsakey.py", line 176, in _from_private_key_file
    self._decode_key(data)
  File "/usr/lib/python2.7/site-packages/paramiko/rsakey.py", line 185, in _decode_key
    data, password=None, backend=default_backend()

  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/primitives/serialization/base.py", line 28, in load_der_private_key
    return backend.load_der_private_key(data, password)
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1089, in load_der_private_key
    bio_data = self._bytes_to_bio(data)
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 462, in _bytes_to_bio
    data_ptr = self._ffi.from_buffer(data)
TypeError: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object

[Maiken Pedersen]

And hello again,

FYI:

I reverted to using my older elasticluster version 1.3.dev0

And with a combination of

paramiko 2.1.2

cryptography 2.6.1

It is now running.

It might also work with the newest version of paramiko, but I have not tried that yet.

Maiken

On Tuesday, May 7, 2019 at 9:46:26 AM UTC+2, Maiken Pedersen wrote:

[Riccardo Murri]

Hello Maiken,

this error seems to come from deep down the Paramiko dependency stack
-- can you please try with the latest ElastiCluster Docker image? It
was built yesterday night so it should have all the newest packages.
Instructions here:
https://elasticluster.readthedocs.io/en/latest/install.html#quickstart

Thanks,
R

[Maiken Pedersen]

Hi,

thanks for your answer!

I did try with 1.3.dev13 (downloaded via git clone though), and I saw the same problems when I had the version of paramiko 2.4.0.

In both cases (1.3.dev0 and 1.3.dev13) it works with paramiko 2.1.1 though.

Maiken

[Maiken Pedersen]

Sorry, premature reply above.

1.3.dev13 does not work with paramiko 2.1.1 (or 2.1.2) 

Best,

Maiken

[Riccardo Murri]

Hello Maiken,

> 1.3.dev13 does not work with paramiko 2.1.1 (or 2.1.2)

To me, the error seems to come from PyOpenSSL rather than Paramiko
itself? Paramiko ends up using PyOpenSSL through the `cryptography`
Python package, so probably the version of all three packages should
be compared.

I have different installations of ElastiCluster running on Paramiko
2.2.1 / cryptography 2.0.3 / PyOpenSSL 17.2.0
and Paramiko 2.4.2 / cryptography 2.6.1 / pyOpenSSL 17.5.0 and both
are running fine.

Paramiko 2.1.x dates back to 2016-2017; is there a reason why you want
to run with such an old version?

Ciao,
R

[Maiken Pedersen]

Hi,

I seem to always get into a big python module dependency mess.

With 1.3dev13 and  following (see below) modules available

I get

2019-05-07 16:41:53 elasticluster-final.novalocal gc3.elasticluster[4934] WARNING CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.

No handlers could be found for logger "paramiko.transport"

2019-05-07 16:41:53 elasticluster-final.novalocal gc3.elasticluster[4934] ERROR Error: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object

Traceback (most recent call last):

  File "/home/centos/elasticluster_07052019/src/elasticluster/__main__.py", line 206, in main
    return self.params.func()
  File "/home/centos/elasticluster_07052019/src/elasticluster/subcommands.py", line 91, in __call__
    return self.execute()
  File "/home/centos/elasticluster_07052019/src/elasticluster/subcommands.py", line 475, in execute
    cluster.start()
  File "/home/centos/elasticluster_07052019/src/elasticluster/cluster.py", line 492, in start
    started_nodes, self.start_timeout, self.ssh_probe_timeout)
  File "/home/centos/elasticluster_07052019/src/elasticluster/cluster.py", line 642, in _gather_node_ip_addresses
    timeout=ssh_timeout)
  File "/home/centos/elasticluster_07052019/src/elasticluster/cluster.py", line 1437, in connect
    ssh.connect(str(addr), port=port, **extra)
  File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 338, in connect
    t.start_client(timeout=timeout)
  File "/usr/lib/python2.7/site-packages/paramiko/transport.py", line 499, in start_client
    raise e

TypeError: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object

Aborting because of errors: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object.

(elasticluster_07052019)[centos@elasticluster-final ~]$ sudo pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
adal==1.0.1
ansible==2.7.8
apache-libcloud==2.3.0
appdirs==1.4.3
asn1crypto==0.24.0
azure==3.0.0
azure-batch==4.1.3
azure-common==1.1.12
azure-cosmosdb-nspkg==2.0.2
azure-cosmosdb-table==1.0.3
azure-datalake-store==0.0.22
azure-eventgrid==0.1.0
azure-graphrbac==0.40.0
azure-keyvault==0.3.7
azure-mgmt==2.0.0
azure-mgmt-advisor==1.0.1
azure-mgmt-applicationinsights==0.1.1
azure-mgmt-authorization==0.30.0
azure-mgmt-batch==5.0.1
azure-mgmt-batchai==0.2.0
azure-mgmt-billing==0.1.0
azure-mgmt-cdn==2.0.0
azure-mgmt-cognitiveservices==2.0.0
azure-mgmt-commerce==1.0.1
azure-mgmt-compute==3.0.1
azure-mgmt-consumption==2.0.0
azure-mgmt-containerinstance==0.3.1
azure-mgmt-containerregistry==1.0.1
azure-mgmt-containerservice==3.0.1
azure-mgmt-cosmosdb==0.3.1
azure-mgmt-datafactory==0.4.0
azure-mgmt-datalake-analytics==0.3.0
azure-mgmt-datalake-nspkg==2.0.0
azure-mgmt-datalake-store==0.3.0
azure-mgmt-devtestlabs==2.2.0
azure-mgmt-dns==1.2.0
azure-mgmt-eventgrid==0.4.0
azure-mgmt-eventhub==1.2.0
azure-mgmt-hanaonazure==0.1.1
azure-mgmt-iothub==0.4.0
azure-mgmt-iothubprovisioningservices==0.1.0
azure-mgmt-keyvault==0.40.0
azure-mgmt-loganalytics==0.1.0
azure-mgmt-logic==2.1.0
azure-mgmt-machinelearningcompute==0.4.1
azure-mgmt-managementpartner==0.1.0
azure-mgmt-marketplaceordering==0.1.0
azure-mgmt-media==0.2.0
azure-mgmt-monitor==0.4.0
azure-mgmt-msi==0.1.0
azure-mgmt-network==1.7.1
azure-mgmt-notificationhubs==1.0.0
azure-mgmt-nspkg==2.0.0
azure-mgmt-powerbiembedded==1.0.0
azure-mgmt-rdbms==0.1.0
azure-mgmt-recoveryservices==0.2.0
azure-mgmt-recoveryservicesbackup==0.1.1
azure-mgmt-redis==5.0.0
azure-mgmt-relay==0.1.0
azure-mgmt-reservations==0.1.0
azure-mgmt-resource==1.2.2
azure-mgmt-scheduler==1.1.3
azure-mgmt-search==1.0.0
azure-mgmt-servermanager==1.2.0
azure-mgmt-servicebus==0.4.0
azure-mgmt-servicefabric==0.1.0
azure-mgmt-sql==0.8.6
azure-mgmt-storage==1.5.0
azure-mgmt-subscription==0.1.0
azure-mgmt-trafficmanager==0.40.0
azure-mgmt-web==0.34.1
azure-nspkg==2.0.0
azure-servicebus==0.21.1
azure-servicefabric==6.1.2.9
azure-servicemanagement-legacy==0.20.6
azure-storage-blob==1.1.0
azure-storage-common==1.1.0
azure-storage-file==1.1.0
azure-storage-nspkg==3.0.0
azure-storage-queue==1.1.0
Babel==2.5.3
backports.ssl-match-hostname==3.4.0.2
bcrypt==3.1.4
boto==2.48.0
cachetools==2.1.0
certifi==2018.4.16
cffi==1.11.2
chardet==3.0.4
Cheetah==2.4.4
click==6.7
cliff==2.12.0
cloud-init==0.7.5
cmd2==0.8.7
coloredlogs==10.0
configobj==4.7.2
configparser==3.5.0
contextlib2==0.5.5
cryptography==2.6.1
debtcollector==1.19.0
decorator==3.4.0
deprecation==2.0.2
dogpile.cache==0.6.5
-e git://github.com/gc3-uzh-ch/elasticluster.git@74479fe9c070547e9c6cdac375d0e1087b7dcb12#egg=elasticluster
entrypoints==0.2.3
enum34==1.1.6
funcsigs==1.0.2
functools32==3.2.3.post2
future==0.17.1
futures==3.2.0
google-api-python-client==1.7.3
google-auth==1.5.0
google-auth-httplib2==0.0.3
google-compute-engine==2.8.3
httplib2==0.9.2
humanfriendly==4.12.1
idna==2.6
iniparse==0.4
ip-associations-python-novaclient-ext==0.2
ipaddress==1.0.19
IPy==0.75
iso8601==0.1.12
isodate==0.6.0
Jinja2==2.10
jmespath==0.9.3
jsonpatch==1.23
jsonpointer==1.9
jsonschema==2.6.0
keyring==12.2.1
keystoneauth1==3.13.1
kitchen==1.1.1
Markdown==2.4.1
MarkupSafe==1.0
monotonic==1.4
msgpack==0.5.6
msrest==0.5.0
msrestazure==0.4.32
munch==2.3.1
netaddr==0.7.19
netifaces==0.10.6
oauth==1.0.1
oauth2client==4.1.3
oauthlib==2.1.0
openstacksdk==0.13.0
os-client-config==1.29.0
os-diskconfig-python-novaclient-ext==0.1.3
os-networksv2-python-novaclient-ext==0.26
os-service-types==1.2.0
os-virtual-interfacesv2-python-novaclient-ext==0.20
osc-lib==1.10.0
oslo.config==6.2.1
oslo.context==2.21.0
oslo.i18n==3.20.0
oslo.log==3.38.1
oslo.serialization==2.25.0
oslo.utils==3.36.0
packaging==17.1
paramiko==2.4.2
pathlib2==2.3.2
pbr==4.0.2
perf==0.1
Pillow==2.0.0
ply==3.4
policycoreutils-default-encoding==0.1
prettytable==0.7.2
pyasn1==0.4.5
pyasn1-modules==0.2.1
pyCLI==2.0.3
pycparser==2.18
pycrypto==2.6.1
pycurl==7.19.0
Pygments==1.4
pygobject==3.14.0
pygpgme==0.3
pyinotify==0.9.6
PyJWT==1.6.4
pyliblzma==0.5.3
PyNaCl==1.2.1
pyOpenSSL==19.0.0
pyparsing==2.2.0
pyperclip==1.6.2
pyserial==3.4
python-cinderclient==3.6.0
python-dateutil==2.7.3
python-gflags==3.1.2
python-glanceclient==2.11.1
python-keyczar==0.71rc0
python-keystoneclient==3.16.0
python-neutronclient==6.9.0
python-novaclient==9.1.2
pytz==2018.4
pyudev==0.15
pyxattr==0.5.1
PyYAML==3.12
rackspace-auth-openstack==1.3
rackspace-novaclient==2.1
rax-default-network-flags-python-novaclient-ext==0.4.0
rax-scheduled-images-python-novaclient-ext==0.3.1
requests==2.21.0
requests-oauthlib==1.0.0
requestsexceptions==1.4.0
rfc3986==1.1.0
rsa==3.4.2
scandir==1.7
schema==0.6.8
SecretStorage==2.3.1
seobject==0.1
sepolicy==1.1
shade==1.28.0
simplejson==3.13.2
six==1.11.0
stevedore==1.28.0
subprocess32==3.5.2
typing==3.6.4
unicodecsv==0.14.1
uritemplate==3.0.0
urlgrabber==3.10
urllib3==1.22
virtualenv==1.10.1
warlock==1.3.0
wcwidth==0.1.7
wrapt==1.10.11
yum-metadata-parser==1.1.4

[Maiken Pedersen]

Let us close this one now. As it is not elasticluster per se, but just my dependency mess. 

Thanks!

Maiken

--
You received this message because you are subscribed to a topic in the Google Groups "elasticluster" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticluster/AeF0amoIrgE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticluste...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticluster/06f2d227-f2c5-4fb3-bd46-3478a398434c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.