Resize - or setup - elasticluster.sh - /usr/bin/eatmydata: No such file or directory\n",

[Maiken Pedersen]

Hi again,

so managed to get the newest version of elasticluster working through the elasticluster.sh.

I am trying to resize my cluster.

Since the resize step failed at some point late in the setup step earlier, I need to run it again.

./elasticluster.sh setup $clustername

or

./elasticluster.sh resize -a 0:compute $clustername -t slurm

gives the following:

fatal: [compute003]: FAILED! => {"changed": false, "module_stderr": "/usr/bin/python+eatmydata: line 3: /usr/bin/eatmydata: No such file or directory\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 127}

What is the trick to fix this?

This is on a centos7 cluster.

[centos@elasticluster-final ~]$ ./elasticluster.sh  --version
elasticluster version 1.3.dev13

[Maiken Pedersen]

I "solved it" adding

[setup/ansible-slurm]

slow_but_safer=True

according to the docs/configure.rst

[Riccardo Murri]

Hello Maiken,

> ./elasticluster.sh resize -a 0:compute $clustername -t slurm
>
> gives the following:
>
> fatal: [compute003]: FAILED! => {"changed": false, "module_stderr": "/usr/bin/python+eatmydata: line 3: /usr/bin/eatmydata: No such file or directory\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 127}
>

Sorry, this is definitely a bug in ElastiCluster. Does it happen
only on `compute003` or on other nodes as well?

Can you please post the output of the very first Ansible task
("Prepare VMs for running Ansible") from a run where you get that
error?

Ciao,
R

[Riccardo Murri]

Hello Maiken,

this bug should be fixed in the latest version of the "master" branch.
To upgrade your Docker image, run:

elasticluster.sh --pull --latest

Can you please try and confirm that the fix works? I'll tag 1.3.dev14
once I get confirmation.

Ciao,
R

[Maiken Pedersen]

Hi,
Very nice, thank you very much. I will give it a try tomorrow!
Maiken

[Maiken Pedersen]

I never got that far, but now at last I am ready to give it a try. I will tell you how it went.

[Maiken Pedersen]

Hi,

took a while, but I am finally trying the resize again now with elasticluster version 1.3.dev15

Unfortunately I still see the eatmydata problem:

fatal: [computeb002]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 70, "stderr": "Shared connection to 158.39.48.117 closed.\r\n", "stderr_lines": ["Shared connection to 158.39.48.117 closed."], "stdout": "Running installation command 'yum install -y python2 python2-simplejson' ...\r\nLoaded plugins: fastestmirror\r\nDetermining fastest mirrors\r\n * base: ftp.uninett.no\r\n * extras: ftp.uninett.no\r\n * updates: ftp.uninett.no\r\n\rbase                                                     | 3.6 kB     00:00     \r\n\rextras                                                   | 3.4 kB     00:00     \r\n\rupdates                                                  | 3.4 kB     00:00     \r\n\r(1/4): base/7/x86_64/group_gz                              | 166 kB   00:00     \r\n\r(2/4): extras/7/x86_64/primary_db                          | 200 kB   00:00     \r\n\r(3/4): base/7/x86_64/primary_db                            | 6.0 MB   00:00     \r\n\r(4/4): updates/7/x86_64/primary_db                         | 5.0 MB   00:00     \r\nPackage python-2.7.5-77.el7_6.x86_64 already installed and latest version\r\nNo package \u001b[1mpython2-simplejson\u001b(B\u001b[m available.\r\nNothing to do\r\nLoaded plugins: fastestmirror\r\nLoading mirror speeds from cached hostfile\r\n * base: ftp.uninett.no\r\n * extras: ftp.uninett.no\r\n * updates: ftp.uninett.no\r\nhttps://copr-be.cloud.fedoraproject.org/results/loveshack/livhpc/epel-6-/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found\r\nTrying other mirror.\r\nTo address this issue please refer to the below wiki article \r\n\r\nhttps://wiki.centos.org/yum-errors\r\n\r\nIf above article doesn't help to resolve this issue please use https://bugs.centos.org/.\r\n\r\nNo package \u001b[1mlibeatmydata\u001b(B\u001b[m available.\r\nError: Nothing to do\r\ninstall-py2.sh: ERROR: Binary wrapper '/usr/bin/eatmydata' not found, even after installation. Aborting.\r\n", "stdout_lines": ["Running installation command 'yum install -y python2 python2-simplejson' ...", "Loaded plugins: fastestmirror", "Determining fastest mirrors", " * base: ftp.uninett.no", " * extras: ftp.uninett.no", " * updates: ftp.uninett.no", "", "base                                                     | 3.6 kB     00:00     ", "", "extras                                                   | 3.4 kB     00:00     ", "", "updates                                                  | 3.4 kB     00:00     ", "", "(1/4): base/7/x86_64/group_gz                              | 166 kB   00:00     ", "", "(2/4): extras/7/x86_64/primary_db                          | 200 kB   00:00     ", "", "(3/4): base/7/x86_64/primary_db                            | 6.0 MB   00:00     ", "", "(4/4): updates/7/x86_64/primary_db                         | 5.0 MB   00:00     ", "Package python-2.7.5-77.el7_6.x86_64 already installed and latest version", "No package \u001b[1mpython2-simplejson\u001b(B\u001b[m available.", "Nothing to do", "Loaded plugins: fastestmirror", "Loading mirror speeds from cached hostfile", " * base: ftp.uninett.no", " * extras: ftp.uninett.no", " * updates: ftp.uninett.no", "https://copr-be.cloud.fedoraproject.org/results/loveshack/livhpc/epel-6-/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found", "Trying other mirror.", "To address this issue please refer to the below wiki article ", "", "https://wiki.centos.org/yum-errors", "", "If above article doesn't help to resolve this issue please use https://bugs.centos.org/.", "", "No package \u001b[1mlibeatmydata\u001b(B\u001b[m available.", "Error: Nothing to do", "install-py2.sh: ERROR: Binary wrapper '/usr/bin/eatmydata' not found, even after installation. Aborting."]}

[Riccardo Murri]

Hello Maiken,

sorry for the late reply - I am on vacation until end of June. Can
you please open a bug at
https://github.com/gc3-uzh-ch/elasticluster/issues -- I will take a
look at it when I'm back.

Meanwhile, the workaround is to add:

safe_but_slower=yes

to your cluster's `[setup/*]` section.

Ciao,
R

[Maiken Pedersen]

Very good, will do. Added the workaround as I remembered it from last time. Thanks.

Maiken

[Riccardo Murri]

Hello Maiken, all,

creation of CentOS 7.x and 6.x clusters with the default settings
(`slow_but_safer=no`, i.e., use "eatmydata" to speed up operations)
should now be fixed in the current "master" branch -- can you please
give it a try?

Ciao,
R