Issue during PNDA 5.0 creation - Possibly failing to install platformlibs

[Rafael Zanetti]

Hello!

During PNDA 5.0 creation on OpenStack, the pnda-cli.py script fails with this message:

2018-12-04 13:31:31 ERROR    Error running ssh commands on host 20.2.0.56. See debug log (logs/pnda-cli.1543927880.89.log) for details.
Traceback (most recent call last):
  File "./pnda-cli.py", line 225, in <module>
    main()
  File "./pnda-cli.py", line 213, in main
    console_dns = deployment_target.create(fields)
  File "/home/centos/pnda-cli/cli/backend_base.py", line 81, in create
    self._install_pnda()
  File "/home/centos/pnda-cli/cli/backend_base.py", line 538, in _install_pnda
    self._run_salt_installation('pnda', saltmaster_ip, bastion_ip is not None)
  File "/home/centos/pnda-cli/cli/backend_base.py", line 606, in _run_salt_installation
    self._ssh_client.ssh(salt_commands, saltmaster_ip)
  File "/home/centos/pnda-cli/cli/ssh_client.py", line 119, in ssh
    raise Exception("Error running ssh commands on host %s. See debug log (%s) for details." % (host, LOG_FILE_NAME))
Exception: Error running ssh commands on host 20.2.0.56. See debug log (logs/pnda-cli.1543927880.89.log) for details.

Looking at the log file, the possible reason for the failure was a failed installation of platformlibs, as shown in this excerpt:

----------
ID: platform-libaries-easy-install
Function: cmd.run
Name: /opt/pnda/anaconda/bin/python -m easy_install /opt/pnda/platformlibs-0.2.0-py2.7.egg
Result: False
Comment: Command "/opt/pnda/anaconda/bin/python -m easy_install /opt/pnda/platformlibs-0.2.0-py2.7.egg" run
Started: 13:31:25.462134
Duration: 10.958 ms
Changes:   
----------
    pid:
          18646
    retcode:
          127
    stderr:
          /bin/sh: /opt/pnda/anaconda/bin/python: No such file or directory
    stdout:
    
Summary for anion-hadoop-edge
------------
Succeeded: 7 (changed=5)
 Failed:    1
------------

SSHing into the anion-hadoop-edge machine, I can see that under /opt/pnda there isn't any anaconda installation, so that's why the installation of platformlibs fails.

I've attached the the pnda_env.yaml and the creation log file.

Does anyone know how to deal with this issue?

Thanks!

Rafael

[trsmith2@pndaproject]

Looking at the log, the first failure is actually this one -

2018-12-04 13:12:07 - INFO - 20.2.0.56           ID: graphite-api-install-graphite

2018-12-04 13:12:07 - INFO - 20.2.0.56     Function: pkg.installed

2018-12-04 13:12:07 - INFO - 20.2.0.56         Name: graphite-api

2018-12-04 13:12:07 - INFO - 20.2.0.56       Result: False

2018-12-04 13:12:07 - INFO - 20.2.0.56      Comment: The following packages failed to install/update: graphite-api

2018-12-04 13:12:07 - INFO - 20.2.0.56      Started: 13:09:30.335189

2018-12-04 13:12:07 - INFO - 20.2.0.56     Duration: 27455.573 ms

2018-12-04 13:12:07 - INFO - 20.2.0.56      Changes:   

2018-12-04 13:12:07 - INFO - 20.2.0.56 

2018-12-04 13:12:07 - INFO - 20.2.0.56 Summary for anion-hadoop-edge

2018-12-04 13:12:07 - INFO - 20.2.0.56 --------------

2018-12-04 13:12:07 - INFO - 20.2.0.56 Succeeded: 153 (changed=129)

2018-12-04 13:12:07 - INFO - 20.2.0.56 Failed:      1

2018-12-04 13:12:07 - INFO - 20.2.0.56 --------------

What I'd recommend is ssh'ing to anion-hadoop-edge and using salt-call to run this SLS alone on that node, with -l debug. That might give you some more clues as to what is going wrong. Quite possibly it's a problem with the mirror.

HTH!

[Rafael Zanetti]

Hey trsmith!

Thanks for the reply! I followed your advice and checked that the failure on graphite-api installation was due to this error:

Transaction check error:

  file /usr/lib64/libGL.so.1 conflicts between attempted installs of libglvnd-glx-1:1.0.1-0.8.git5baa1e5.el7.x86_64 and mesa-libGL-17.2.3-8.20171019.el7.x86_64

  file /usr/lib64/libEGL.so.1 conflicts between attempted installs of libglvnd-egl-1:1.0.1-0.8.git5baa1e5.el7.x86_64 and mesa-libEGL-17.2.3-8.20171019.el7.x86_64

I've searched around but couldn't fix that. Do you have any insight on how to deal with it?

Thanks a lot,

Rafael

[trsmith2@pndaproject]

I think it's this - https://bugzilla.redhat.com/show_bug.cgi?id=1584740

The [security related] fix for this has moved a library version up - I expect the mirror will need to be updated to reflect this (i.e. needs a PR).

[Rafael Zanetti]

How could I update it manually to check if it will fix the issue?

[Donald Hunter (donaldh)]

Hi Rafael,

 

Trev and I spent a bit of time investigating the issue indicated by the error below. I cannot reproduce the error but my environment is likely to be quite different from yours.

 

The root cause seems to be that mesa-libGL and mesa-libEGL are already installed on the machine and cause a conflict that means libglvnd-glx and libglvnd-egl cannot be installed.

 

The first thing we checked is whether a new version of graphite-api has been released, with changed dependencies, but the latest version on the EPEL repo is the same version defined in the CentOS RPM dependencies for PNDA, here:

 

https://github.com/pndaproject/pnda/blob/develop/mirror/dependencies/pnda-rpm-package-dependencies-centos.txt#L32

 

There has been a new CentOS 7.6.1810 release which is now picked up by default when running yum install from a 7.5 CentOS machine. This can be a cause of problems but you have configured ADD_ONLINE_REPOS: "NO" in your pnda_env.yaml file so it shouldn’t happen at install time.

 

There are a few possibilities:

 

1. Your mirror was built with incompatible versions of libs for some reason, possibly caused by the CentOS 7.6.1810 release. You could check which versions of libglvnd-glx, libglvnd-egl, mesa-libGL and mesa-libEGL are present in the RPM directory on your mirror.
2. The CentOS base image used for the mirror and the CentOS base image used for the Openstack launched VMs is not the same.
3. The CentOS base image used for the Openstack VMs is perhaps a desktop image or has packages installed that required mesa-libEGL and mesa-libGL to be installed. You could try removing mesa-libGL to see what causes it to be installed.

 

You could try manually running yum install graphite-api to see if you get a more detailed error.

 

I hope something here helps you debug further.

 

Cheers,

Donald.

--
You received this message because you are subscribed to the Google Groups "PNDA Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pnda-users+...@googlegroups.com.
To post to this group, send email to pnda-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pnda-users/90263fbb-d662-4233-9a35-fe282ec6795e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rafael Zanetti]

Hey Donald,

First of all, thanks to you and Trev for the time spent on this issue.

Second, to provide an update on the issue, I had manually tried to install graphite-api, but i didn't get any new information about the issue. Then, I tried to manually install mesa-libEGL to see what would happen, and turns out that after installing it, i was able to install graphite-api. So I modified the pnda installation script to individually install mesa-libEGL before installing the other packages, and it solved this issue.

Now the installation is failing for another reason (some issue with Cloudera Manager) and I'm trying to fix that.

Finally, on an off-topic note, is any of the other ways to build PnDA (AWS, VMWare, bare matel) more recommended than OpenStack to deploy PnDA?

To post to this group, send email to pnda...@googlegroups.com.

[Donald Hunter (donaldh)]

Hi Rafael,

I am glad to hear you have made some progress. I will try out your mesa-libEGL solution and see if it provides a robust fix for this problem – though I won’t get to it before January.

All the PNDA install mechanisms share the same Saltstack based install process after the VMs are started and configured with networking and storage. AWS uses cloud-formation templates followed by Saltstack install. Openstack uses heat templates followed by Saltstack install, etc. The hardest to do, but most flexible is probably the existing-machines install because it is a manual process to set up bare-metal or VMs with networking and storage before running pnda-cli to launch the Saltstack install.

Cheers,

Donald.