Newer version of Matlab Runtime for USPEX 10.x

460 views
Skip to first unread message

Scott Warren

unread,
Aug 9, 2019, 12:52:11 PM8/9/19
to USPEX
Dear USPEX Developers,

We have USPEX 9.4.4 running successfully on our cluster, but we have been unable to get 10.3 working.  Our conclusion is that some of the libraries in Matlab Runtime v9.1 are not compatible with Centos 7.6.  Our hope is that your team can bundle a newer version of Matlab Runtime within USPEX.  For example, R2019a of MCR would be great!

The point of this post is to ask for a newer Matlab Runtime version.  The rest of this post will explain how we arrived at this conclusion.


We have built a new cluster for running USPEX.  Our cluster details are the following:

Our setup:
Warewulf / OpenHPC / Centos7.6
Slurm
VASP 5.4.4
USPEX 9.4.4 with Matlab 2019a <-- This works!
USPEX 10.3 with built-in Matlab runtime v9.1 (R2016B) and Miniconda + numpy, scipy, matplotlib, etc.
1      : whichCluster (local submission)


When we run USPEX 10.3 (for example, with tutorial EX01), the first generation proceeds without errors.  Once all calculations are converged, the log prints out:
status = Local optimization finished

Next, Matlab tries to generate several PDFs, but it fails to create them.  In the USPEX log, it generates several errors:

Warning: File 'Energy_vs_N.pdf' not found.
> In makeFigures (line 23)
  In WriteGenerationOutput_300 (line 26)
  In EA_300 (line 34)
  In Start (line 52)
  In USPEX (line 39)
Warning: File 'Fitness_vs_N.pdf' not found.
> In makeFigures (line 41)
  In WriteGenerationOutput_300 (line 26)
  In EA_300 (line 34)
  In Start (line 52)
  In USPEX (line 39)
Warning: File 'Enthalpy_vs_Volume.pdf' not found.
> In makeFigures (line 62)
  In WriteGenerationOutput_300 (line 26)
  In EA_300 (line 34)
  In Start (line 52)
  In USPEX (line 39)
Warning: File 'E_series.pdf' not found.
> In makeFigures (line 82)
  In WriteGenerationOutput_300 (line 26)
  In EA_300 (line 34)
  In Start (line 52)
  In USPEX (line 39)
Warning: File 'Variation-Operators.pdf' not found.
> In makeFigures (line 99)
  In WriteGenerationOutput_300 (line 26)
  In EA_300 (line 34)
  In Start (line 52)
  In USPEX (line 39)




I was able to "solve" this problem by placing these PDF files in the "results1" folder shortly after a new run begins.  (The file content is unimportant; all that matters is having the correct file name.)  When the first generation completes, the log prints out:

status = Local optimization finished

Then, the log continues to print out:

still_runing is present, USPEX has to exit, please see NOT_YET file for details

If I look at my Slurm "out.#" file, which I created using:
#SBATCH --output=out.%j
The out.# shows: 
Error using axes
While setting property 'Parent' of class ''Axes'':
Can't reload '/opt/ohpc/pub/libs/intel/impi/uspex/10.3/v91/bin/glnxa64/libmwosgserver.so'
Error in subplot

Error in makeFigures (line 144)

Error in WriteGenerationOutput_300 (line 26)

Error in EA_300 (line 34)

Error in Start (line 52)

Error in USPEX (line 39)




There are several Matlab website posts about the "libmwosgserver.so" error.  The best one is:


The post states:

This issue may be related to a version conflict between system libraries and the libstdc++ version included with MATLAB. You may be able to resolve this issue by making MATLAB use your system's libstdc++ instead of the libstdc++ shipped with MATLAB. While you can accomplish this in the way described by Joseph above, there is a better way:
From a Linux shell (with appropriate permissions to make modifications in your MATLAB installation, i.e. use "sudo" if needed or work with your system administrator):
1. cd into your MATLAB sys/os/glnxa64 directory.
2. Create a new directory named "old":
mkdir old
3. Move libstdc++* into the old directory:
mv libstdc++* old
Now MATLAB no longer can find its own libstdc++ and it will automatically switch to working with your system's libstdc++.

I followed these instructions but the same "libmwosgserver.so" error appeared.

An alternate solution (on the same webpage) is:

I had the same problem. I was able to solve it by copying the relevant dynamic library into place.
cp /usr/lib64/libstdc++.so.6 /usr/local/MATLAB/R2017a/bin/glnxa64

Now, when I submit a run, immediately upon starting the run, the slurm output states:
Error in USPEX (line 32)
MATLAB:builtins:MessageCatalogLoadFailed
/uspex/10.3/v91/bin/glnxa64/matlab_helper: /uspex/10.3/v91/bin/glnxa64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /uspex/10.3/v91/bin/glnxa64/matlab_helper)
/uspex/10.3/v91/bin/glnxa64/matlab_helper: /uspex/10.3/v91/bin/glnxa64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required b/uspex/10.3/v91/bin/glnxa64/libmwcpp11compat.so)
/uspex/10.3/v91/bin/glnxa64/matlab_helper: /uspex/10.3/v91/bin/glnxa64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /uspex/10.3/v91/bin/glnxa64/matlab_helper)
/uspex/10.3/v91/bin/glnxa64/matlab_helper: /uspex/10.3/v91/bin/glnxa64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /uspex/10.3/v91/bin/glnxa64/libmwcpp11compat.so)
Error using error
Unable to load a message catalog 'MATLAB:python'. Please check the file location and format.
Error in python_uspex (line 85)




There are several Matlab website posts about this error:

https://www.mathworks.com/matlabcentral/answers/329796-issue-with-libstdc-so-6

The best post states:

This issue might be due to an incompatibility between the libstdc++ shipped with MATLAB and the libstdc++ shipped with the system that the other application was linked against.
"CXXABI_1.3.9 not found" error message indicates that the libstdc++.so.6 library that is included with MATLAB is missing ABI versions the system needs to draw graphical content.
This is caused because the version of this particular library packaged with MATLAB is older than the versions that come with newer Linux operating systems, which causes compatibility issues.
You can try the following workarounds: 1. Renaming the libstdc++.so.6 library file so that MATLAB cannot find it and is forced to use the system's version of the library. This file is located in matlabroot/sys/os/glnxa64 . Renaming it to libstdc++.so.6.old should suffice. (where "matlabroot" is root installation directory of MATLAB)
2. Forcing MATLAB to load the system version by setting the $LD_PRELOAD environment variable to the newer version of libstdc++.
Also, since you are trying to call an external program from MATLAB, this may be a third party issue. But, you can try setting the library paths correctly in 'LD_LIBRARY_PATH' system variable.



This leads to a dead end for me, since I have already forced Matlab to load the system version!

Thus, I come to the conclusion that the portion of Matlab Runtime MCR R2016B that is needed to draw PDFs has libraries/requirements that are incompatible with Centos 7.6.
If you disagree with my conclusion, perhaps you can let me know what step I've done wrong.
If you agree with my conclusion, I would be very grateful if you could provide a new USPEX version with a newer Matlab Runtime.

Thank you!
Scott

Aleksandra Oranskaia

unread,
Aug 22, 2019, 3:34:25 PM8/22/19
to scottchar...@gmail.com, USPEX
Hi Scott,

Let me share my experience in case it will be helpful.

Right now on my workstation (whichCluster:0) on Ubuntu 18 I accidentally am running simultaneously USPEX-9.4.4/Matlab2019a and USPEX-10.3/in-built-Matlab 2016b :-))
I am not sure, but does not it eliminate the hypothesis of conflicting libraries?

(I did it because I was testing sth in USPEX-9.4.4, got the result and forgot to kill the job, than submitted another one with USPEX-10.3 and now in processes and logs I see both jobs running in absolutely smooth way).


Thus, I come to the conclusion that the portion of Matlab Runtime MCR R2016B that is needed to draw PDFs has libraries/requirements that are incompatible with Centos 7.6.

I have centOS 7.6.1810 on virtualbox of Ubuntu 18 (using it for USPEX-10.3-LAMMSP because this combination does not work in Ubuntu, probably it has sth to do with not 100% correct USPEX compilation). EX04 runs OK: though I do not see any pdf’s, but I do not see any complains in logs either (however, in reference folder there are pdf’s).


Best,
Alex.
___
Aleksandra Oranskaia (M.Sc.)
ChemS PhD student, KAUST
Phone: +966 50 1335254

--
You received this message because you are subscribed to the Google Groups "USPEX" group.
To unsubscribe from this group and stop receiving emails from it, send an email to uspex+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/uspex/1da93cce-b6d3-46b1-83f8-64a29f8eafc6%40googlegroups.com.


This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

Scott Warren

unread,
Aug 23, 2019, 10:25:17 PM8/23/19
to USPEX
Thanks for the reply, Aleksandra!  I agree, your information makes it seem like this is not a problem of conflicting libraries.

It is interesting that EX04 does not generate any errors in the log -- that is helpful for you!  It seems, however, our cases share the problem that the PDFs are not being generated by the active run.  If everything else works well for you, then this points to some incompatibility of PDF creation component of Matlab Runtime 2016b with Centos 7.6.

I've found, also, that the PDFs in the reference folder are not created by Matlab Runtime.  They are instead copied directly from a subdirectory in your USPEX installation.  So, unfortunately, they do not give evidence that PDFs are being created correctly by Matlab Runtime.

I would be interested to hear if other uses have this PDF creation issue in Centos 7.6.  If more cases exist, perhaps the developers will help to fix this!

Scott
To unsubscribe from this group and stop receiving emails from it, send an email to us...@googlegroups.com.

Зэд Икс

unread,
Aug 24, 2019, 12:48:09 PM8/24/19
to USPEX
Dear Scott,

USPEX 10.3 has no problem with generating pdf files. It is already more than 6 months after the release of compiled version and no such problem was reported from users or my colleagues within the group, while we use 10.3 on our Centos 7.6.18 machine.

The warnings (such as the following) that you mentioned, means that at the end of each generation, USPEX tries to delete the pdf files of the previous generation, in order to generate new pdf files for the current generation. Therefore, it is absolutely normal to face such warnings at the end of the first generation (since there are no pdf files from the previous generation that USPEX wants to remove), and these warnings will not cause any error or problem in continuing the calculations.

Warning: File 'Energy_vs_N.pdf' not found.
> In makeFigures (line 23)
  In WriteGenerationOutput_300 (line 26)
  In EA_300 (line 34)
  In Start (line 52)
  In USPEX (line 39)


As I see in your first post, your calculation faced the following error: 
Error using axes
While setting property 'Parent' of class ''Axes'':
Can't reload '/opt/ohpc/pub/libs/intel/impi/uspex/10.3/v91/bin/glnxa64/libmwosgserver.so'
Error in subplot
Error in makeFigures (line 144)


First, I need to mention that the error you faced, will NEVER stop your calculation (errors in the file makeFigures are not allowed to stop calculations). So if you see still_running, it means your calculation whether is running or has stopped with another error (you can find in your outputs).

Second, this is not USPEX bug or similar things that I can solve. It is related specifically to your system.
The best way is to find a proper solution by googling. I see the solutions that you tried, didn't help. 
Maybe you can try this one:

I suggest you remove everything and install USPEX again (follow the README file - do not forget to execute "chmod +rw ..*/archive")
Maybe better to install in the home instead of root and see if the problem exists.

Since there is a possibility, that this error occurs only in EX04, I can run this example and see if the problem appears or not.

Aleksandra Oranskaia

unread,
Aug 24, 2019, 1:57:52 PM8/24/19
to zahed.a...@gmail.com, USPEX
On Aug 24, 2019, at 7:48 PM, Зэд Икс <zahed.a...@gmail.com> wrote:

while we use 10.3 on our Centos 7.6.18 machine.

You are using developer version. Have you tested precompiled version on centOS that is included in public release?

For USPEX-10.3-LAMMPS whatever I was trying for months in Ubuntu 18, did not work (USPEX-9.4.4-LAMMPS works ok in Ubuntu 18), but it worked on centOS.
Out of curiosity I asked SergeyL. to run on Ubuntu 18 both developer and precompiled USPEX-10.3-LAMMPS: developer worked, precompiled did Not.


Best,
Alex.

___
Aleksandra Oranskaia (M.Sc.)
ChemS PhD student, KAUST
Phone: +966 50 1335254

Зэд Икс

unread,
Aug 24, 2019, 4:36:52 PM8/24/19
to USPEX
Dear Alexandra,

First of all, in this post, I'm trying to help Scott to solve the specific raised issue, and of course, when I say we use USPEX 10.3 version, I mean the compiled version and not developer version (we almost never use developer version for real calculations).

Second of all, I don't know what is the problem or error that you are talking about (whatever stopped USPEX from working), so I cannot help. 

Third, to both of you:
Just now I'm running EX04, on my Ubuntu 18.04.3. Using COMPILED version and not developer version.

At first, I had a problem, similar to what Scott mentioned (related to an older version of libstdc++.so.6) and generally, USPEX couldn't call lammps (the same problem Alexandra mentioned in the other post), so no convergence and finally getting stuck in the first generation.
But just easily by the following command (found on the internet as Scott found) I solved the problem and run the quick test for 5 generations (the test is attached and it INCLUDES PDF files).

locate libstdc++.so.6

which gives you the list of libstdc++.so libraries in the system, you can find the place of the lib with the newer version and replace it with the one in the MCR path.
Better to first rename libstdc++.so.6* in the MCR path.

For example, in my case:
cd /usr/local/MATLAB/MATLAB_Runtime/v91/sys/os/glnxa64/
sudo mv libstdc++.so.6   xxx_l_ibstdc++.so.6
sudo mv libstdc++.so.6.0.20   xxx_l_ibstdc++.so.6.0.20
sudo cp /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25   .

I used sudo, maybe you don't need. 

The problem was SOLVED in 5 minutes and was related to the libs installed in the machine that I'm using and not compiled USPEX.
No need a month for that.
test_ex04.tgz

Aleksandra Oranskaia

unread,
Aug 24, 2019, 8:09:04 PM8/24/19
to zahed.a...@gmail.com, USPEX

Aleksandra Oranskaia

unread,
Aug 24, 2019, 8:09:08 PM8/24/19
to zahed.a...@gmail.com, USPEX

Dear Zahed,


Could you please let me know how to get EX04 log for whichCluster:0 that would mention the library (that prevented USPEX from executing LAMMPS in Ubuntu 18)? Accounting for the fact, that i did NOT have any separate version of Matlab installed except USPEX’s runtime? (I installed Matlab for the 1st time a few days ago for 9.4.4 to run It with LAMMPS in Ubuntu 18).

I was reporting this thing from April (see one of the topics below, you replied there), since than i was following what a few developers were suggesting as well as trying my own solutions.


Sent from my iPhone
--
You received this message because you are subscribed to the Google Groups "USPEX" group.
To unsubscribe from this group and stop receiving emails from it, send an email to uspex+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/uspex/851b4630-4163-4960-9d5a-5f69cc99f753%40googlegroups.com.
<test_ex04.tgz>

Зэд Икс

unread,
Aug 25, 2019, 5:43:37 AM8/25/19
to USPEX
Dear Alexandra,

By default, USPEX will not output external problems that are not related to USPEX (and do not stop the code), otherwise, the output for even a well-running calculation would be messy. Although I can set another default for this specific situation.
Since these are not USPEX bugs and related to the third-party software, machine, etc., it is users' responsibility to play with the problem and find the solution for their own case.

Finding the problem here is easy though. If compiled USPEX cannot lunch lammps, just open Matlab (with the same version as USPEX), lunch lammps and see what messages from UNIX you get.

For this case, there are several msgs, but the two of them which are problematic written bellow, clearly indicating that LAMMPS requires libraries with newer versions.

/usr/bin/lmp_daily: ../sys/os/glnxa64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /usr/bin/lmp_daily)
/usr/bin/lmp_daily: ../sys/os/glnxa64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /usr/bin/lmp_daily)

To unsubscribe from this group and stop receiving emails from it, send an email to us...@googlegroups.com.
<test_ex04.tgz>
Reply all
Reply to author
Forward
0 new messages