Problem with MATLAB tests

[Bryan W. Weber]

Hi Ray,

Today I was testing the MATLAB install procedure, and there seem to be some problems with the MATLAB test suite run by SCons. Specifically, the testImportCTI test fails because python couldn't find ctml_writer.py. I tracked it down to two problems (based on r2637):

1) The PYTHONPATH is specified as SOURCE_ROOT/interfaces/matlab/../../build/python2 but the ctml_writer.py file is actually in SOURCE_ROOT/interfaces/matlab/../../build/python2/cantera

2) The PYTHONPATH is not set if it exists in the users environment. From interfaces/matlab/testpath.m:

% Set path to Python module

if strcmp(getenv('PYTHONPATH'), '')

    setenv('PYTHONPATH', fullfile(cantera_root, 'build', 'python2'))

end

#1 just seems like a simple bug and if I comment out the if/end statements to fix #2, the tests pass. But if I print the value of PYTHONPATH set by the test module, i.e. 

% Set path to Python module

%if strcmp(getenv('PYTHONPATH'), '')

    setenv('PYTHONPATH', fullfile(cantera_root, 'build', 'python2', 'cantera'))

    disp(getenv('PYTHONPATH'))

%end

the PYTHONPATH displayed is SOURCE_ROOT/interfaces/matlab/../../build/python2/cantera

which doesn't even contain the original value of my PYTHONPATH. Further, after the tests have passed with these changes, if I

echo $PYTHONPATH

from the shell, my original PYTHONPATH is printed without the additional MATLAB testing path. So is there a reason to check if the user has defined their PYTHONPATH? Of course, in case there is a reason, the easy fix would be to add an else clause to append the appropriate path to PYTHONPATH.

Thanks!

Bryan

[Ray Speth]

Bryan,

I'm not sure why this is causing problems on your system. The correct PYTHONPATH entry is just "<cantera root>/build/python2". The ctml_writer module should be in the 'cantera' package, so that it is imported as "from cantera import ctml_writer". You can see the Python code that's being run through Matlab in ct2ctml.cpp. There's a try/except block to check for ctml_writer in the location that it was for the old Python module, where it was imported using just "import ctml_writer". What happens if you run, from the cantera root directory:

    PYTHONPATH=build/python2 python

Then, in the Python shell:

    from cantera import ctml_writer

    print ctml_writer.__file__

I think the reason for checking for the PYTHONPATH was to make it possible to run the test suite outside of SCons. When running the Matlab test suite from SCons, the PYTHONPATH environment variable is actually determined in test/SConscript. If you put the disp(getenv('PYTHONPATH')) call before the 'if' statement, you should see that it's already set to <cantera_root>/build/python2. The reason for overwriting PYTHONPATH is to make sure that the tests are always running using the version of ctml_writer that is in the Cantera source tree, rather than an installed copy of Cantera. Since ctml_writer doesn't depend on anything outside the Python standard library, it shouldn't be necessary to include anything else in the PYTHONPATH.

Also, just to clarify: Environment variables are generally passed down from a parent process to its children. However, any changes in those variables in the child process do not propagate back up to the parent. This is why the environment variables in your shell remain unchanged.

Regards,

Ray

[Bryan W. Weber]

Hi Ray,

Here is the input line and output:

$ PYTHONPATH=build/python2 python -c "from cantera import ctml_writer; print ctml_writer.__file__"

/home/bryan/software/cantera-dev/build/python2/cantera/ctml_writer.pyc

So clearly Python can find the file in that case. The actual error written to the screen from the test is:

-------------- start of converter log --------------

Traceback (most recent call last):

  File "<stdin>", line 7, in <module>

ImportError: No module named ctml_writer

--------------- end of converter log ---------------

so based on the code in ct2ctml, it seems like 'from cantera import ctml_writer' throws an ImportError exception that gets caught, so 'import ctml_writer' is tried. Why the first ImportError is thrown is a mystery to me. 

If I print the 'sys.path' in that python segment in ct2ctml.cpp, the <cantera root>/build/python2 is the first entry after empty string, so it seems the proper directory is on the path. Furthermore, if I print sys.path when I check from python if I can import ctml_writer (as I did earlier), it is identical to the path printed from ct2ctml.cpp.

Bryan

[Bryan W. Weber]

Hi Ray,

I found the solution. The version of libgfortran that comes with MATLAB R2012a is incompatible with the liblapack installed by apt-get on Ubuntu 12.04.3.

/usr/local/MATLAB/R2012a/sys/os/glnxa64/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3gf)

I found that this was the error by catching the exception in ct2ctml.cpp and printing it to the screen. I had to symlink the libgfortran.so.3.0.0 from /usr/lib/x86_64-linux-gnu to $MATLAB_ROOT/sys/os/glnxa64 

It seems weird that loading the MEX file went fine until it tried to load the ctml_writer in Python. It also seems weird that adding the 'cantera' directory to PYTHONPATH fixed the problem. Anyways, I'm glad its fixed now.

Bryan

[Bryan W. Weber]

Hi Ray,

Now that I've got the install working, I've got some follow up questions. I've got the install to work with a virtual Python environment so I can compile the svn version without blowing away my working 2.1 release, hence the .virtualenvs folders below.

My first question is: why does it appear that the install is duplicated in <prefix>/lib and <prefix>/local/lib? That is, I specified my prefix in cantera.conf - /home/bryan/.virtualenvs/cantera-matlab - and the lib and local/lib directories underneath the prefix each contain what appears to be a self contained install. I don't have the python prefix option set in cantera.conf, so it shouldn't be installing in the USER mode...

My second question is related to the output that is printed when the install finishes. In particular, it says:

"An m-file to set the correct matlab path for Cantera is at:  /home/bryan/.virtualenvs/cantera-matlab/matlab/ctpath.m"

but the <prefix>/matlab directory doesn't exist. "ctpath.m" does exist at <prefix>/lib/cantera/matlab/toolbox though. Also, in the line:

"Matlab samples              /home/bryan/.virtualenvs/cantera-matlab/lib/cantera/matlab/toolbox/matlab"

the ".../toolbox/matlab" directory doesn't exist (of course ".../toolbox" does). Are these mistakes or am I missing something?

Thanks!

Bryan

[Ray Speth]

Hi Bryan,

Matlab's habit of including outdated copies of system libraries like libgfortran and libstdc++ have caused me plenty of headaches too. What I've done in the past as a workaround is to copy the .matlab7rc.sh file supplied by Matlab (in the 'bin' directory) to my home directory, and modify it to set LDPATH_PREFIX to a directory somewhere in my home directory, to which I copy the system versions of libgcc_s.so, libgfortran.so, and libstdc++.so.

The install action shouldn't be duplicating any files under <prefix>. In fact, I don't think anything should be installed in <prefix>/local. Can you remove all the Cantera-related files from <prefix> and save the output of running 'scons install'? If it is duplicating files, I'd be interested in seeing that output.

All of the issues with the paths printed at the end of 'scons install' are just mistakes. All the Matlab toolbox files including ctpath.m should be in <prefix>/lib/cantera/matlab/toolbox and the samples should be in <prefix>/share/cantera/samples/matlab. It looks like I missed updating these messages when I was updating the toolbox's installation path.

Regards,

Ray

[Bryan W. Weber]

Hi Ray,

Thanks for getting back to me. I'll keep that tip about MATLAB in mind in the future. Is there any way for the testing script to let the user know that the error is not in the install or the import but in that MATLAB is having issues, especially because this seems to not be an uncommon problem? In my case, it was as simple as passing the error into the except clause and printing it.

I've attached my SCons install output and the cantera.conf file. The install sequence was:

make virtual environment and activate it

pip install numpy scipy cython 3to2 Sphinx pygments matplotlib sphinxcontrib-doxylink

Update cantera.conf to the correct prefix

scons build && scons test && scons install

I've got outputs for the build and test segments too, if you want them. It doesn't seem like anything in the SCons install output would suggest that files are going to <prefix>/local. However, the <prefix>/lib and <prefix>/local/lib directories have identical structure as best I can tell. 

To add the proper directories to the MATLAB path, would you recommend adding "run path/to/ctmethods.m" to a "startup.m" file, or running the setup_cantera script?

Thanks!

Bryan

[Ray Speth]

Bryan,

With the removal of the old Python module, the ctml_writer.py file will always be inside the cantera Python package, so I think the try/except construct can be removed from ct2ctml, which will let the error propagate through.

I found this Stack Overflow question:

    http://stackoverflow.com/questions/8227120/strange-local-folder-inside-virtualenv-folder

which seems to suggest that the <prefix>/local folder is just a symlink to <prefix>, generated by virtualenv. If so, this isn't a problem with the Cantera install process, and you don't actually have any duplicate files.

I would say that adding 'run <prefix>/lib/cantera/matlab/toolbox/ctpath.m' to your startup.m is a better solution than always needing to run setup_cantera, but either should work.

Regards,

Ray

[Bryan W. Weber]

Hi Ray,

Thanks for the advice and all the help! Happy holidays!

Bryan