building sire on CentOS 6.5
113 views
Skip to first unread message
julien
Oct 6, 2014, 12:47:50 PM10/6/14
to sire-de...@googlegroups.com
Hi Chris,
I am helping George who is struggling to compile a version of sire that works on CentOS 6.5
So far I have corelib and python compiled and the code seems to work. I now need to figure out how to add some extra python modules so that George's code can run.
At the moment I am trying to finish the install of setuptools and pip. However this fails
cd /build_bundled/setuptools
~/sire.app/bundled/bin/python3 setup.py install --prefix=/home/$USER/sire.app/bundled/
~/sire.app/bundled/bin/python3 setup.py install --prefix=/home/$USER/sire.app/bundled/
with this error
copying setuptools.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
creating 'dist/setuptools-0.9.8-py3.3.egg' and adding 'build/bdist.linux-x86_64/egg' to it
Traceback (most recent call last):
File "setup.py", line 204, in <module>
dist = setuptools.setup(**setup_params)
File "/home/siredeveloper/sire.app/bundled/lib/python3.3/distutils/core.py", line 148, in setup
dist.run_commands()
File "/home/siredeveloper/sire.app/bundled/lib/python3.3/distutils/dist.py", line 929, in run_commands
self.run_command(cmd)
File "/home/siredeveloper/sire.app/bundled/lib/python3.3/distutils/dist.py", line 948, in run_command
cmd_obj.run()
File "/home/siredeveloper/nautilus-source/buildpython/build_bundled/setuptools/setuptools/command/install.py", line 73, in run
self.do_egg_install()
File "/home/siredeveloper/nautilus-source/buildpython/build_bundled/setuptools/setuptools/command/install.py", line 93, in do_egg_install
self.run_command('bdist_egg')
File "/home/siredeveloper/sire.app/bundled/lib/python3.3/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/siredeveloper/sire.app/bundled/lib/python3.3/distutils/dist.py", line 948, in run_command
cmd_obj.run()
File "/home/siredeveloper/nautilus-source/buildpython/build_bundled/setuptools/setuptools/command/bdist_egg.py", line 246, in run
dry_run=self.dry_run, mode=self.gen_header())
File "/home/siredeveloper/nautilus-source/buildpython/build_bundled/setuptools/setuptools/command/bdist_egg.py", line 545, in make_zipfile
z = zipfile.ZipFile(zip_filename, mode, compression=compression)
File "/home/siredeveloper/sire.app/bundled/lib/python3.3/zipfile.py", line 900, in __init__
_check_compression(compression)
File "/home/siredeveloper/sire.app/bundled/lib/python3.3/zipfile.py", line 583, in _check_compression
"Compression requires the (missing) zlib module")
RuntimeError: Compression requires the (missing) zlib module
Comparing between the bundled python libs I get on ubuntu 12.04 and on CentOS6.5, I think python3.3 is missing some libraries to enable zlib support.
e.g. on ubuntu12.04
julien@ubuntu:~/sire.app/bundled/lib/python3.3/lib-dynload$ ll zlib.cpython-33m.so
-rwxr-xr-x 1 julien julien 78288 Sep 17 22:27 zlib.cpython-33m.so*
on CentOS 6.5
[siredeveloper@localhost lib-dynload]$ pwd
/home/siredeveloper/sire.app/bundled/lib/python3.3/lib-dynload
[siredeveloper@localhost lib-dynload]$ ll zl*
ls: cannot access zl*: No such file or directory
I am trying to figure out how to have control on the build procedure of python3.3 when compiling sire
I suspect I may need to do something here
[siredeveloper@localhost bundled]$ pwd
/home/siredeveloper/nautilus-source/python/bundled
And edit install_python.cmake to set some setup options.
Where exactly are the python source files kept in the build? I strangely cannot locate them, maybe they are deleted later during the install process ?
I also suspect I will have to do the same to enable ssl support etc...
Let me know if I am on the right track :-)
Best wishes,
Julien
Christopher Woods
Oct 7, 2014, 10:07:38 AM10/7/14
to Sire Developers
Hi Julien,
zlib can be a bit of a pain. Qt compiles and includes its own version
of zlib when it is compiled in Sire, and Python will sometimes and
sometimes not include zlib... Obviously zlib is needed to install
easy_install, pip etc... The cmake file I wrote to compile Python
should have it build in its own local copy of zlib, so it is useful to
know that this is failing on CentOS 6.5. To understand how to fix it,
I will take you now through how Python is bundled and installed with
Sire (which is very similar to how everything is bundled and
installed).
First, the bundled software is distributed as a tarball in the
"bundled" directory, e.g. python/bundled/python3.tar.gz. Each bundle
comes with an install_xxx.cmake file, e.g. install_python.cmake which
is used to configure and install the bundle. This cmake file is used
to unpack, configure, install and then set the cmake flags to link
with the libraries in each bundle. The first thing the cmake file does
is see if the bundle has been already compiled and installed into the
sire.app/bundled directory. If it has, it just sets the paths and
exits. If the bundle has not been installed, then it will then unpack
the tarball for that bundle into the build_bundled directory, e.g.
BUILD_DIR/python/build_bundled/python3. It will then run the configure
and make commands for this bundle in this build_bundled directory. In
the case of Python, it runs configure with the options placed in
${PYTHON_OPTIONS}, e.g. --enable-shared --prefix=${BUNDLE_STAGEDIR}
CC=${CMAKE_C_COMPILER} CXX=${CMAKE_CXX_COMPILER}
LDFLAGS=-WL,-rpath='$$ORIGIN/../lib'
(the last part configures python so that it has an RPATH that lets it
find all of its libraries in install_directory/../lib, e.g. if
installed in sire.app/bundled/bin, it will look in
sire.app/bundled/lib)
cmake will then compile using the command "make -k -j ${NCORES}", and,
once complete, will run "make install" (which will install to
${BUNDLE_STAGEDIR} which is actually ${CMAKE_INSTALL_PREFIX}/bundled,
i.e. /path/to/sire.app/bundled.
After this is complete, for the install_python.cmake, it will then try
to unpack and install setuptools and pip using a similar process.
However, this doesn't always work, which is why it sometimes has to be
performed manually.
Finally(!), every time cmake is run, on Apple, it will set the @rpath
on all of the bundled libraries etc. This is needed because RPATH is a
lot harder to use on OS X than Linux, as you have to actually change
the internal name of the library to allow RPATH to work (OS X links by
looking for the internal name of the library, which is not the file
name. You have to use the "install_name_tool" with OS X, which is
called ${CMAKE_INSTALL_NAME_TOOL} to set the name of the library to
"@rpath/libSomething.dylib", rather than the default
"libSomething.dylib". This is why you will see a lot of "if (APPLE)"
sections drifting around the CMakeLists.txt files...
Your assessment of the problem (missing zlib) is correct. Normally,
python has, in my experience, compiled up and used its own internal
copy of zlib which is bundled automatically in
bundled/lib/python3.3/lib-dynload/zlib.***
It looks like CentOS 6.5, for some reason, has made python not
automatically compile and install its own copy of zlib. Either python
detected that zlib was available and so used the system version (and
then this sire.app was copied to another machine), or it compiled
against a zlib, and then the RPATH stuff somehow made python forget
where zlib was installed. The solution is to find out the configure
option when compiling python to force it to compile and use its own
internal copy of zlib. Once you know which configure option to use,
then add it to ${PYTHON_OPTIONS} list in
SOURCE_DIR/python/bundled/install_python.cmake. You then have to be
careful to recompile and reinstall to new directories (i.e. create a
new build directory and make sure that you are installing to a new
directory, e.g. $HOME/sire.app doesn't exist). Otherwise either python
won't compile (it will use the old version), or the new configure
won't run or will have problems.
I hope this helps,
Best wishes,
Christopher
zlib can be a bit of a pain. Qt compiles and includes its own version
of zlib when it is compiled in Sire, and Python will sometimes and
sometimes not include zlib... Obviously zlib is needed to install
easy_install, pip etc... The cmake file I wrote to compile Python
should have it build in its own local copy of zlib, so it is useful to
know that this is failing on CentOS 6.5. To understand how to fix it,
I will take you now through how Python is bundled and installed with
Sire (which is very similar to how everything is bundled and
installed).
First, the bundled software is distributed as a tarball in the
"bundled" directory, e.g. python/bundled/python3.tar.gz. Each bundle
comes with an install_xxx.cmake file, e.g. install_python.cmake which
is used to configure and install the bundle. This cmake file is used
to unpack, configure, install and then set the cmake flags to link
with the libraries in each bundle. The first thing the cmake file does
is see if the bundle has been already compiled and installed into the
sire.app/bundled directory. If it has, it just sets the paths and
exits. If the bundle has not been installed, then it will then unpack
the tarball for that bundle into the build_bundled directory, e.g.
BUILD_DIR/python/build_bundled/python3. It will then run the configure
and make commands for this bundle in this build_bundled directory. In
the case of Python, it runs configure with the options placed in
${PYTHON_OPTIONS}, e.g. --enable-shared --prefix=${BUNDLE_STAGEDIR}
CC=${CMAKE_C_COMPILER} CXX=${CMAKE_CXX_COMPILER}
LDFLAGS=-WL,-rpath='$$ORIGIN/../lib'
(the last part configures python so that it has an RPATH that lets it
find all of its libraries in install_directory/../lib, e.g. if
installed in sire.app/bundled/bin, it will look in
sire.app/bundled/lib)
cmake will then compile using the command "make -k -j ${NCORES}", and,
once complete, will run "make install" (which will install to
${BUNDLE_STAGEDIR} which is actually ${CMAKE_INSTALL_PREFIX}/bundled,
i.e. /path/to/sire.app/bundled.
After this is complete, for the install_python.cmake, it will then try
to unpack and install setuptools and pip using a similar process.
However, this doesn't always work, which is why it sometimes has to be
performed manually.
Finally(!), every time cmake is run, on Apple, it will set the @rpath
on all of the bundled libraries etc. This is needed because RPATH is a
lot harder to use on OS X than Linux, as you have to actually change
the internal name of the library to allow RPATH to work (OS X links by
looking for the internal name of the library, which is not the file
name. You have to use the "install_name_tool" with OS X, which is
called ${CMAKE_INSTALL_NAME_TOOL} to set the name of the library to
"@rpath/libSomething.dylib", rather than the default
"libSomething.dylib". This is why you will see a lot of "if (APPLE)"
sections drifting around the CMakeLists.txt files...
Your assessment of the problem (missing zlib) is correct. Normally,
python has, in my experience, compiled up and used its own internal
copy of zlib which is bundled automatically in
bundled/lib/python3.3/lib-dynload/zlib.***
It looks like CentOS 6.5, for some reason, has made python not
automatically compile and install its own copy of zlib. Either python
detected that zlib was available and so used the system version (and
then this sire.app was copied to another machine), or it compiled
against a zlib, and then the RPATH stuff somehow made python forget
where zlib was installed. The solution is to find out the configure
option when compiling python to force it to compile and use its own
internal copy of zlib. Once you know which configure option to use,
then add it to ${PYTHON_OPTIONS} list in
SOURCE_DIR/python/bundled/install_python.cmake. You then have to be
careful to recompile and reinstall to new directories (i.e. create a
new build directory and make sure that you are installing to a new
directory, e.g. $HOME/sire.app doesn't exist). Otherwise either python
won't compile (it will use the old version), or the new configure
won't run or will have problems.
I hope this helps,
Best wishes,
Christopher
Reply all
Reply to author
Forward
0 new messages