sire build issues

[julien]

Hi Chris, 

I hope you are enjoying the conference. 

Thanks for the answers on docstrings and unit tests. 

I have made good progress and I think I should be able to ask George to finish the rest of the work soon.

There is one install problem left that I would like to discuss with you. 

It is quite useful to be able to install bumpy and scipy to the sire python install. Nautilus needs those as dependencies. However scipy needs to have working blas/lapack libraries 

At the moment I am using the versions provided by ubuntu package but I think it would make sense to ship those with sire to solve installation problems on a system where the user doesn’t have root permissions. 

Now I can see that in corelib cmake has already a SIRE_BUNDLEBLASLAPACK option which is on by default. 

However src/bundled doesn’t have BLAS/LAPACK source files. 

Is there a reason why you started including this dependency and then backed off or it is just  unfinished work ? 

Another issue I have is that every time I check out a fresh version of my branch, or that I touch a CMAKELists file I get a CMake error in core lib when recompiling because of :

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.

Please set them or make sure they are set and tested correctly in the CMake files:

TBB_LIBRARY

    linked by target "SireMM" in directory /home/julien/software/devel/sirejulien/corelib/src/libs/SireMM

TBB_MALLOC_LIBRARY

    linked by target "SireMM" in directory /home/julien/software/devel/sirejulien/corelib/src/libs/SireMM

This is easily solved by 

‘ccmake’, ‘c’, ‘g’ , ‘make'

By default SIRE_BUNDLE_TBB is ON

It does look like tbb is built and installed, so it sounds like a bug in the cmake build protocol.

If these two problems, and the pip auto build bug are fixed, then it would be relatively easy to write install instructions to add Nautilus support to an able version of Sire. 

It would look roughly like this at the moment

# Install

~/sire.app/bundled/bin/pip install numpy

~/sire.app/bundled/bin/pip install cython

~/sire.app/bundled/bin/pip install scipy

~/sire.app/bundled/bin/pip install mdtraj

cd $SIRESOURCE/python2

tar -xvf nautilus-patch.tar

cd Tools

patch < CMakeLists.patch

cd $SIRESOURCE/buildpython2

make install

# Test

cd $SIRE/test/Tools

$SIRE/bin/python test_nautilus.py

When a public release of Nautilus becomes possible, we could adapt the above instructions to update the Sire python2  build procedure (with maybe a SIRE_ENABLE_NAUTILUS flag)

A potential weakness is that this procedure may fail in the future if the python dependencies become backward incompatible. 

On the other hand it sounds like a maintenance nightmare to bundle everything. I think we are better off in general to proceed this way and fix issues as they appear. 

Does this sound all sensible to you  ?

Best wishes, 

Julien

[Christopher Woods]

Hi Julien,

> I hope you are enjoying the conference.

Yes, it was very good, but I am now feeling information overload ;-)

> Thanks for the answers on docstrings and unit tests.
>
> I have made good progress and I think I should be able to ask George to
> finish the rest of the work soon.

Excellent

> There is one install problem left that I would like to discuss with you.
> It is quite useful to be able to install bumpy and scipy to the sire python
> install. Nautilus needs those as dependencies. However scipy needs to have
> working blas/lapack libraries

See my answer to Sajid. I know you need numpy and scipy, but this will
complicate the installation. Unfortunately python 'hides' every time
it uses gcc/g++ to compile code when it builds modules. This means
that I don't know which non-system libraries are linked to python
modules and so cannot ensure that these will be correctly bundled in
the binary installation. Extra python modules have to be compiled by
the user using the python that is bundled with Sire (which will do its
best to link to the bundled libraries), but these compilations may
fail if the bundle or the system is missing one of the libraries
needed by the new python module (e.g. if libssl.so.6 is missing, as in
the case of Sajid or my test machine).

> At the moment I am using the versions provided by ubuntu package but I think
> it would make sense to ship those with sire to solve installation problems
> on a system where the user doesn’t have root permissions.

I don't think that this will work as the binary packages provided by
ubuntu are unlikely to be binary compatible with the python libraries
bundled or compiled with Sire. I think that you will need to compile
and install the libraries using pip and then use "ldd" to track down
all the dependent libraries and add them into the bundle. You will
then need to learn how to adjust the RPATH of libraries so that these
can then be found in the bundle without the user having to manually
set LD_LIBRARY_PATH (I can show you how to do this, but it can be a
little tricky).

> Now I can see that in corelib cmake has already a SIRE_BUNDLEBLASLAPACK
> option which is on by default.
>
> However src/bundled doesn’t have BLAS/LAPACK source files.
>
> Is there a reason why you started including this dependency and then backed
> off or it is just unfinished work ?

This is the BLAS/LAPACK library that is used by SireMaths/NMatrix and
SireMaths/NVector to provide various matrix / vector operations (e.g.
see src/libs/SireMaths/sire_blas.h and sire_lapack.h). This library is
hidden from the other parts of Sire to make sure that it won't clash
with any other BLAS/LAPACK library that may be used in the code (as I
wanted to bundle a 'slow' blas library that is guaranteed to work
everywhere, so I could provide the functionality that is only exposed
in NMatrix and NVector. You can assume that this blas/lapack library
doesn't exist, as it is not visible outside of SireMaths.

> Another issue I have is that every time I check out a fresh version of my
> branch, or that I touch a CMAKELists file I get a CMake error in core lib
> when recompiling because of :
>
>
> CMake Error: The following variables are used in this project, but they are
> set to NOTFOUND.
>
> Please set them or make sure they are set and tested correctly in the CMake
> files:
>
> TBB_LIBRARY
>
> linked by target "SireMM" in directory
> /home/julien/software/devel/sirejulien/corelib/src/libs/SireMM
>
> TBB_MALLOC_LIBRARY
>
> linked by target "SireMM" in directory
> /home/julien/software/devel/sirejulien/corelib/src/libs/SireMM
>
>
> This is easily solved by
>
> ‘ccmake’, ‘c’, ‘g’ , ‘make'
>
>
> By default SIRE_BUNDLE_TBB is ON
>
> It does look like tbb is built and installed, so it sounds like a bug in the
> cmake build protocol.

Yes - this is a problem for me as well. I haven't been able to fix it.
You can work around it by running "cmake ." in the top level build
directory every time that you add a new file (the problem seems to be
that re-running cmake from anything other than the top build directory
causes the above error). You just need to run "cmake ." and don't need
to run "ccmake' followed by 'c, g, make'. After you have run "cmake ."
you can run "make" in any directory and it will work.

> If these two problems, and the pip auto build bug are fixed, then it would
> be relatively easy to write install instructions to add Nautilus support to
> an able version of Sire.

Yes... although I think from what I said above that you may find that
compilation of scipy/numpy will cause lots of unknown problems based
on which libraries / compilers etc. are installed on the users'
machines.

> It would look roughly like this at the moment
>
> # Install
>
> ~/sire.app/bundled/bin/pip install numpy
>
> ~/sire.app/bundled/bin/pip install cython
>
> ~/sire.app/bundled/bin/pip install scipy
>
> ~/sire.app/bundled/bin/pip install mdtraj
>
> cd $SIRESOURCE/python2
>
> tar -xvf nautilus-patch.tar
>
> cd Tools
>
> patch < CMakeLists.patch
>
> cd $SIRESOURCE/buildpython2
>
> make install
>
> # Test
>
> cd $SIRE/test/Tools
>
> $SIRE/bin/python test_nautilus.py
>
> When a public release of Nautilus becomes possible, we could adapt the above
> instructions to update the Sire python2 build procedure (with maybe a
> SIRE_ENABLE_NAUTILUS flag)
>
> A potential weakness is that this procedure may fail in the future if the
> python dependencies become backward incompatible.
>
> On the other hand it sounds like a maintenance nightmare to bundle
> everything. I think we are better off in general to proceed this way and fix
> issues as they appear.
>
> Does this sound all sensible to you ?

This sounds sensible, and it will be a maintenance nightmare to bundle
everything. However, python's module system is so convoluted that we
may need to put in the effort to bundle (or at the very least provide
a binary download that can be patched into a binary install of Sire).
Solving people's compile and install problems is extremely time
consuming and installing from source or via a complicated set of
commands is a major barrier to adoption. If we can bundle it all then
we should. Ideally then people could just download a "nautilus.run"
file, and then unpack that to give them sire.app/bin/nautilus (or even
nautilus/run, which calls nautilus/sire.app/bin/nautilus). I don't
think that it is a big issue if different Sire apps all end up
shipping with different copies of Sire. Indeed, this may be simpler
than trying to build a single Sire binary bundle that supports all of
the different python modules that different Sire apps will need...

What do you think?

Christopher

[julien]

Hi Chris, 

I think it would be a mistake to not make it easier to link Sire with other python modules. There are many other useful software libraries with a python interface in development and it is very important that we maximise interoperability between different software libraries 

I never had problems using numpy with Sire. Half of Nautilus uses numpy because for some operations the code was faster (in our hands) than what could be done with Sire datastructures. We just had to be careful with the ways modules are imported to avoid name clashes (e.g. import numpy as np). The majority of potential developers will be more comfortable with Python than C++ and it is only natural that they will want to use the modules they already know well.  

Scipy just needs a statically linked blas library. I don’t think it should be so hard to compile the blas sources to link to this library as part of the installation process.   

I agree with you that it would be a headache to provide a general support for any dependencies for a python library. However I don’t advocate we do that. I suggest we support blas/lapack  so we can have numpy (which a large portion of python developers will want immediately). We document how we added blas/lapack support in sire so that developers could further extend the build procedure should they wish to include other libraries.  If possible we recommend to link against static libraries. 

I don’t have any problems using the blas/lapack libraries from the Ubuntu packages on my laptop (probably because static). However this isn’t clearly the way to distribute the code. 

/This sounds sensible, and it will be a maintenance nightmare to bundle 

/everything. However, python's module system is so convoluted that we 

/may need to put in the effort to bundle (or at the very least provide 

/a binary download that can be patched into a binary install of Sire). 

/Solving people's compile and install problems is extremely time 

/consuming and installing from source or via a complicated set of 

/commands is a major barrier to adoption. If we can bundle it all then 

/we should. Ideally then people could just download a "nautilus.run" 

/file, and then unpack that to give them sire.app/bin/nautilus (or even 

/nautilus/run, which calls nautilus/sire.app/bin/nautilus). I don't 

/think that it is a big issue if different Sire apps all end up 

/shipping with different copies of Sire. Indeed, this may be simpler 

/than trying to build a single Sire binary bundle that supports all of 

/the different python modules that different Sire apps will need... 

/

/What do you think? 

Sounds like wine for molecular simulations !  I think this is a good solution to provide the software to users. If you can show me how this could be done by making a bundle of one app, I would be happy to test this way of packaging sire applications for Nautilus or somd (that will ship with mdtraj/numpy/openmm) and to write a documentation describing the build procedure that other developers could adapt to make their own bundle.  

Clearly we have two different problems: 1) how to make the code useful to developers of other simulation software 2) how to make the code easy to run for end-users. I think both categories of users are very important but they have different needs/abilities and a different solution is required. 

Best wishes, 

Julien 

[Christopher Woods]

Hi Julien,

I see your point, and don't want to deliberately make it difficult to
combine Sire with other Python libraries. Certainly numpy/scipy are so
widely used that we can consider supporting them to be a priority (and
not too difficult, as you are showing).

BLAS is difficult to bundle because of the issue of it being a
performance library. A statically linked blas library will be less
efficient than one of the optimised libraries, and the decision of
which blas library to use is normally made by a cluster manager or by
the user at install time. It is similar to the OpenMM problem - namely
that a bundled OpenMM library may not be as efficient as one that has
been tuned and installed for a specific cluster. If we bundle blas (or
OpenMM), then we need to find a way to allow the bundled library to be
overridden by the system library. There are mechanisms to do this, but
it adds a layer of complexity and source of bugs that I have so far
being trying to avoid... But I do accept that it is something that we
will need to do. Ultimately, with blas a slow implementation that is
functional is better than nothing ;-).

In terms of dependencies, scipy needs more than just blas. If you pull
in all of scipy then you also get matplotlib, that depends on X, and
also depends on the (comparatively) new version of freetype. This
caused me no end of problems at the Monte Carlo workshop a couple of
weeks back, as the freetype on the linux distribution of those
desktops was too old for matplotlib, leading me to have to compile it
all from scratch. It is possible to install only parts of scipy so
that we avoid this problem, but we then hit the other problem that a
normal scipy user may expect all of scipy to be available (and indeed,
as these are also all quickly developing codes, they may expect the
latest version of scipy to be available, which may not fit into our
test and release schedule).

This is why I advocate the option of making it easy for other people
to install (from source via pip) any python module into Sire, but that
we don't ship Sire with any modules (beyond easy_install, pip and
nosetools). However, when we ship a Sire app, we bundle everything
that we need for that app, in the knowledge that users of the app will
only be using the front-end, and won't expect to encounter or play
with a python interpreter. So, we would bundle a compiled version of
the necessary parts of scipy/numpy with nautilus and distribute that
as nautilus.app

>> Sounds like wine for molecular simulations ! I think this is a good
>> solution to provide the software to users. If you can show me how this could
>> be done by making a bundle of one app, I would be happy to test this way of
>> packaging sire applications for Nautilus or somd (that will ship with
>> mdtraj/numpy/openmm) and to write a documentation describing the build
>> procedure that other developers could adapt to make their own bundle.

Yes - it is a little like wine. The way you would build the app would
be to compile up a sire.app which contains nautilus on an old Linux
distribution (I am using Scientific Linux 5 - the same one as for
generating the python wrappers). Then, once nautilus is working, you
will need to use "ldd" to work out which libraries are used by
scipy/numpy. You will either need to compile these libraries again to
set their prefix to $SIRE/bundled or you could just copy them into
$SIRE/bundled/lib. Sire uses RPATH to allow binaries to find shared
libraries in a directory that is relative to the install directory of
the binary (see http://www.cmake.org/Wiki/CMake_RPATH_handling). The
Sire binaries are set up to look for shared libraries in "../lib" and
"../bundled/lib" (e.g. if $SIRE/bin/python was in
/home/chris/sire.app/bin/python, then it will look for libraries in
/home/chris/sire.app/bin/../lib and
/home/chris/sire.app/../bundled/lib). Enabling RPATH can be a pain as
you have to add it in at the link step when making the library and
executable (on OS X to set the name of the library to
@rpath/library_name.dylib and to tell each executable to look for
libraries in @executable_path/.:@executable_path/../${SIRE_LIBS}:@executable_path/../${SIRE_BUNDLED_LIBS},
and on linux you don't need to rename libraries, but you must tell
each executable to look in
\$ORIGIN/.:\$ORIGIN/../${SIRE_LIBS}:\$ORIGIN/../${SIRE_BUNDLED_LIBS} -
this is all set up in the CMakeLists.txt in the top level of corelib,
and also in the CMakeLists.txt in the top level of python - just
search for RPATH in these files).

Once you have got it working, there is a "package_sire" script that
packages up the current binary directory into a sire.run installer.
Just copy package_sire and edit it to make package_nautilus to package
it up into a nautilus.run (and then extract to nautilus.app - this
will just require renaming sire to nautilus). Then it should all be
ready to go ;-) (I am sure it will, in reality, be much more painful
than this...)

>> Clearly we have two different problems: 1) how to make the code useful to
>> developers of other simulation software 2) how to make the code easy to run
>> for end-users. I think both categories of users are very important but they
>> have different needs/abilities and a different solution is required.

Agreed :-)

Best wishes,

Christopher

[julien]

    But if I get it right, there is no need to specify individual additional libraries in the install files. All executables in sire.app/bin/ ''know'' to check ../lib and ../bundled/lib/

So for instance, to compile setuptools for python I add to add zlib-devel to CentOS with yum install zlib-devel

Know I have the required library in 

/home/siredeveloper/sire.app/bundled/lib/python3.3/lib-dynload

[siredeveloper@localhost lib-dynload]$ ldd zlib.cpython-33m.so 

linux-vdso.so.1 =>  (0x00007fff087ff000)

libz.so.1 => /lib64/libz.so.1 (0x00007f518e102000)

libpython3.3m.so.1.0 => not found

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f518dee4000)

libc.so.6 => /lib64/libc.so.6 (0x00007f518db50000)

/lib64/ld-linux-x86-64.so.2 (0x00000031ffe00000)

Since this library depends on more than the  CentOS libraries that come with a vanilla install, I should bundle libz.so.1 to guarantee that ~/sire.app will work on any CentOS 6.5 machine (where the user may not be able to yum install missing libraries).

To get pip to work I also need _ssl.cpython-33m.so which I got upon compiling python by installing before openssl (yum install openssl-devel) 

[siredeveloper@localhost lib-dynload]$ ldd _ssl.cpython-33m.so 

linux-vdso.so.1 =>  (0x00007fff8b368000)

libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007fd55f5fd000)

libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007fd55f21c000)

libpython3.3m.so.1.0 => not found

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd55efff000)

libc.so.6 => /lib64/libc.so.6 (0x00007fd55ec6a000)

libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fd55ea26000)

libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fd55e740000)

libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fd55e53b000)

libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fd55e30f000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007fd55e10b000)

libz.so.1 => /lib64/libz.so.1 (0x00007fd55def4000)

/lib64/ld-linux-x86-64.so.2 (0x00000031ffe00000)

libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fd55dce9000)

libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fd55dae6000)

libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fd55d8cb000)

libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fd55d6ac000)

Tedious work....

Presumably there will be no way to have a portable binary, the final package will be very OS specific. 

Ahh....Isn't there a better way to compile a portable python binary ?

Julien

 

[Christopher Woods]

Hi Julien,

> But if I get it right, there is no need to specify individual additional
> libraries in the install files. All executables in sire.app/bin/ ''know'' to
> check ../lib and ../bundled/lib/

Yes, I believe that this is true for Linux. However, for OS X it is
different as the executable has a list of libraries against which it
is linked, e.g. executable foo is linked against libBar.dylib.
libBar.dylib is, however, the internal name of the library, e.g. this
could be the internal name of the library in file libNotBar.dylib! foo
will search for libBar.dylib in the DYLD_LIBRARY_PATH. However, to get
RPATH to work, you need to instead tell foo that it is instead linked
to @rpath/../lib/libBar.dylib, and you then have to change the
internal name of libBar.dylib to @rpath/../lib/libBar.dylib. @rpath is
expanded at the point when the executable starts, so at that time,
install_path/bin/foo will know it is linked to
install_path/bin/../lib/libBar.dylib, and will *only* look in this
file for a library that is called install_path/bin/../lib/libBar.dylib
(hence why we had to change the internal library name to
@rpath/../lib/libBar.dylib. foo will not look in this directory for
any other library files. So it will not "know" to check ../lib and
../bundled/lib. This means you have to set these things manually for
*every* *single* (annoying!) library that could be used by foo (and
anything that this then links against, loads dynamically, etc. etc. -
so yes, bundling python stuff on OS X can be quite annoying!) ;-)

> So for instance, to compile setuptools for python I add to add zlib-devel to
> CentOS with yum install zlib-devel
>
> Know I have the required library in
> /home/siredeveloper/sire.app/bundled/lib/python3.3/lib-dynload
>
> [siredeveloper@localhost lib-dynload]$ ldd zlib.cpython-33m.so
> linux-vdso.so.1 => (0x00007fff087ff000)
> libz.so.1 => /lib64/libz.so.1 (0x00007f518e102000)
> libpython3.3m.so.1.0 => not found
> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f518dee4000)
> libc.so.6 => /lib64/libc.so.6 (0x00007f518db50000)
> /lib64/ld-linux-x86-64.so.2 (0x00000031ffe00000)
>
> Since this library depends on more than the CentOS libraries that come with
> a vanilla install, I should bundle libz.so.1 to guarantee that ~/sire.app
> will work on any CentOS 6.5 machine (where the user may not be able to yum
> install missing libraries).

Yes, you could do this and it will work, It will work because the
libraries that libz depends on are all standard Linux system
libraries, so you don't have to bundle them and you can be sure that
they exist on most (all?) Linux's. The libraries are;

linux-vdso.so.1 - this is the linux library that is used to load
dynamic shared objects (dso's) - i.e. dynamic libraries. This is part
of the linux system and should not be bundled. You can always assume
that this will exist. If ever this changes, or linux-vdso.so.2 comes
out, then we will have to start making different linux binaries...

libpthread.so.0 - this is the system threading library. This is part
of the linux distribution and should not be bundled. This defines how
an application splits into multiple threads and is now standard on all
distributions. Again, if this changes or increases version, then we
will have to create a different linux binary.

libc.so.6 - this is the standard C library, which is part of the
system and cannot be bundled. The C library increases its version
number regularly, but is backwards compatible. This means that
binaries / libraries built against older versions of libc will link to
and use newer versions of libc. However, binaries / libraries built
against newer versions of libc may not work when used with older
versions. This is why I compile Sire on an old (SL 5) linux
distribution so that the resulting binary will work on all Linux
distributions that have a libc that is this old or newer.

ld_linux-x64.-64.so.2 - this is another standard library that is used
for launching the program and assembling the executable from the
shared libraries. This is part of the linux distribution and should
not be bundled. This defines the architecture of the binary (e.g.
x86-64).

Indeed, looking at the problem again, I see that the issue is that I
assume that libz.so is installed and available on all Linux
distributions. This is because I view zlib as a standard system
library (similar to libm.so, which is the standard C maths library,
which I also assume is installed on all Linux distributions and so
does not need to be bundedl). Perhaps you are right and we should look
to bundle our own copy of zlib...

> To get pip to work I also need _ssl.cpython-33m.so which I got upon
> compiling python by installing before openssl (yum install openssl-devel)
>
> [siredeveloper@localhost lib-dynload]$ ldd _ssl.cpython-33m.so
> linux-vdso.so.1 => (0x00007fff8b368000)
> libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007fd55f5fd000)
> libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007fd55f21c000)
> libpython3.3m.so.1.0 => not found
> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd55efff000)
> libc.so.6 => /lib64/libc.so.6 (0x00007fd55ec6a000)
> libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fd55ea26000)
> libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fd55e740000)
> libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fd55e53b000)
> libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fd55e30f000)
> libdl.so.2 => /lib64/libdl.so.2 (0x00007fd55e10b000)
> libz.so.1 => /lib64/libz.so.1 (0x00007fd55def4000)
> /lib64/ld-linux-x86-64.so.2 (0x00000031ffe00000)
> libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fd55dce9000)
> libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fd55dae6000)
> libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fd55d8cb000)
> libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fd55d6ac000)
>
> Tedious work....

Yes, very tedious. Unlike zlib, pulling in OpenSSL pulls in lots of
other Linux libraries, including those for networking and
authentication. It is very difficult / impossible to bundle these, and
they are very distro dependent. This is why I have tried to avoid
pulling in SSL...

> Presumably there will be no way to have a portable binary, the final package
> will be very OS specific.
> Ahh....Isn't there a better way to compile a portable python binary ?

Yes, if you pull in too many extra dependencies then it becomes almost
impossible to create a single, portable python library. We then have
to get into the business of creating apt-get or yum packages that can
make it work with different distributions, but this is very messy, and
also these packages are only ever used by sysadmins (I want Sire to be
easily installable by a user into their own home directory without the
need to change anything on the system or have root access).

Best wishes,

Christopher

[julien]

Hi Chris, 

So we said our goal is to have a version of sire that has pip working after python has been compiled so it is possible for users to add python modules as needed (and let them sort out additional dependencies they may have to resolve/introduce in the process). 

For some reasons zlib and _ssl were missing in CentOS 6.5. I don't know what OS Sajid was using, but it looks like _ssl was also missing in that case. Without zlib I can't install setuptools and without _ssl I can't install pip. 

In my case I solved this problem by installing system zlib and ssl libraries with the yum packages zlib-devel and openssl-devel. I then recompiled python and this time the libraries zlib and _ssl were added to bundled/lib/python3.3/lib-dynload/  

Only then could I do python ; import _ssl  and install pip.

I restated this as I think it is possible I didn't explain clearly what the problem was. 

I think, if that's possible, we should do as you suggest and force python to use internal zlib and _ssl libraries such that the resulting lib-dynload/*.so modules do not link to anything outside of the bundled python libs. 

As you suggested, this should be done on SL5 so we can test next whether a tarball of sire.app can be unpacked on commonly used Linux distributions. However we should also test that the *.cmake files are all setup such that it is possible to recompile the same sources on a range of different Linux OSes, yielding everytime a binary that is portable (across a reasonable range of modern distribs).

At the moment I am setup to test that the binary/compil procedure will work on:

ubuntu 12.04

centos 6.5

SL5 

OSX (never done it so far but hey that's my native OS :-) )

Are there other distribs that we should consider testing against regularly?

Once we know we have something that can be robustly recompiled on the most widely used OSes, it will be easier to test and document the procedure to package sire into specific apps with their own additional dependencies.

Best wishes

Julien

[Christopher Woods]

Hi Julien,

Thanks - yes I fully understand the problem now (not having zlib and
ssl leading to the original compilation of python not then supporting
pip / setuptools). Sorry for the earlier confusion ;-)

Yes, I agree that I need to find a way of getting python to use
internal zlib and ssl and will try some things out to sort out the
problem. This will then leave us with a sire.app/bin/python that will
support pip and easy_install.

In terms of distributions, I have a set of virtual machines(*) that
has about a dozen different distributions from over the last 5 or so
years. The problem is that everything worked on all of them, as I had
originally installed these distros as graphical development servers,
and so zlib / ssl was installed on all of them by default. I am not
sure how we solve this problem, other than perhaps test on "minimal"
installs of the distributions?

I agree that once this is stable, it will be easier to document the
process of compiling and installing dependent modules.

Thanks for your help with this - it has been useful to re-explore how
I did it and to bounce around a few ideas.

Best wishes,

Christopher

(*) Well, I had a set of virtual machines... I have moved desks and
computer and am currently waiting for my new iMac desktop. I won't
have access to my virtual machines until I have my iMac and can
reinstall everything... I have been promised it should be arriving
very soon...

> --
> You received this message because you are subscribed to the Google Groups
> "Sire Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sire-develope...@googlegroups.com.
> To post to this group, send email to sire-de...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sire-developers.
> For more options, visit https://groups.google.com/d/optout.



--
---------------------------------------------------------
Christopher Woods
+44 (0) 7786 264562
http://chryswoods.com

[julien]

Hi Chris, 

In terms of distributions, I have a set of virtual machines(*) that
has about a dozen different distributions from over the last 5 or so
years. The problem is that everything worked on all of them, as I had
originally installed these distros as graphical development servers,
and so zlib / ssl was installed on all of them by default. I am not
sure how we solve this problem, other than perhaps test on "minimal"
installs of the distributions?

I think that might be the safest way to go. 

 

I agree that once this is stable, it will be easier to document the
process of compiling and installing dependent modules.

I think we have hit another portability issue. I passed on a CentOS VM with sire working to George, but while he can run ~/sire.app/bin/python 

he gets an ''illegal instruction'' error everytime python imports a Sire module (but not other modules such as numpy).

After some discussions with Gaetano, he thinks that this is because I compiled sire with the flag SIRE_DISABLE_AVX OFF on my MBA (Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz) but George's laptop has an AMD Phenom(tm) II N950 Quad-Core Processor.

I have asked George to rebuild sire with SIRE_DISABLE_AVX ON to see if that helps. 

Does this sound like a possible reason for the incompatibility ? 

Are there other flags that you think might need to be turned on or off to produce a binary that will be portable to different common hardware ? I'd rather accept loss of performance to have a functional binary. We are always free to compiled different specialised flavors (or better, give instructions to recompile for a specific machine). 

Best wishes, 

Julien

 

> email to sire-developers+unsubscribe@googlegroups.com.
> To post to this group, send email to sire-developers@googlegroups.com.

[Christopher Woods]

Hi Julien,

Yes, "illegal instruction" errors are caused when AVX code is run on a
processor that doesn't support AVX. Many processors now support AVX,
but not all, which is why I've added the SIRE_DISABLE_AVX option. I
compile the portable SL5 binary with AVX disabled exactly to solve
this issue. There are no other flags that need disabling, unless you
want to disable SSE to allow the code to run on extremely old
hardware... ;-)

Cheers,

Christopher

>> > email to sire-develope...@googlegroups.com.
>> > To post to this group, send email to sire-de...@googlegroups.com.

>> > Visit this group at http://groups.google.com/group/sire-developers.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> ---------------------------------------------------------
>> Christopher Woods
>> +44 (0) 7786 264562
>> http://chryswoods.com
>

> --
> You received this message because you are subscribed to the Google Groups
> "Sire Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to sire-develope...@googlegroups.com.
> To post to this group, send email to sire-de...@googlegroups.com.