Should SSL support be checked in 'prereq' ?
Dr. David Kirkby
hashlib module is a bit irritating. There is is a specific test for this in
spkg-install.
-------------------------
# Make sure sufficient crypto support is available in the built python.
# This is critical.
python -c "import hashlib"
if [ $? -eq 0 -a -f "$SAGE_LOCAL/bin/python" ]; then
echo "hashlib module imported"
exit 0
else
echo "hashlib module failed to import"
exit 1
fi
-----------------------------
Faillure to build the haslib module results from a failure to find the SSL
libraries. Would it not be more sensible to check for these in prereq, then exit
if they are not found. It would save someone getting half a build.
As a second point, would it seem sensible to add a slightly more informative
error message to python's spkg-install, to indicate the probably cause of the
problem (lack of OpenSSL library) and how one might remedy that?
If there is agreement on these, I'll create two trac tickets, and address them
individually.
PS, it would be really useful if someone would review my latest update to prereq:
http://trac.sagemath.org/sage_trac/ticket/8052
which was created in response to a ticket William marked as 'critical'
http://trac.sagemath.org/sage_trac/ticket/8026
though was a duplicate of one I'd create a couple of tickets before him.
http://trac.sagemath.org/sage_trac/ticket/8024
As such, I closed William's ticket as 'duplicate' but I have solved the problem.
Dave
William Stein
If we require SSL, then perhaps we should:
(1) get rid of GNUtls, libcdk, libgcrypt, libgpgerror, and anything
other spkg related to that stack.
(2) should include openssl in Sage.
The reason is that the openssl license is GPL-incompatible. If Sage
requires it to run, then we have to think about the implications of
that.
Technically, I think none of the GPL-'d parts of the Sage binary link
with any of the crypto/ssl libraries. I think the only thing in all
of Sage that binary links directly with ssl is Python itself. It's
a possibly subtle license question whether we can legally ship openSSL
or not. Personally, I believe it is technically legal. However, we
would certainly have to change the statement in the README.txt about
all components of SAGE being GPLv2+ compatible.
Note that a few years ago, Sage did ship opensll and not ship the
gnutls stack. Then a student in my Sage class pointed out that
openssl is licensed in a GPL-incompatible way, and that's when we
switched to GNUtls. Overall, unfortunately it seems that the GNUtls
library and stack is unfortunately not a drop-in replacement for
openssl, and seems to be generally less popular and less well
supported. That things would turn out this way was much less clear
3-4 years ago.
-- William
Dr. David Kirkby
Which is of course a pretty important part of Sage!
Would the answer be to not require that Sage build python with SSL support?
Obviously it would disable cryptographic support in Sage. Would that make Sage
unusable - I somewhat doubt it?
I can't think of any application I personally have for Sage that would require
At the minute python builds, but spkg-install exists due to a failure to have
hashlib. That is a totally unacceptable situation in my honest opinion.
> It's
> a possibly subtle license question whether we can legally ship openSSL
> or not. Personally, I believe it is technically legal. However, we
> would certainly have to change the statement in the README.txt about
> all components of SAGE being GPLv2+ compatible.
I was not actually suggesting shipping OpenSSL, as I knew there were license
implications.
But I think you would have to agree it is pretty annoying for someone to
download Sage, start a build, then the build fail due to lack of OpenSSL.
I do not believe this issue is unique to Solaris either. I very much doubt
Cygwin, or many small linux distos come with OpenSSL, but it would need to be
selected, then downloaded from a server. So one could not argue the operating
system comes with it.
Personally, I do not believe it is legal to ship OpenSSL and for Sage to remain
GPL, unless you could get the python developers agree to add an clause that
permits linking against OpenSSL.
http://www.openssl.org/support/faq.html#LEGAL2
says. "If you are using GPL software developed by others, you may want to ask
the copyright holder for permission to use their software with OpenSSL."
That might well be worth doing. I can't see it would harm python in any way,
especially given Python will often be linked against OpenSSL anyway.
> Note that a few years ago, Sage did ship opensll and not ship the
> gnutls stack. Then a student in my Sage class pointed out that
> openssl is licensed in a GPL-incompatible way, and that's when we
> switched to GNUtls.
I agree with him/her.
> Overall, unfortunately it seems that the GNUtls
> library and stack is unfortunately not a drop-in replacement for
> openssl, and seems to be generally less popular and less well
> supported. That things would turn out this way was much less clear
> 3-4 years ago.
>
> -- William
I can appreciate that.
I feel there needs to be something done to stop the unfortunate situation of
someone starting to build Sage, the build failing some hours later, due to a
reason we could predict within 1 minute of them starting the build.
Dave
Dr. David Kirkby
> I was not actually suggesting shipping OpenSSL, as I knew there were
> license implications.
>
> But I think you would have to agree it is pretty annoying for someone to
> download Sage, start a build, then the build fail due to lack of OpenSSL.
>
> I do not believe this issue is unique to Solaris either. I very much
> doubt Cygwin, or many small linux distos come with OpenSSL, but it would
> need to be selected, then downloaded from a server. So one could not
> argue the operating system comes with it.
I just installed cygwin. Sure enough, there is no SSL support by default. You
have to make a positive effort to select it. As such, I do not believe OpenSSL
is a library that is part of the normal Cygwin distribution. Just selecting all
the defaults will give you no OpenSSL libraries.
It seems to me, there are 3 choices.
1) Get the Python developers to agree to allow python to link against OpenSSL.
Then you could
* Ship OpenSSL
* Get rid of a load of stuff.
That seems the best solution to me, IF they would agree. It's not one you can
expect to do today though.
2) Stop supporting Sage on any platform which does not come with SSL as part of
the normal operating system distribution - that would include both Solaris and
Cygwin. That seems the dumbest idea.
3) Change Sage so that the hashlib module of python is not essential for a
functioning Sage. That is I suspect the easiest option. I don't claim to
understand how Sage builds fully, but I would have thought crypto support was
not a requirement.
Dave
Dr. David Kirkby
> 3) Change Sage so that the hashlib module of python is not essential for
> a functioning Sage. That is I suspect the easiest option. I don't claim
> to understand how Sage builds fully, but I would have thought crypto
> support was not a requirement.
What I mean is, that a useful maths program can be made without it requiring
cryptographic support. It would be more useful with it, but not essential to
have a useful GPL program.
Dave
Robert Bradshaw
The only place I know it's used is to serve up secure notebooks, but I
bet its used elsewhere too. I see another option
4) Write our own hashlib using gnutls that gets installed if the SSL
one isn't made. Whether or not this is sufficient depends on how much
we use from it.
- Robert
Dr. David Kirkby
IF that is all, then that hardly seems a major loss of functionality. I bet most
people don't use the secure notebooks anyway. I can see they have advantages
though, especially for commercial users. I admit, that is something I would like
myself, but I personally would just install OpenSSL.
> 4) Write our own hashlib using gnutls that gets installed if the SSL one
> isn't made. Whether or not this is sufficient depends on how much we use
> from it.
>
> - Robert
Sounds logical to me, though I think I remember William saying gnutls was
something like 100x slower than OpenSSL for something. I can't recall what
however - it might have been a dream!
Dave
Martin Albrecht
> > bet its used elsewhere too. I see another option
>
> IF that is all, then that hardly seems a major loss of functionality. I bet
> most people don't use the secure notebooks anyway. I can see they have
> advantages though, especially for commercial users. I admit, that is
> something I would like myself, but I personally would just install
> OpenSSL.
I do use the encrypted notebook and I would consider it a *major*
functionality loss if the option to encrypt my calculations which I send over
then net could not be encrypted anymore in a convenient manner. We should be
pushing to make the notebook more secure instead of giving up on security
completely. Of course, talk on my end is cheap and I haven't done anything
recently to make the notebook more secure.
Cheers,
Martin
--
name: Martin Albrecht
_pgp: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x8EF0DC99
_otr: 47F43D1A 5D68C36F 468BAEBA 640E8856 D7951CCF
_www: http://www.informatik.uni-bremen.de/~malb
_jab: martinr...@jabber.ccc.de
Dr. David Kirkby
>>> The only place I know it's used is to serve up secure notebooks, but I
>>> bet its used elsewhere too. I see another option
>> IF that is all, then that hardly seems a major loss of functionality. I bet
>> most people don't use the secure notebooks anyway. I can see they have
>> advantages though, especially for commercial users. I admit, that is
>> something I would like myself, but I personally would just install
>> OpenSSL.
>
> I do use the encrypted notebook and I would consider it a *major*
> functionality loss if the option to encrypt my calculations which I send over
> then net could not be encrypted anymore in a convenient manner. We should be
> pushing to make the notebook more secure instead of giving up on security
> completely. Of course, talk on my end is cheap and I haven't done anything
> recently to make the notebook more secure.
>
> Cheers,
> Martin
>
I do agree the secure notebooks is useful. As someone who does contract work,
the ability to share data with a customer via a secure mechanism would be
important.
I suspect it would be possible to tunnel the traffic over SSH though, so having
security, even whilst the browser is using http, not https. You would know more
about that than me. But that adds considerably to the complexity for the end user.
But there does seem to be an issue with the license. In any case, if someone
installed OpenSSL, the security would be there - which is what I'd personally
just do.
But I can see no justification for letting someone try to build Sage, for it to
fail a couple of hours later, in a way we could have predicted in less than a
minute.
I think the 'prerequ' configure script should check for OpenSSL and exit if it
is not present, since we can be 100% sure Sage will not build without it.
Exactly what error message is produced would need to be discussed. But it's
crazy for 'prereq' to not check for it. Just as it would be crazy to not check
for perl, when we know it is needed.
I've recently updated 'prereq' to try to ensure that people are made aware of
build issues before they start.
* gcc too old.
* perl too old.
* gcc/g++/gfortran not all the same version
* not a mix of GNU and non-GNU compilers
etc etc.
To me, checking for OpenSSL support should be added to that list, since it is a
requirement to build Sage.
Dave
William Stein
<david....@onetel.net> wrote:
> William Stein wrote:
> Personally, I do not believe it is legal to ship OpenSSL and for Sage to
> remain GPL, unless you could get the python developers agree to add an
> clause that permits linking against OpenSSL.
>
> http://www.openssl.org/support/faq.html#LEGAL2
>
> says. "If you are using GPL software developed by others, you may want to
> ask the copyright holder for permission to use their software with OpenSSL."
Python is not GPL software.
>> Note that a few years ago, Sage did ship opensll and not ship the
>> gnutls stack. Then a student in my Sage class pointed out that
>> openssl is licensed in a GPL-incompatible way, and that's when we
>> switched to GNUtls.
>
> I agree with him/her.
Do you mean that you agree with the statement: "The Openssl license is
not GPL-compatible?"
--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org
Dan Drake
> 3) Change Sage so that the hashlib module of python is not essential
> for a functioning Sage. That is I suspect the easiest option. I
> don't claim to understand how Sage builds fully, but I would have
> thought crypto support was not a requirement.
People in this thread have been discussing this issue only in terms of
encryption, but hashlib also does...well, hashing, such as MD5 and SHA1.
We make a big deal out of Sage being Python-based, and Python in turn
says that the "batteries are included" -- these batteries typically
include being able to easily take hashes of strings, so I would strongly
prefer that the hashlib module stay the way it is.
(Also, SageTeX needs md5 hashing support in its current state, and I'd
be unhappy if, just after getting it merged, some of its functionality
needs to be changed!)
Dan
--
--- Dan Drake
----- http://mathsci.kaist.ac.kr/~drake
-------
Dr. David Kirkby
Yes.
The FSF wrote the GPL, and the FSF say the OpenSSL license is not compatible
with the GPL.
http://www.fsf.org/licensing/licenses/index_html#GPLIncompatibleLicenses
"The license of OpenSSL is a conjunction of two licenses, one of them being the
license of SSLeay. You must follow both. The combination results in a copyleft
free software license that is incompatible with the GNU GPL."
I think this is likely to be a non-issue with regard to Solaris 10, as I
discovered today that OpenSSL does ship as part of the Solaris operating system
- it hides in /usr/sfw/lib. As such, we can use it, without breaching the GPL.
(The only problem might be if the version shipped with Solaris is too old to
work with Python).
For OpenSolaris, it is an issue, there is no OpenSSL library included.
I personally think the best approach will be to approach the developers of Open
Solaris, Cygwin, and anything else which does not ship with OpenSSL to include
it. Pointing out that it makes difficult to use GPL software, might well get
them to include it by default.
I will try to peruse this avenue with Open Solaris. Since you have a contact at
Sun, who wants to see Sage on Solaris, I suggest you raise the issue with him too.
Dave
Robert Bradshaw
> But I can see no justification for letting someone try to build
> Sage, for it to fail a couple of hours later, in a way we could have
> predicted in less than a minute.
>
> I think the 'prerequ' configure script should check for OpenSSL and
> exit if it is not present, since we can be 100% sure Sage will not
> build without it. Exactly what error message is produced would need
> to be discussed. But it's crazy for 'prereq' to not check for it.
> Just as it would be crazy to not check for perl, when we know it is
> needed.
>
> I've recently updated 'prereq' to try to ensure that people are made
> aware of build issues before they start.
>
> * gcc too old.
> * perl too old.
> * gcc/g++/gfortran not all the same version
> * not a mix of GNU and non-GNU compilers
>
> etc etc.
>
> To me, checking for OpenSSL support should be added to that list,
> since it is a requirement to build Sage.
Whatever the long term solution for OpenSSL/crypto support, I
certainly agree that, as things stand now, adding it to prereq is
better than a build that fails half way through.
- Robert
Dr. David Kirkby
Thank you Robert.
I checked the python source code, and found it searches two directories for the
libraries, in addition to the normal search path. It has this bit of code in the
top-level setup.py.
----------------------------------------------
ssl_libs = find_library_file(self.compiler, 'ssl',lib_dirs,
['/usr/local/ssl/lib',
'/usr/contrib/ssl/lib/'
] )
---------------------------------------------
I have submitted an RFE to the Python bug list
http://bugs.python.org/issue7836
and suggest they add /usr/sfw/lib to that list, as that is where the libraries
are installed on Solaris.
What I propose we do in Sage is:
1) Modify the setup.py of python so it also searches for libssl.so in
/usr/sfw/lib, which will find the libraries on Solaris 10. That is ticket
http://trac.sagemath.org/sage_trac/ticket/8152
but I'm not going to bother doing that, until there is agreement over this.
2) Search for libssl.so in the standard directories, which the linker looks in
(/usr/lib etc)
3) Search for libssl.so in the two directories searched for by python now
(/usr/local/ssl/lib and /usr/contrib/ssl/lib/) AND in /usr/sfw/lib.
If OpenSSL is not found in any of those locations, then we should exit with a
message:
----------------------------------------------------------
Unable to find the OpenSSL libraries, which are usually part as part of the
operating system. However, the location of the OpenSSL installation does vary
with between different platforms. Please set LD_LIBRARY_PATH to the location of
the directory containing your OpenSSL libraries (libssl.so)
----------------------------------------------------------
(I would propose using that exact wording. Feel free to comment on it, change
it, but let's agree on the exact wording of the error message, since it is
important).
With the above wording, we are *not* explicitly telling someone to
install OpenSSL, just to set LD_LIBRARY_PATH to where OpenSSL is installed.
I just checked my HP-UX system (flames to /dev/null) and OpenSSL is installed
somewhere else on that. So it's not unreasonable to say that people should set
LD_LIBRARY_PATH to the location of their OpenSSL libraries. It varies from
system to system.
Can we have any agreement on this?
Dave
Robert Bradshaw
This may have been true at one point, but looking at the current
Python sources [0,1] it seems that Python provides native
implementations of hashlib if OpenSSL is not found, so import hashlib
should never fail (#7761 was an artifact that *no* distutils
extensions built...). What might be more of an issue is _ssl not
building [2], which may be easier to fix (especially if we don't use
any parts of _ssl that gnutls doesn't support). I'm not sure why this
ticket was closed without resolving that issue. OTOH, I'm doubtful we
use _ssl anywhere except for the notebook, as it is all about secure
sockets. Gnutls claims to have support for twisted.
Of course I'm just jumping into the issue now, so there is probably a
lot of history, and pragmatic background of how stuff actually works,
that I'm missing. But can anyone verify that we actually even require
OpenSSL (which is the running assumption)?
- Robert
[0] http://svn.python.org/view/python/branches/release26-maint/setup.py?revision=77865&view=markup
# search for _hashlib
[1] http://svn.python.org/view/python/branches/release26-maint/Lib/hashlib.py?view=markup
[2] http://trac.sagemath.org/sage_trac/ticket/6359#comment:13
Dr. David Kirkby
> On Sat, Jan 30, 2010 at 10:16 PM, Dr. David Kirkby
> <david....@onetel.net> wrote:
>> Faillure to build the haslib module results from a failure to find the
>> SSL
>> libraries. Would it not be more sensible to check for these in prereq,
>> then
>> exit if they are not found. It would save someone getting half a build.
>
> This may have been true at one point, but looking at the current Python
> sources [0,1] it seems that Python provides native implementations of
> hashlib if OpenSSL is not found, so import hashlib should never fail
> (#7761 was an artifact that *no* distutils extensions built...).
Looking at the source code, I can understand why you come to this conclusion. I
would agree with you.
But the fact is, that it does not build. Someone obviously added the test in
Sage, which I admit could be outdated.
-------------------------
# Make sure sufficient crypto support is available in the built python.
# This is critical.
python -c "import hashlib"
if [ $? -eq 0 -a -f "$SAGE_LOCAL/bin/python" ]; then
echo "hashlib module imported"
exit 0
else
echo "hashlib module failed to import"
exit 1
fi
-----------------------------
When I had this problem on OpenSolaris, William told me to install OpenSSL.
Yesterday I rebuilt python, having changed the permissions of the /usr/local/ssl
directory to 000. Sure enough, it fails.
I can't fault your logic based on looking at the source code, but in practice I
can assure you that the failure to have OpenSSL libraries results in a failure
for the hashlib module to build, so results in a failure of the Sage build.
I'm happy to report that as a bug to the Python developers, but it does not
solve the more pressing issue in Sage.
A google finds other failing to get haslib installed on linux too.
> What
> might be more of an issue is _ssl not building [2], which may be easier
> to fix (especially if we don't use any parts of _ssl that gnutls doesn't
> support).
In which case, perhaps we should test for _ssl too.
> I'm not sure why this ticket was closed without resolving that
> issue. OTOH, I'm doubtful we use _ssl anywhere except for the notebook,
> as it is all about secure sockets. Gnutls claims to have support for
> twisted.
>
> Of course I'm just jumping into the issue now, so there is probably a
> lot of history, and pragmatic background of how stuff actually works,
> that I'm missing. But can anyone verify that we actually even require
> OpenSSL (which is the running assumption)?
>
> - Robert
I believe that is true we currently need OpenSSL - at least on Solaris, and I
don't see why it should be any different on any other platform. If you have a
Linux machine where you have root access and can ensure that there are no SSL
libraries to be found, I suspect you will get the same issue.
There is this python bug about hashlib. I'm not suggesting this is the cause of
failure, but it might indicate the hashlib stuff in python is poorly implemented.
http://bugs.python.org/issue6281
The ticket starts by saying "The try statement at the end of hashlib.py is some
of the worst python code I've had the mispleasure of reading for a long time."
Dave