OpenSSL as a new systemwide Sage dependency ?

[Emmanuel Charpentier]

Dear list,

The fact that we can't ship openSSL (see uncountable theads in sage-devel and others) seems to pose more and more difficulties. See for example this thread on sage-support, and especially Dima's answer, as well as this annoying ticket, discussed in this saga .

Could'nt we add OpenSSL as a prerequisite to Sage, and it"s development files as a prerequisite to building Sage ? This would require of the user to install OpenSSL systemwide, thus making it "system software" and satisfying the strange licensing requirements that bother us.

One could even do that indirectly, by requiring a systemwide libcurl supporting https : this would de facto enforce the systemwide installation of OpenSSL (or a reasonable facsimile). That's what I was trying to do in this proposal... (IIRC, the problem with libcurl is also bound to OpenSSL : libcurl itself is not a problem. But I'll have to check : if this is true, we can require OpenSSL and ship libcurl which will then compile cleanly).

Comments ? Especially wrt Macs, which seem to be further encumbered by Apple's dirty tricks...

Should we have a vote ?

--
Emmanuel Charpentier

[Erik Bray]

I'm inclined to agree. Add to that all (or at least most) of Sage's
other dependencies as well ;)

[Thierry]

Hi,

On Mon, Nov 21, 2016 at 03:21:31AM -0800, Emmanuel Charpentier wrote:
> Dear list,
>
> The fact that we can't ship openSSL (see uncountable theads in sage-devel
> and others) seems to pose more and more difficulties. See for example this

> thread <https://groups.google.com/forum/#!topic/sage-support/rDV9uGT2ViM>

> on sage-support, and especially Dima's answer

> <https://groups.google.com/d/msg/sage-support/rDV9uGT2ViM/GuKDbhSKAwAJ>, as
> well as this annoying ticket <https://trac.sagemath.org/ticket/21767>,
> discussed in this saga
> <https://groups.google.com/forum/#!topic/sage-devel/QaBdHSNJuKg> .


Note that Dima's answer is somehow misleading, since downloading openssl
from the Sage mirrors does not require SSL.

Hence the following is still possible, without having openssl-dev as a
system prerequisite:

- check that openssl-dev (or equivalent) is installed system-wide
- if not:
- warn the user and suggest/recommend her to install it
- as an alternative, propose to download and install openssl from the
Sage mirrors via http
- build Sage

Ciao,
Thierry

> Could'nt we add OpenSSL as a prerequisite to Sage, and it"s development
> files as a prerequisite to building Sage ? This would require of the user
> to install OpenSSL systemwide, thus making it "system software" and
> satisfying the strange licensing requirements that bother us.
>
> One could even do that indirectly, by requiring a systemwide libcurl
> supporting https : this would de facto enforce the systemwide installation
> of OpenSSL (or a reasonable facsimile). That's what I was trying to do in this

> proposal <https://trac.sagemath.org/ticket/21767#comment:41>... (IIRC, the

> problem with libcurl is also bound to OpenSSL : libcurl itself is not a
> problem. But I'll have to check : if this is true, we can require OpenSSL
> and ship libcurl which will then compile cleanly).
>
> Comments ? Especially wrt Macs, which seem to be further encumbered by
> Apple's dirty tricks...
>
> Should we have a vote ?
>
> --
> Emmanuel Charpentier
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[Jeroen Demeyer]

Should we really add OpenSSL as a dependency to fix a few very specific
issues?

I'm not saying that I'm against the proposal, but we should really weigh
the pros against the cons. The main "con" is: adding an extra dependency
which might make it even harder for ordinary users to compile Sage from
source.

99% of Sage users will probably never need OpenSSL within Sage. If you
break something for 2% of users while fixing something for 1% of users,
that is a net loss of 1%.

Jeroen.

[Dima Pasechnik]

On Monday, November 21, 2016 at 11:21:31 AM UTC, Emmanuel Charpentier wrote:

Dear list,

The fact that we can't ship openSSL (see uncountable theads in sage-devel and others) seems to pose more and more difficulties. See for example this thread on sage-support, and especially Dima's answer, as well as this annoying ticket, discussed in this saga .

Could'nt we add OpenSSL as a prerequisite to Sage, and it"s development files as a prerequisite to building Sage ? This would require of the user to install OpenSSL systemwide, thus making it "system software" and satisfying the strange licensing requirements that bother us.

Try installing OpenSSL on an OSX 10.12 Mac using just XCode!

You might be in for a surprise.

[Emmanuel Charpentier]

Le 21 nov. 2016 18:26, "Dima Pasechnik" <dim...@gmail.com> a écrit :
>
>
>
> On Monday, November 21, 2016 at 11:21:31 AM UTC, Emmanuel Charpentier wrote:
>>
>> Dear list,
>>
>> The fact that we can't ship openSSL (see uncountable theads in sage-devel and others) seems to pose more and more difficulties. See for example this thread on sage-support, and especially Dima's answer, as well as this annoying ticket, discussed in this saga .
>>
>> Could'nt we add OpenSSL as a prerequisite to Sage, and it"s development files as a prerequisite to building Sage ? This would require of the user to install OpenSSL systemwide, thus making it "system software" and satisfying the strange licensing requirements that bother us.
>
>
> Try installing OpenSSL on an OSX 10.12 Mac using just XCode!
> You might be in for a surprise.

Oh.

How is that done currently  ?

I was aware os the existence of some Apple's shenanigans, but not to this extend...

>> One could even do that indirectly, by requiring a systemwide libcurl supporting https : this would de facto enforce the systemwide installation of OpenSSL (or a reasonable facsimile). That's what I was trying to do in this proposal... (IIRC, the problem with libcurl is also bound to OpenSSL : libcurl itself is not a problem. But I'll have to check : if this is true, we can require OpenSSL and ship libcurl which will then compile cleanly).
>>
>> Comments ? Especially wrt Macs, which seem to be further encumbered by Apple's dirty tricks...
>>
>> Should we have a vote ?
>>
>> --
>> Emmanuel Charpentier
>>

> --
> You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/92OdoUbBDbE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.

[Dima Pasechnik]

On Monday, November 21, 2016 at 4:09:41 PM UTC, Thierry (sage-googlesucks@xxx) wrote:

Hi,

On Mon, Nov 21, 2016 at 03:21:31AM -0800, Emmanuel Charpentier wrote:
> Dear list,
>
> The fact that we can't ship openSSL (see uncountable theads in sage-devel
> and others) seems to pose more and more difficulties. See for example this
> thread <https://groups.google.com/forum/#!topic/sage-support/rDV9uGT2ViM>
> on sage-support, and especially Dima's answer
> <https://groups.google.com/d/msg/sage-support/rDV9uGT2ViM/GuKDbhSKAwAJ>, as
> well as this annoying ticket <https://trac.sagemath.org/ticket/21767>,
> discussed in this saga
> <https://groups.google.com/forum/#!topic/sage-devel/QaBdHSNJuKg> .


Note that Dima's answer is somehow misleading, since downloading openssl
from the Sage mirrors does not require SSL.  

I was referring to the fact that https://github.com/sagemath/binary-pkg

does not work on OSX 10.12, due to this SSL blues.

Unless I misunderstand, we currently aren't able to build distributable Sage binaries on OSX 10.12.

[Emmanuel Charpentier]

Le lundi 21 novembre 2016 à 09:34 -0800, Dima Pasechnik a écrit :

On Monday, November 21, 2016 at 4:09:41 PM UTC, Thierry (sage-googlesucks@xxx) wrote:

Hi,

On Mon, Nov 21, 2016 at 03:21:31AM -0800, Emmanuel Charpentier wrote:
> Dear list,
>
> The fact that we can't ship openSSL (see uncountable theads in sage-devel
> and others) seems to pose more and more difficulties. See for example this
> thread <https://groups.google.com/forum/#!topic/sage-support/rDV9uGT2ViM>
> on sage-support, and especially Dima's answer
> <https://groups.google.com/d/msg/sage-support/rDV9uGT2ViM/GuKDbhSKAwAJ>, as
> well as this annoying ticket <https://trac.sagemath.org/ticket/21767>,
> discussed in this saga
> <https://groups.google.com/forum/#!topic/sage-devel/QaBdHSNJuKg> .


Note that Dima's answer is somehow misleading, since downloading openssl
from the Sage mirrors does not require SSL.  

I was referring to the fact that https://github.com/sagemath/binary-pkg

does not work on OSX 10.12, due to this SSL blues.

Unless I misunderstand, we currently aren't able to build distributable Sage binaries on OSX 10.12.

OK. That's different. If I read you correctly, you mean that a binary packaging of Sage won't run on a Mac that hasn't somehow received OpenSSL's baptism ? Hence two questions :

1) Can OpenSSL be installed on a "virgin" Mac ? If so, how ?

2) Can a Max where OpenSSL has been installed run a Sage binary package ?

If the answer to those questions are both "yes", this describes a situation very close to what I suggest to endorse by depending officially on OpenSSL.

--

Emmanuel Charpentier

 

You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/92OdoUbBDbE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.

[Dima Pasechnik]

On Monday, November 21, 2016 at 5:50:29 PM UTC, Emmanuel Charpentier wrote:

Le lundi 21 novembre 2016 à 09:34 -0800, Dima Pasechnik a écrit :

On Monday, November 21, 2016 at 4:09:41 PM UTC, Thierry (sage-googlesucks@xxx) wrote:

Hi,

On Mon, Nov 21, 2016 at 03:21:31AM -0800, Emmanuel Charpentier wrote:
> Dear list,
>
> The fact that we can't ship openSSL (see uncountable theads in sage-devel
> and others) seems to pose more and more difficulties. See for example this
> thread <https://groups.google.com/forum/#!topic/sage-support/rDV9uGT2ViM>
> on sage-support, and especially Dima's answer
> <https://groups.google.com/d/msg/sage-support/rDV9uGT2ViM/GuKDbhSKAwAJ>, as
> well as this annoying ticket <https://trac.sagemath.org/ticket/21767>,
> discussed in this saga
> <https://groups.google.com/forum/#!topic/sage-devel/QaBdHSNJuKg> .


Note that Dima's answer is somehow misleading, since downloading openssl
from the Sage mirrors does not require SSL.  

I was referring to the fact that https://github.com/sagemath/binary-pkg

does not work on OSX 10.12, due to this SSL blues.

Unless I misunderstand, we currently aren't able to build distributable Sage binaries on OSX 10.12.

OK. That's different. If I read you correctly, you mean that a binary packaging of Sage won't run on a Mac

no, I said that one *cannot build* such a distro on OSX 10.12. It still works on OSX 10.11.

[Samuel Lelievre]

Emmanuel Charpentier:

> Dima Pasechnik:

>

> > I was referring to the fact that

> > https://github.com/sagemath/binary-pkg

> > does not work on OSX 10.12, due to this SSL blues.

> >

> > Unless I misunderstand, we currently aren't able

> > to build distributable Sage binaries on OSX 10.12.

>

> OK. That's different. If I read you correctly, you mean that

> a binary packaging of Sage won't run on a Mac that hasn't

> somehow received OpenSSL's baptism ?

To clarify, "binary-pkg" is Volker Braun's replacement

for the former "sage -bdist", ie the engine to produce

a binary distribution from an existing source install.

In other words, say you have built Sage from source on

architecture X, and you want to produce binaries for all

your friends that are also using architecture X, then

that's what you run.

[Volker Braun]

Actually OSX is foobar'ed even then, Apple's ancient openssl just doesn't support TLSv1.2. Some sites are already requiring that:

osx:~ vbraun$ openssl s_client -connect www.kernel.org:443

CONNECTED(00000003)

write:errno=54

[Emmanuel Charpentier]

What my dentist's very little mind understands so far :

1) OpenSSL *can* be installed on a Max OSX 10.12

2) Sage *can* be built successfully on OSX 10.12

3) sage-pkg *cannot* create a redistributable binary

So far, tha'ts but half a problem (we still don't have binary packages for cygwin either...). More serious it Volker's observation (other mail in this thread) that 

4) OpenSSL on Mac OSX currently currently doesn't support TLS V1.2

if I understand correctly what he means.

What are the known ways to get a correctly-functionning libcurl on Mac OSX 10.12 ?

--

Emmanuel Charpentier

--

[Emmanuel Charpentier]

Le lundi 21 novembre 2016 à 10:17 -0800, Volker Braun a écrit :

Actually OSX is foobar'ed even then, Apple's ancient openssl just doesn't support TLSv1.2. Some sites are already requiring that:

osx:~ vbraun$ openssl s_client -connect www.kernel.org:443

CONNECTED(00000003)

write:errno=54

Mmm According to https://www.openssl.org/, OpenSSL latest version is 1.1.0c (dated Noc 10, 2016), which, according to http://mac.softpedia.com/get/Security/OpenSSL.shtml, compiles on a Mac.

Does this compilation present special difficulties ?

--

Emmanuel Charpentier

Dear list,

The fact that we can't ship openSSL (see uncountable theads in sage-devel and others) seems to pose more and more difficulties. See for example this thread on sage-support, and especially Dima's answer, as well as this annoying ticket, discussed in this saga .

Could'nt we add OpenSSL as a prerequisite to Sage, and it"s development files as a prerequisite to building Sage ? This would require of the user to install OpenSSL systemwide, thus making it "system software" and satisfying the strange licensing requirements that bother us.

One could even do that indirectly, by requiring a systemwide libcurl supporting https : this would de facto enforce the systemwide installation of OpenSSL (or a reasonable facsimile). That's what I was trying to do in this proposal... (IIRC, the problem with libcurl is also bound to OpenSSL : libcurl itself is not a problem. But I'll have to check : if this is true, we can require OpenSSL and ship libcurl which will then compile cleanly).

Comments ? Especially wrt Macs, which seem to be further encumbered by Apple's dirty tricks...

Should we have a vote ?

--
Emmanuel Charpentier

[Dima Pasechnik]

On Monday, November 21, 2016 at 7:10:49 PM UTC, Emmanuel Charpentier wrote:

Le lundi 21 novembre 2016 à 10:17 -0800, Volker Braun a écrit :

Actually OSX is foobar'ed even then, Apple's ancient openssl just doesn't support TLSv1.2. Some sites are already requiring that:

osx:~ vbraun$ openssl s_client -connect www.kernel.org:443

CONNECTED(00000003)

write:errno=54

Mmm According to https://www.openssl.org/, OpenSSL latest version is 1.1.0c (dated Noc 10, 2016), which, according to http://mac.softpedia.com/get/Security/OpenSSL.shtml, compiles on a Mac.

Does this compilation present special difficulties ?

in fact, it seems to work; last time I tried it at sage -sh prompt, with Sage's gcc, and it didn't do what I needed.

I'm trying the whole binary-pkg thing now, with openssl built by Xcode and installed in /usr/local

(not sure whether the build has used extra things I have installed, like extra Perl packages---

yes, OpenSSL 1.1.* has its own, Perl-based, configure system...)

Anyhow, I'm stuck at the same place - that git cannot be built, see

https://github.com/sagemath/binary-pkg/issues/8

[Dima Pasechnik]

On Monday, November 21, 2016 at 5:26:12 PM UTC, Dima Pasechnik wrote:

On Monday, November 21, 2016 at 11:21:31 AM UTC, Emmanuel Charpentier wrote:

Dear list,

The fact that we can't ship openSSL (see uncountable theads in sage-devel and others) seems to pose more and more difficulties. See for example this thread on sage-support, and especially Dima's answer, as well as this annoying ticket, discussed in this saga .

Could'nt we add OpenSSL as a prerequisite to Sage, and it"s development files as a prerequisite to building Sage ? This would require of the user to install OpenSSL systemwide, thus making it "system software" and satisfying the strange licensing requirements that bother us.

Try installing OpenSSL on an OSX 10.12 Mac using just XCode!

You might be in for a surprise.

I take it back - it works; you might still have to keep telling your building system where to find headers and libraries, 

like ./configure --with-openssl=/usr/local (etc)

This is not quite "systemwide", if you ask me.

That is, I suppose we would need to have a --with-openssl= parameter in the toplevel Sage configure taking care of this.

Dima

[Emmanuel Charpentier]

OK. Let's try again :
I have two questions :

1. What are the parts (standard, optional or experimental, except, of course, the openssl package itself) of Sage that need (directly or indirectly) a secure transport layer but would accept either openSSL or reasonable substitutes such as Gnu TLS or Mozilla's NSS ?
2. What are the parts (standard, optional or experimental, except, of course, the openssl package itself) of Sage that (directly or indirectly) need openSSL, no substitute accepted ?

My favorite itch to be scratched (namely R), seems to fall in the first category, but I have trouble proving it : I would need a reasonable test machine with no openSSL library to check whether R installs or not using only Gnu TLS.  All the Linux *desktop* installation I tried install openSSL, one way or another (even Debian, which is notably prudish about licensing). I would have to install an ultra-basic virtual machine. This setup could be used to prove or disprove the dependencies of various parts of Sage.

There are only two possible results, and two sets of action :

1. If no part of Sage depends on openSSL exclusively : fine. package and ship Gnu TLS as a standard package, and be done with the damn thing
2. If some part of Sage need openSSL exclusively : since we *can* use a systemwise installation but cannot (pseudo-legally) *ship* it, we just *have to* depend on this systemwide installation. Add it to the prerequisites, and be done with it.

So this inventory is crucial.

What do you know about these dependencies ?

--
Emmanuel Charpentier

[Emmanuel Charpentier]

Dear Thierry,

Le lundi 21 novembre 2016 17:09:41 UTC+1, Thierry (sage-googlesucks@xxx) a écrit :

Hi,

On Mon, Nov 21, 2016 at 03:21:31AM -0800, Emmanuel Charpentier wrote:
> Dear list,
>
> The fact that we can't ship openSSL (see uncountable theads in sage-devel
> and others) seems to pose more and more difficulties. See for example this
> thread <https://groups.google.com/forum/#!topic/sage-support/rDV9uGT2ViM>
> on sage-support, and especially Dima's answer
> <https://groups.google.com/d/msg/sage-support/rDV9uGT2ViM/GuKDbhSKAwAJ>, as
> well as this annoying ticket <https://trac.sagemath.org/ticket/21767>,
> discussed in this saga
> <https://groups.google.com/forum/#!topic/sage-devel/QaBdHSNJuKg> .


Note that Dima's answer is somehow misleading, since downloading openssl
from the Sage mirrors does not require SSL.

Hence the following is still possible, without having openssl-dev as a
system prerequisite:

- check that openssl-dev (or equivalent) is installed system-wide
- if not:
  - warn the user and suggest/recommend her to install it
  - as an alternative, propose to download and install openssl from the
    Sage mirrors via http

I am not sure we can (pseudo-)legally do that.

Advices ?
 

[Thierry]

Hi,

On Sun, Nov 27, 2016 at 03:56:28AM -0800, Emmanuel Charpentier wrote:
> Dear Thierry,
>
> Le lundi 21 novembre 2016 17:09:41 UTC+1, Thierry (sage-googlesucks@xxx) a
> écrit :

[...]

> > - check that openssl-dev (or equivalent) is installed system-wide
> > - if not:
> > - warn the user and suggest/recommend her to install it
> > - as an alternative, propose to download and install openssl from the
> > Sage mirrors via http
> >
>
> I am not sure we can (pseudo-)legally do that.

I am not a lawyer either, but i guess that if we can not, then we are
already ((pseudo-)legally) wrong, since it is currently possible to
download and install openssl from the Sage mirrors via http, see
${SAGE_ROOT}/build/pkgs/openssl/

The only thing that we currently forbid to ourselves is to put openssl
tarball in the Sage source tarball, as a standard package.

Ciao,
Thierry

[Thierry]

Hi,

On Sun, Nov 27, 2016 at 03:52:59AM -0800, Emmanuel Charpentier wrote:
> OK. Let's try again :
> I have two questions :
>

> 1. What are the parts (standard, optional or experimental, except, of

> course, the openssl package itself) of Sage that need (directly or
> indirectly) a secure transport layer but would accept either openSSL or
> reasonable substitutes such as Gnu TLS or Mozilla's NSS ?

> 2. What are the parts (standard, optional or experimental, except, of

> course, the openssl package itself) of Sage that (directly or indirectly)
> need openSSL, no substitute accepted ?
>
> My favorite itch to be scratched (namely R), seems to fall in the first
> category, but I have trouble proving it : I would need a reasonable test
> machine with no openSSL library to check whether R installs or not using
> only Gnu TLS. All the Linux *desktop* installation I tried install
> openSSL, one way or another (even Debian, which is notably prudish about
> licensing). I would have to install an ultra-basic virtual machine. This
> setup could be used to prove or disprove the dependencies of various parts
> of Sage.

A priori (?), openssl package should not interfere if you do not have
libssl-dev installed.

I tried building Sage 7.3 on a VM without libssl-dev, but with both
libgnutls28-dev and libgnutls-openssl27 installed (on a Debian jessie).
Sage builds and tests fine, but i do not have SSL support when using pip:

./sage -pip search blah
SSLError: Can't connect to HTTPS URL because the SSL module is not
available.

Ciao,
Thierry

> There are only two possible results, and two sets of action :
>

> 1. If no part of Sage depends on openSSL exclusively : fine. package and

> ship Gnu TLS as a standard package, and be done with the damn thing

> 2. If some part of Sage need openSSL exclusively : since we *can* use a

> systemwise installation but cannot (pseudo-legally) *ship* it, we just
> *have to* depend on this systemwide installation. Add it to the
> prerequisites, and be done with it.
>
>
> So this inventory is crucial.
>
> What do you know about these dependencies ?
>
> --
> Emmanuel Charpentier
>
> Le lundi 21 novembre 2016 12:21:31 UTC+1, Emmanuel Charpentier a écrit :
> >
> > Dear list,
> >
> > The fact that we can't ship openSSL (see uncountable theads in sage-devel
> > and others) seems to pose more and more difficulties. See for example this

> > thread <https://groups.google.com/forum/#!topic/sage-support/rDV9uGT2ViM>

> > on sage-support, and especially Dima's answer

> > <https://groups.google.com/d/msg/sage-support/rDV9uGT2ViM/GuKDbhSKAwAJ>,
> > as well as this annoying ticket <https://trac.sagemath.org/ticket/21767>,
> > discussed in this saga

> > <https://groups.google.com/forum/#!topic/sage-devel/QaBdHSNJuKg> .

> >
> > Could'nt we add OpenSSL as a prerequisite to Sage, and it"s development
> > files as a prerequisite to building Sage ? This would require of the user
> > to install OpenSSL systemwide, thus making it "system software" and
> > satisfying the strange licensing requirements that bother us.
> >
> > One could even do that indirectly, by requiring a systemwide libcurl
> > supporting https : this would de facto enforce the systemwide installation
> > of OpenSSL (or a reasonable facsimile). That's what I was trying to do in this

> > proposal <https://trac.sagemath.org/ticket/21767#comment:41>... (IIRC,

> > the problem with libcurl is also bound to OpenSSL : libcurl itself is not a
> > problem. But I'll have to check : if this is true, we can require OpenSSL
> > and ship libcurl which will then compile cleanly).
> >
> > Comments ? Especially wrt Macs, which seem to be further encumbered by
> > Apple's dirty tricks...
> >
> > Should we have a vote ?
> >
> > --
> > Emmanuel Charpentier
> >
> >
>

[Emmanuel Charpentier]

Dear Thierry,

Thank you for this answer : you have done a large work that I planned for my next opportunity. Comments below :

This seems a serious problem, given the increasing dependency of Sage on pip packages. This is also one aspect of more and more services migrating to secured protocols (the new https requirement of R, which started this epopea, is another example).

I think that your experiment demonstrates that GnuTLS does not (yet) offer a substitute to (at least some) OpenSSL functionalities.

My experiments with R led me to think the same thing.

So I think it's time to bite the bullet, acknowledge that we depend on OpenSSL and document it. We should also test for it when building Sage.

For the first task, I think that the right places are :

• README.md in the root directory ;
• The developer's guide.

For the second task, the most logical place would be in the toplevel configure file, after checking for a "minimal" toolset (C compiler, make, etc..). Since at this point, we do not have pkg-config, we have to use the Great White Shark method : bite and see what happens. In this case,, compile, link and run a minimal program festing the existence and basic functionality of of libssl. Ideas on what to test an how are much welcome : I'm a bit out of my depth here...

We could also wait for the installation of Sage's pkg-config, which makes the ssl test trivial, but, IIRC that comes a bit late (i. e. after Sage's python compilation), and we leave the user with partially unusable Sage's parts, that whould have to be recompiled after a correct OpenSSL installation. It's probably better to fail early.

Again, your advice, criticisms and ideas are welcome.


 

Ciao,
Thierry

Again, thank you !

--
Emmanuel Charpentier
 

[Volker Braun]

On Linux, you can build Sage with and without openssl. If you ever hit the network you really should build with openssl(-devel) available, it will be picked up automatically. But its not a requirement. Though we should probably strongly recommend it in the installation instructions.

GnuTLS and other implementations won't solve our problem, as Python's _ssl module is specifically written against OpenSSL and can't be linked against anything else.

On OSX, you can do either

a) nothing => no https support,

a) supply the (missing) openssl headers for the system openssl. This is still a shitty solution as it doesn't (and probably will never) support TLSv12.

c) compile your own openssl implementation AND bring your own copy of the root certificates as your self-compiled openssl will not be able to access the OSX certificate store. Distributing the resulting binary has some license issues.

[Francois Bissey]

I notice anaconda ship its own openssl, on OS X at least.
Are they purposefully avoid GPL in all their packages?

François

[Jean-Pierre Flori]

On Thursday, December 1, 2016 at 11:47:40 PM UTC+1, Volker Braun wrote:

On Linux, you can build Sage with and without openssl. If you ever hit the network you really should build with openssl(-devel) available, it will be picked up automatically. But its not a requirement. Though we should probably strongly recommend it in the installation instructions.

Yes.

GnuTLS and other implementations won't solve our problem, as Python's _ssl module is specifically written against OpenSSL and can't be linked against anything else.

On OSX, you can do either

a) nothing => no https support,

Note a very simple patch allows to build R without ranting about ssl... Sure this is bad for the mondane user with internet access and no license issues, but it simplifies our task.

[Dima Pasechnik]

On Thursday, December 1, 2016 at 10:47:40 PM UTC, Volker Braun wrote:

On Linux, you can build Sage with and without openssl. If you ever hit the network you really should build with openssl(-devel) available, it will be picked up automatically. But its not a requirement. Though we should probably strongly recommend it in the installation instructions.

GnuTLS and other implementations won't solve our problem, as Python's _ssl module is specifically written against OpenSSL and can't be linked against anything else.

On OSX, you can do either

a) nothing => no https support,

a) supply the (missing) openssl headers for the system openssl. This is still a shitty solution as it doesn't (and probably will never) support TLSv12.

c) compile your own openssl implementation AND bring your own copy of the root certificates as your self-compiled openssl will not be able to access the OSX certificate store. Distributing the resulting binary has some license issues.

Do you understand the story about root certs here? Is it a missing python code (in some package, existing or not?) that would be able to access OSX certs store? Or is it more fundamental than this?

[Emmanuel Charpentier]

Le jeudi 1 décembre 2016 23:47:40 UTC+1, Volker Braun a écrit :

On Linux, you can build Sage with and without openssl. If you ever hit the network

Who doesn't ? I can see two (quite marginal) use cases :

• Military high-security un-networked machine users (who install and update their software via sneakernet) ;
• Monks (probably not even a plurality of them).

As far as I can tell, everybody else will have someday to hit the network (at least for PIP packages...).
 

you really should build with openssl(-devel) available, it will be picked up automatically. But its not a requirement. Though we should probably strongly recommend it in the installation instructions.

That's more or less what I propose. You seem to be less affirmative than I would.

GnuTLS and other implementations won't solve our problem, as Python's _ssl module is specifically written against OpenSSL and can't be linked against anything else.

On OSX, you can do either

a) nothing => no https support,

Unacceptable, for practicability reasons (see above)

a) supply the (missing) openssl headers for the system openssl. This is still a shitty solution as it doesn't (and probably will never) support TLSv12.

Hardly acceptable (how longer the PIP and R repositories will remain reachable without TLSv12 ?) 

c) compile your own openssl implementation AND bring your own copy of the root certificates as your self-compiled openssl will not be able to access the OSX certificate store. Distributing the resulting binary has some license issues.

Dubious : where can one get "its own copy of the root certs" ? How secure *this* would be ? (And I have dark doubts on the possible consequences of the licensing issues...).

The logical conclusion would be to avoid Macs for this kind of use. (But I have to confess an old and deeply ingrained prejudice about these nice but way overpriced machines and Apple's shenanigans that forever try to chain you to ONE supplier, namely themselves...).

Should we discourage Sage potential users to buy Macs ?

Note that we *could* also (at least theoretically) try to replace Python's __ssl module with our own, more accommodating one, linked to our own TLSv12-enabled library. Somehow, I don't see *that* happen in a foreseeable future :-).

HTH (but I doubt it...)

--
Emmanuel Charpentier

[Volker Braun]

On Friday, December 2, 2016 at 9:39:13 AM UTC+1, Dima Pasechnik wrote:

Do you understand the story about root certs here? Is it a missing python code (in some package, existing or not?) that would be able to access OSX certs store? 

Apple has the root certs in their own keychain, which OpenSSL can't read (i.e. Apple did not upstream their patches to OpenSSL). You can manually extract the root certs or download an independent copy of them. Either way, a self-compiled OpenSSL will not benefit from OS updates to the root cert store.

[Jeroen Demeyer]

On 2016-12-02 12:50, Emmanuel Charpentier wrote:
> Le jeudi 1 décembre 2016 23:47:40 UTC+1, Volker Braun a écrit :
>
> On Linux, you can build Sage with and without openssl. If you ever
> hit the network
>
>
> Who doesn't ?

The 95% of Sage users who just uses Sage for computations and doesn't
want to install any packages.

That's why I'm against depending on system OpenSSL: you are adding an
extra hurdle for everybody for the benefit of a minority of users.

[Thierry]

Hi,

On Fri, Dec 02, 2016 at 03:50:07AM -0800, Emmanuel Charpentier wrote:
[...]

> As far as I can tell, everybody else will have someday to hit the network
> (at least for PIP packages...).

It is not always possible. At least many users of Sage Debian Live from
places where there is not (much) network should not be dependent of
internet connection, not only in remote areas (which was the use case for
the creation of SDL): it has also been required during various workshops,
where this was necessary e.g. when the wifi does not allow guests to be
connected.

We should still be able to propose a self-contained software (while at the
same time let the possibility to use distro's deps). Regarding source
tarball, it might (?) be a good idea to ship an extended tarball with all
optional and pip packages (e.g. by letting pip to look into the upstream/
directory before going to PyPI).

BTW, it is pretty important that Sage only connects to the internet when
explicitely asked by the user (e.g. when using oeis). Currently, pip is
leaking when used, just to check latest version, it is pretty annoying.
The use of a remote three.js CDN is also problematic with that respect.

Ciao,
Thierry

[Emmanuel Charpentier]

Le samedi 03 décembre 2016 à 10:07 +0100, Jeroen Demeyer a écrit :
> On 2016-12-02 12:50, Emmanuel Charpentier wrote:
> > Le jeudi 1 décembre 2016 23:47:40 UTC+1, Volker Braun a écrit :
> >
> >     On Linux, you can build Sage with and without openssl. If you
> > ever
> >     hit the network
> >
> >
> > Who doesn't ?
>
> The 95% of Sage users who just uses Sage for computations and
> doesn't 
> want to install any packages.

Hmmm... Where do you pull your "95%" figure from ?

> That's why I'm against depending on system OpenSSL: you are adding
> an 
> extra hurdle for everybody

It's, at worst, an extra dependency. Of which we already have some lot
(see README.md...).

> for the benefit of a minority of users.

Again : why "minority" ?

And why do you postulate that Sage and its components have a vocation
to stay self-enclosed and forever stable ? The activity of the sage-
devel list shows that the ability to update, upgrade and extend Sage
is important.

That is true, with severe aggravation, for R : while essential, the
"core" functionality of R is small (if not tiny) when compared to the
9000+ packages available. an unextensible R would be as useful as a
teaspoon to empty the Mediterranean...

On the top of that, Sage itself uses the net extensively. It uses tens
of Python packages, which need maintainance and possibly extension.

I'm afraid that your "average user" who "just uses Sage for
computations and doesn't want to install any packages" is about as
mythical as Ronald Reagan's "welfare queen" : a nice strawman useful to
hide a serious problem.

(In fact, two serious problems : Volker's explanation of Apple's root
certs problem makes me think that this is, for Mac users, even more
serious : they are chained to "the Apple" store" for the life of their
Macs. More of this as an answer to Volker's post).

It remains to be seen if our "openssl" package allows a Sage system,
initially compiled with, say, GnuTLS, to use, say PIP. That could be an
hypocritical but efficient workaround.

HTH,

--
Emmanuel Charpentier

[Emmanuel Charpentier]

Dear Thierry,

If you still have this VM, could you try to add the current "openssl"
Sage package and tell us if , after this addition, "sage -pip list"
works ?

Thank you in advance !

--
Emmanuel Charpentier

[Emmanuel Charpentier]

This is an extremely serious problem, which I didn't grasp initially. (To me, it's probably a conta-indication of Macs to anything a bit serious : somehow, I have less trust in Apple's administration of the root certs than, say, Debian's. Prejudiced ? Certainly : I've been burned before...).

Do you know if openSSL could be retro-patched to be able to use the systemwide installation of Apple's root certs (which, by hypothesis, would be updated as needed) as a default ? I think that this question has both technical and (pseudo-)legal aspects.

--

Emmanuel Charpentier

[Thierry]

On Sat, Dec 03, 2016 at 10:43:07AM +0100, Emmanuel Charpentier wrote:
[...]

> It remains to be seen if our "openssl" package allows a Sage system,
> initially compiled with, say, GnuTLS, to use, say PIP. That could be an
> hypocritical but efficient workaround.

No, you will have to recompile (at least) python2.

Ciao,
Thierry


> HTH,

[Thierry]

Hi,

On Sat, Dec 03, 2016 at 10:48:18AM +0100, Emmanuel Charpentier wrote:
[...]

> If you still have this VM, could you try to add the current "openssl"
> Sage package and tell us if , after this addition, "sage -pip list"
> works ?

I droped it, but i can create a new one. Do you mean just add `./sage -i
openssl` after compiling ? Otherwise, give me the exact order of the
operations you want to test (starting from initial deps and make command).

I am pretty sure openssl will not be taken into account until you
recompile python2 (at least it was the case a few months ago), python
seems not trying to guess system's libs at runtime.

Ciao,
Thierry

[Emmanuel Charpentier]

Le samedi 03 décembre 2016 à 13:02 +0100, Thierry a écrit :
> Hi,
>
> On Sat, Dec 03, 2016 at 10:48:18AM +0100, Emmanuel Charpentier wrote:
> [...] 
> > If you still have this VM, could you try to add the current
> > "openssl"
> > Sage package and tell us if , after this addition, "sage -pip list"
> > works ?
>
> I droped it, but i can create a new one. Do you mean just add `./sage
> -i
> openssl` after compiling ?

That's what I thought. But you gave me (below) another idea. See below

> Otherwise, give me the exact order of the
> operations you want to test (starting from initial deps and make
> command).
>
> I am pretty sure openssl will not be taken into account until you
> recompile python2 (at least it was the case a few months ago), python
> seems not trying to guess system's libs at runtime.

Ok. So we might try the following sequence :

1) in a pristine, openssl-virgin VM, compile Sage, and test pip's
ability to work (test 0, negative, as you proved).

2) install Sage's openssl, test pip's ability (test 1)

If test1 positive, we win.

If test 1 is negative, as you think

3a) update Python (./sage -f python2), and re-test pip's ability (test
2)

if test 2 positive, we win (serendipitously).

if test2 negative, you just proved that the openssl package is not
sufficient. Which answers the "dependency on OpenSSL" question.

It would be useful to keep this VM around, because the same tests will
be interesting to run in order to test R's ability to use https-only R
repositories.


HTH,

[Thierry]

On Sat, Dec 03, 2016 at 02:50:19PM +0100, Emmanuel Charpentier wrote:
[...]

> Ok. So we might try the following sequence :
>
> 1) in a pristine, openssl-virgin VM, compile Sage, and test pip's
> ability to work (test 0, negative, as you proved).
>
> 2) install Sage's openssl, test pip's ability (test 1)
>
> If test1 positive, we win.
>
> If test 1 is negative, as you think
>
> 3a) update Python (./sage -f python2), and re-test pip's ability (test
> 2)
>
> if test 2 positive, we win (serendipitously).

I will try this, but it is supposed to work already (this is why an
optional openssl Sage package exists and is maintained).

I think i do not understand why it let us win, since we can not distribute
openssl tarball within the Sage source code anyway.

Ciao,
Thierry

[Emmanuel Charpentier]

The key words are "within the Sage source"... ;-)

More on this later (i'a bit overwhelmed right now).

--
Emmanuel Charpentier

Ciao,
Thierry

--

You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/92OdoUbBDbE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to sage-devel+unsubscribe@googlegroups.com.

[Thierry]

On Sat, Dec 03, 2016 at 05:55:19PM +0100, Emmanuel Charpentier wrote:
> The key words are "within the Sage source"... ;-)
>
> More on this later (i'a bit overwhelmed right now).

Note that the following is already possible, though it seems irrelevant
for macs, which are the real problem (since most (if not all) GNU/Linux
distros ship some openssl-dev package):

On Mon, Nov 21, 2016 at 05:09:36PM +0100, Thierry wrote:
[...]
>
> - check that openssl-dev (or equivalent) is installed system-wide
> - if not:
> - warn the user and suggest/recommend her to install it
> - as an alternative, propose to download and install openssl from the
> Sage mirrors via http