Python's _ssl module wont't compile correctly against OpenSSL 1.1

226 views
Skip to first unread message

Emmanuel Charpentier

unread,
Dec 17, 2016, 3:18:50 AM12/17/16
to sage-devel
Still pursuing our SSL curse (see this thread among others ; and, BTW, Trac#22058 needs review), I noted that I was unable to get a functional pip on any "new" installation (e. g. a virtual machine with Debian testing).

The build logs (see included file) showed that the _ssl extension doesn't compile cleanly, with symptoms similar to those seen in git.

If I understand it correctly, any attempt to install Sage on a machine with OpenSSL>=1.1 (i. e. all major distributions, AFAICT, except the antique Debian "stable"), is doomed to have serious problems communicating over SSL/HTTPS.

However, existing installations can use OpenSSL runtime library >=1.1, which is still binary-compatible. What has changed is the way the macros defined in the development libraries are used to declare SSL-related types.

A bit of Googling led me to this Python bug, which seems relevant (and offer a solution). Questions :
  • Is this ticket-worthy ? (IMHO, it's a damn *blocker* bug...)
  • Should we :
    • just port the proposed patch, or
    • upgrade python to , e. g., 2.7.11-2 (current in Deboan testing, ours is 2.7.10.3) ?

Advice ? Votes ?

HTH,

NB : that again re-asks the question of depending on systemwide OpenSSL (for running : this is acknowledged by recent versions of git, for example) and on OpenSSL development libraries (for building). More on this when I'll have time to completely explore the problem on a "new" installation.

--
Emmanuel Charpentier

python2-2.7.10p3.log-with-openssl

Erik Bray

unread,
Dec 19, 2016, 5:28:28 AM12/19/16
to sage-devel
On Sat, Dec 17, 2016 at 9:18 AM, Emmanuel Charpentier
<emanuel.c...@gmail.com> wrote:
> Still pursuing our SSL curse (see this thread among others ; and, BTW,
> Trac#22058 needs review), I noted that I was unable to get a functional pip
> on any "new" installation (e. g. a virtual machine with Debian testing).
>
> The build logs (see included file) showed that the _ssl extension doesn't
> compile cleanly, with symptoms similar to those seen in git.
>
> If I understand it correctly, any attempt to install Sage on a machine with
> OpenSSL>=1.1 (i. e. all major distributions, AFAICT, except the antique
> Debian "stable"), is doomed to have serious problems communicating over
> SSL/HTTPS.
>
> However, existing installations can use OpenSSL runtime library >=1.1, which
> is still binary-compatible. What has changed is the way the macros defined
> in the development libraries are used to declare SSL-related types.
>
> A bit of Googling led me to this Python bug, which seems relevant (and offer
> a solution). Questions :
>
> Is this ticket-worthy ? (IMHO, it's a damn *blocker* bug...)
> Should we :
>
> just port the proposed patch, or
> upgrade python to , e. g., 2.7.11-2 (current in Deboan testing, ours is
> 2.7.10.3) ?
>
>
> Advice ? Votes ?

Looks to me like the OpenSSL 1.1 fix for Python was merged and
backported to the 2.7.x branch as well, being obviously
security-critical, though a new 2.7.x release hasn't come out yet.
I'd be surprised if that patch hasn't already been backported by the
major distros. But in any case I agree we should pull that patch into
sage as well. I agree there should be a ticket.

Emmanuel Charpentier

unread,
Dec 22, 2016, 5:13:01 AM12/22/16
to sage-devel
Notwithstanding the deafening silence of this list, and following Eric Bray's advice, this is now Trac#22089.

This ticket is marked "critical" (I feel personally that this should be a "blocker" ticket).

Advice requested about whether :
  • patch our current python (2.7.10), or
  • upgrade our current python to, e. g. 2.7.11 (current in Debian testing)
  • or even to 2.7.12 (current in cygwin) or 2.7.13 (current python.org's release, but only 5 days old).
HTH,

--
Emmanuel Charpentier

Samuel Lelievre

unread,
Dec 22, 2016, 6:36:34 AM12/22/16
to sage-devel
Upgrade to Python 2.7.12 is done at #19735.

Upgrade to Python 2.7.13 is in progress at #22037.

Emmanuel Charpentier

unread,
Dec 22, 2016, 8:50:03 AM12/22/16
to sage-devel
Well, I upgraded my local Sage (which used to have a functional pip...) and ... :

charpent@SAP5057241:/usr/local/sage-7$ sage -python
Python 2.7.12 (default, Dec 22 2016, 11:47:35)
[GCC 6.2.1 20161124] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
charpent@SAP5057241:/usr/local/sage-7$ sage -pip search lalala
Exception:
Traceback (most recent call last):
  File "/usr/local/sage-7/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/sage-7/local/lib/python2.7/site-packages/pip/commands/search.py", line 43, in run
    pypi_hits = self.search(query, options)
  File "/usr/local/sage-7/local/lib/python2.7/site-packages/pip/commands/search.py", line 60, in search
    hits = pypi.search({'name': query, 'summary': query}, 'or')
  File "/usr/local/sage-7/local/lib/python/xmlrpclib.py", line 1243, in __call__
    return self.__send(self.__name, args)
  File "/usr/local/sage-7/local/lib/python/xmlrpclib.py", line 1602, in __request
    verbose=self.__verbose
  File "/usr/local/sage-7/local/lib/python2.7/site-packages/pip/download.py", line 764, in request
    headers=headers, stream=True)
  File "/usr/local/sage-7/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 518, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/sage-7/local/lib/python2.7/site-packages/pip/download.py", line 378, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/sage-7/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/sage-7/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/sage-7/local/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 46, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/local/sage-7/local/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 477, in send
    raise SSLError(e, request=request)
SSLError: Can't connect to HTTPS URL because the SSL module is not available.


The problem still stands (has worsened, actually...). However, the last changes in the patch included in Python were dated 2016-09-05, whereas Python 2.7.12 (implicitly -0) is dated 2016-06-25. Our current version may be too hold to get the patch.

HTH,

--
Emmanuel Charpentier

Dima Pasechnik

unread,
Dec 24, 2016, 7:51:28 AM12/24/16
to sage-devel


On Thursday, December 22, 2016 at 1:50:03 PM UTC, Emmanuel Charpentier wrote:
Well, I upgraded my local Sage (which used to have a functional pip...) and ... :

charpent@SAP5057241:/usr/local/sage-7$ sage -python
Python 2.7.12 (default, Dec 22 2016, 11:47:35)
[GCC 6.2.1 20161124] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
charpent@SAP5057241:/usr/local/sage-7$ sage -pip search lalala

and what is the output of 
sage -pip -V

(it could be that you upgraded your pip to v9, and this is the root of the problem here?)

Emmanuel Charpentier

unread,
Dec 24, 2016, 11:00:37 AM12/24/16
to sage-devel


Le samedi 24 décembre 2016 13:51:28 UTC+1, Dima Pasechnik a écrit :


On Thursday, December 22, 2016 at 1:50:03 PM UTC, Emmanuel Charpentier wrote:
Well, I upgraded my local Sage (which used to have a functional pip...) and ... :

charpent@SAP5057241:/usr/local/sage-7$ sage -python
Python 2.7.12 (default, Dec 22 2016, 11:47:35)
[GCC 6.2.1 20161124] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
charpent@SAP5057241:/usr/local/sage-7$ sage -pip search lalala

and what is the output of 
sage -pip -V

I most cautiously didn't. Don't have that machine on hand at the time. On my test VM with the same problem, I get :

 pip 8.1.2 from /usr/local/sage-7/local/lib/python2.7/site-packages (python 2.7)

(no dump of the SSL-related error when pip searching (identocal to what I posted) : this test VM has been kept to an absolute minimum of systemwide software, hence no X, no utilities, no nothing. (especially no cut'n-paste from the console...).

BTW, those tests showed me that R 3.3.2 (my original itch to scratch) *can* be compiled against gnutls and reach HTTPS R repositories.

I plan to recompile against openssl to try to port the Python patch. More on this later...

HTH,

--
Emmanuel Charpentier

Emmanuel Charpentier

unread,
Jan 4, 2017, 7:25:48 AM1/4/17
to sage-devel


Le samedi 24 décembre 2016 17:00:37 UTC+1, Emmanuel Charpentier a écrit :
[ Snip...]

I plan to recompile against openssl to try to port the Python patch. More on this later...

Well, Trac#22089 was an abject failure ; I think that our best bet is Trac#22037, which Jeroen Demeyer is working on.

HTH,

--
Emmanuel Charpentier

Reply all
Reply to author
Forward
0 new messages