[Python-Dev] Remove embedded expat library?

[Victor Stinner]

Hi,

Python embeds a copy of the expat library which already got two major
security vulnerabilities:

"CVE-2016-0718: expat bug #537"
http://python-security.readthedocs.io/vuln/cve-2016-0718_expat_bug_537.html

"Issue #26556: Expat 2.1.1"
http://python-security.readthedocs.io/vuln/issue_26556_expat_2.1.1.html

Would it be possible to maintain this dependency on an external
repository which would be easier to maintain? Like
http://svn.python.org/projects/external/ used to build Python on
Windows.

I expect that all Linux distributions build Python using
--with-system-expat. It may become the default? What about macOS and
other operating systems?

By the way, Zachary Ware is working on converting this repository to
Git. I don't know his progress:
- https://github.com/python/cpython-bin-deps
- https://github.com/python/cpython-source-deps

Victor
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Ned Deily]

On Jun 9, 2017, at 08:43, Victor Stinner <victor....@gmail.com> wrote:
> Python embeds a copy of the expat library which already got two major
> security vulnerabilities:
>
> "CVE-2016-0718: expat bug #537"
> http://python-security.readthedocs.io/vuln/cve-2016-0718_expat_bug_537.html
>
> "Issue #26556: Expat 2.1.1"
> http://python-security.readthedocs.io/vuln/issue_26556_expat_2.1.1.html
>
> Would it be possible to maintain this dependency on an external
> repository which would be easier to maintain? Like
> http://svn.python.org/projects/external/ used to build Python on
> Windows.

Considering how infrequently there have been releases of upstream expat over the past decade or so, even if we had caught every release, I don't see that the benefit to maintaining our embedded copy as a separate repo rather than have to apply the upstreams copies a few time (to each Python branch) is worth the disruptive effort in changing the current Python build process (assuming we don't change the default, see below). Now, as to whether we should continue to provide our own copy of expat, that's a different question. Are there differences between our bundled expat and upstream?

> I expect that all Linux distributions build Python using
> --with-system-expat. It may become the default? What about macOS and
> other operating systems?

The current default is --with-system-expat=no so, unless builders of Python take explicit action, the bundled version of expat is used. Using the bundled version is also currently the case for the python.org macOS installer, no idea what other distributors do. Apple supplies a version of expat with macOS so we presumably we could use the system version for the installer. Presumably (Zach?) we would need to continue to supply a version of expat for Windows builds. But do we need to for others? If it were only Windows, *then* perhaps it might make sense to make all the changes to move expat out of cpython into the common repo for third-party Windows libs.

> By the way, Zachary Ware is working on converting this repository to
> Git. I don't know his progress:
> - https://github.com/python/cpython-bin-deps
> - https://github.com/python/cpython-source-deps

It appears that the project itself is moving from Sourceforge to Github:

https://github.com/libexpat/libexpat

--
Ned Deily
n...@python.org -- []

[Zachary Ware]

On Fri, Jun 9, 2017 at 11:03 AM, Ned Deily <n...@python.org> wrote:
> On Jun 9, 2017, at 08:43, Victor Stinner <victor....@gmail.com> wrote:
>> I expect that all Linux distributions build Python using
>> --with-system-expat. It may become the default? What about macOS and
>> other operating systems?
>
> The current default is --with-system-expat=no so, unless builders of Python take explicit action, the bundled version of expat is used. Using the bundled version is also currently the case for the python.org macOS installer, no idea what other distributors do. Apple supplies a version of expat with macOS so we presumably we could use the system version for the installer. Presumably (Zach?) we would need to continue to supply a version of expat for Windows builds. But do we need to for others? If it were only Windows, *then* perhaps it might make sense to make all the changes to move expat out of cpython into the common repo for third-party Windows libs.

Yes, we would need to continue providing a version for Windows. It
would be a relatively small change to move it to the externals
repository.

I would be fine with switching to `--with-system-expat=yes` by default
and building from externals on Windows in 3.7, and removing the
bundled expat in 3.8.

>> By the way, Zachary Ware is working on converting this repository to
>> Git. I don't know his progress:
>> - https://github.com/python/cpython-bin-deps
>> - https://github.com/python/cpython-source-deps

PR 1783 (https://github.com/python/cpython/pull/1783); needs another
review from Steve to make sure I haven't made a complete mess of
things, then it should be ready to go. Anyone else on Windows (Terry
Reedy?) who can test it and provide feedback, please do!

--
Zach

[Terry Reedy]

On 6/9/2017 12:40 PM, Zachary Ware wrote:

> PR 1783 (https://github.com/python/cpython/pull/1783); needs another
> review from Steve to make sure I haven't made a complete mess of
> things, then it should be ready to go. Anyone else on Windows (Terry
> Reedy?) who can test it and provide feedback, please do!

Something like the following condensed log?

f:\dev>git clone https://github.com/python/cpython tem
f:\dev>cd tem
f:\dev\tem>git remote add upstream https://github.com/python/cpython
f:\dev\tem>git pr 1783
f:\dev\tem>pcbuild\build.bat -e
Fetching external libraries...
Fetching bzip2-1.0.6...
Fetching openssl-1.0.2k...
Fetching sqlite-3.14.2.0...
Fetching tcl-core-8.6.6.0...
Fetching tk-8.6.6.0...
Fetching tix-8.4.3.6...
Fetching xz-5.2.2...
Fetching external binaries...
Fetching nasm-2.11.06...
...

# Appeared to finish with usual warnings, but no errors.
f:\dev\tem>python -m test -j14
...
367 tests OK.

38 tests skipped:
test_crypt test_curses test_dbm_gnu test_dbm_ndbm test_devpoll
test_epoll test_fcntl test_fork1 test_gdb test_grp test_ioctl
test_kqueue test_nis test_openpty test_ossaudiodev test_pipes
test_poll test_posix test_pty test_pwd test_readline test_resource
test_smtpnet test_socketserver test_spwd test_syslog
test_threadsignals test_timeout test_tix test_tk test_ttk_guionly
test_urllib2net test_urllibnet test_wait3 test_wait4 test_winsound
test_xmlrpc_net test_zipfile64

Total duration: 2 min 15 sec
Tests result: SUCCESS
-----

Duration aside, my working clone master, without the patch, gives the
same result. (The usual debug build takes twice as long.) In this
respect, the patch seems ready to go. If you have any questions about
the compile or test logs, I will try to keep them around at least a few
hours.




--
Terry Jan Reedy

[Ronald Oussoren]

> On 9 Jun 2017, at 18:03, Ned Deily <n...@python.org> wrote:
>
> On Jun 9, 2017, at 08:43, Victor Stinner <victor....@gmail.com> wrote:
>
>> I expect that all Linux distributions build Python using
>> --with-system-expat. It may become the default? What about macOS and
>> other operating systems?
>
> The current default is --with-system-expat=no so, unless builders of Python take explicit action, the bundled version of expat is used. Using the bundled version is also currently the case for the python.org macOS installer, no idea what other distributors do. Apple supplies a version of expat with macOS so we presumably we could use the system version for the installer. Presumably (Zach?) we would need to continue to supply a version of expat for Windows builds. But do we need to for others? If it were only Windows, *then* perhaps it might make sense to make all the changes to move expat out of cpython into the common repo for third-party Windows libs.

I don’t think it would be a good idea to rely on the system provided libexpat on macOS, as Apple is not exactly fast w.r.t. upgrading their external dependencies and could easily stop updating libraries when the no longer need them (see for example the mess w.r.t. OpenSSL).

Ronald

[Victor Stinner]

Le 11 juin 2017 09:38, "Ronald Oussoren" <ronaldo...@mac.com> a écrit :

I don’t think it would be a good idea to rely on the system provided libexpat on macOS, as Apple is not exactly fast w.r.t. upgrading their external dependencies and could easily stop updating libraries when the no longer need them (see for example the mess w.r.t. OpenSSL).

Ok, but can't we download expat instead of keeping an old copy in our repisitory?

Having a copy is useful when we modify it. I don't that it is the case here.

Victor

[Ronald Oussoren]

On 11 Jun 2017, at 12:10, Victor Stinner <victor....@gmail.com> wrote:

Le 11 juin 2017 09:38, "Ronald Oussoren" <ronaldo...@mac.com> a écrit :

I don’t think it would be a good idea to rely on the system provided libexpat on macOS, as Apple is not exactly fast w.r.t. upgrading their external dependencies and could easily stop updating libraries when the no longer need them (see for example the mess w.r.t. OpenSSL).

Ok, but can't we download expat instead of keeping an old copy in our repisitory?

Sure. The script that creates the installer already downloads a number of libraries, adding one more shouldn’t be a problem. 

Having a copy is useful when we modify it. I don't that it is the case here.

I don’t know why expat was included in the CPython tree and if those reasons are still valid. I therefore have no opinion on keeping it, other than that expat shouldn’t be kept in the CPython tree unless there’s a good reason for doing so. 

BTW. Removing 3th-party libraries from the source tree doesn’t fully isolate us from security issues in those libraries due to shipping the libraries in binary installers on Windows and macOS.  The removal does make maintenance easier (no need to guess whether or not there are local patches).

Ronald

Victor

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com