add curl as a standard package

[Emmanuel Charpentier]

It looks like curl, which was formerly installed by R installation, must now be installed in order to install R.

One can note that a problem will rarely exists, since most Sage users have curl installed in their system. However, a user having only sage installed (e. g. Sage's virtual machine for Windows) will see it...

If we want to keep R a standard package, we have to either

• patch R and add curl's installation (heavy : the R core Team threw the towel on it),
• add curl to Sage's prerequisites (lighter to us, but still heavy to the users),
• or make it a standard package, on which R must depend.

The latter solution has my reference(much simpler).

What is yours ?

--
Emmanuel Charpentier

[Vincent Delecroix]

In between?

* having a curl pacakge that will be installed if not present on the
system (~the same way we deal with gcc)

Vincent

[Michael Orlitzky]

And the same for pcre and xz. Check for it in ./configure, and give us a
way to override it, like --with-xz[=path].

[Emmanuel Charpentier]

Great idea, if I can implement it. Looking in the $PATH is not quite easy to do portably. I'm currently looking at configure.in, and begin to suspect I've fallen in a trap...

Ideas ?

[Emmanuel Charpentier]

Not the same problem. To detect curl, it is enough to find an executable named "curl" in the path. As far as I know, R uses a library, and this is known to be not so easy to find.

[ I'm currently looking at autotools tutorials to understand how to check for this kind of things, and start to wonder in which fire I've jumped from our familiar frying pan... ]

The same is true for pcre.

Bonus question : how can the R package depend of such a package that would install nothing ?

Alternative : in the spkg-install script, test somehow for the existence of the sought binary/library, and do nothing if found. Would such a trick still mark the package as installed ?

[Jean-Pierre Flori]

Maybe it is not that bad if the automagic detection system is not that good.
If the automatic stuff works magically on usual config, let's say large Linux distros, that's enough.
As long as we have "--with-x" options to pass to configure for making Sage more distro friendly and let smart people do whatever they want quite easily.

[Michael Orlitzky]

On 10/24/2016 09:46 AM, Emmanuel Charpentier wrote:
>
> Not the same problem. To detect curl, it is enough to find an executable
> named "curl" in the path. As far as I know, R uses a library, and this
> is known to be not so easy to find.
>

Which library does R link against? If you can find out its name and a
function that R tries to use, you can check for the library's (and
function's) presence with AC_CHECK_LIB. Or if you need specific headers,
there's AC_CHECK_HEADERS.

> The same is true for pcre.

PCRE is the same. They're both 19 years old and everyone else already
builds this way so it's easy to find examples to copy/paste. In fact, R
itself has a working configure script that checks for curl/pcre.

[Emmanuel Charpentier]

Bonus question for the build system experts : can an spkg-install script recursively call $SAGE_ROOT/sage -i <some other package> and get the return status back ? A simple solution would then be to test for the existence of the relevant binaries/executables in the R spkg-install script, and recursively install the (optional) relevant packages before proceeding to install R.

What do you think ?

[Dima Pasechnik]

On Monday, October 24, 2016 at 4:39:27 PM UTC, Emmanuel Charpentier wrote:

Bonus question for the build system experts : can an spkg-install script recursively call $SAGE_ROOT/sage -i <some other package> and get the return status back ? A simple solution would then be to test for the existence of the relevant binaries/executables in the R spkg-install script, and recursively install the (optional) relevant packages before proceeding to install R.

isn't it what package dependencies are for?

[Jean-Pierre Flori]

On Monday, October 24, 2016 at 6:39:27 PM UTC+2, Emmanuel Charpentier wrote:

Bonus question for the build system experts : can an spkg-install script recursively call $SAGE_ROOT/sage -i <some other package> and get the return status back ? A simple solution would then be to test for the existence of the relevant binaries/executables in the R spkg-install script, and recursively install the (optional) relevant packages before proceeding to install R.

What do you think ?

I would say the good solution would be to mimick what is done for GCC.
Some calls to the autoconf macros here:
https://github.com/sagemath/sage/blob/master/configure.ac#L393

Also have a look at what is done for the integer multi precision library:
https://github.com/sagemath/sage/blob/master/configure.ac#L657

What is done for BLAS can also help.

I don't have much time these days, but if you don't feel at ease enough with autotools I can give it a try.

[Jean-Pierre Flori]

There is also the need_to_install  magic:
https://github.com/sagemath/sage/blob/master/configure.ac#L807

[Jeroen Demeyer]

Bad idea. Use the top-level configure for that.

[Emmanuel Charpentier]

Okay. I I have followed you correctly, we have two (mutually incompatible) options :

• (Dima's option) : package curl, pcre and xz as standard packages, and make R depend on them (unconditionnally).
• (Jean-Pierre option) : add tjem to Sage's core, but build them if and only if they are not installed systemwide (= useable at Sage's compile time) ; do this before trying to build R.

The first option is extremely simple and failsafe. The cost is about 15 MB (sum of all installed files in a temporary "prefix" directory, without shaving anything) and about 3 minutes of compilation time (no parallelism used). We might shave 4,9 MB of docs (local/share/(doc|man). Binaries are  about 0.9MB. The total potential cots (sum of sizes of the three build directories) is 89 MB (= potential cost if the three build directories ate kept after building i. e. for debugging purposes).

The second one requires hacking the main Sage configuration. I do not feel currently quite at ease with this one, but I'm just reading the relevant docs. I have no idea of the potential costs, save for what can be deduced from the costs of Dima's options.

The first option is compatible with the "batteries included" philosophy flaunted by Sage. are the potential 10-15 MB savings worth of hacking the main Sage config file ?

Your advice ?

--
Emmanuel Charpentier

[Dima Pasechnik]

On Monday, October 24, 2016 at 9:45:18 PM UTC, Emmanuel Charpentier wrote:

Okay. I I have followed you correctly, we have two (mutually incompatible) options :

• (Dima's option) : package curl, pcre and xz as standard packages, and make R depend on them (unconditionnally).
• (Jean-Pierre option) : add tjem to Sage's core, but build them if and only if they are not installed systemwide (= useable at Sage's compile time) ; do this before trying to build R.

The first option is extremely simple and failsafe. The cost is about 15 MB (sum of all installed files in a temporary "prefix" directory, without shaving anything) and about 3 minutes of compilation time (no parallelism used). We might shave 4,9 MB of docs (local/share/(doc|man). Binaries are  about 0.9MB. The total potential cots (sum of sizes of the three build directories) is 89 MB (= potential cost if the three build directories ate kept after building i. e. for debugging purposes).

I don't really follow you: note that gcc is also a standard package, but it only really gets installed if the system gcc is 

not good enough. That is to say, if curl, pcre, and xz are available system-wide, Sage should not try to install them,

no megabytes used...

[Jean-Pierre Flori]

On Tuesday, October 25, 2016 at 12:48:39 AM UTC+2, Dima Pasechnik wrote:

On Monday, October 24, 2016 at 9:45:18 PM UTC, Emmanuel Charpentier wrote:

Okay. I I have followed you correctly, we have two (mutually incompatible) options :

• (Dima's option) : package curl, pcre and xz as standard packages, and make R depend on them (unconditionnally).
• (Jean-Pierre option) : add tjem to Sage's core, but build them if and only if they are not installed systemwide (= useable at Sage's compile time) ; do this before trying to build R.

The first option is extremely simple and failsafe. The cost is about 15 MB (sum of all installed files in a temporary "prefix" directory, without shaving anything) and about 3 minutes of compilation time (no parallelism used). We might shave 4,9 MB of docs (local/share/(doc|man). Binaries are  about 0.9MB. The total potential cots (sum of sizes of the three build directories) is 89 MB (= potential cost if the three build directories ate kept after building i. e. for debugging purposes).

I don't really follow you: note that gcc is also a standard package, but it only really gets installed if the system gcc is 

not good enough. That is to say, if curl, pcre, and xz are available system-wide, Sage should not try to install them,

no megabytes used...

I don't think the real issue is megabytes, but rather making sage more modular and easier to ship into distributions (other than sage-the-distrib).
 

 

The second one requires hacking the main Sage configuration. I do not feel currently quite at ease with this one, but I'm just reading the relevant docs. I have no idea of the potential costs, save for what can be deduced from the costs of Dima's options.

The first option is compatible with the "batteries included" philosophy flaunted by Sage. are the potential 10-15 MB savings worth of hacking the main Sage config file ?

You can already add them as standard packages and then we can work on making them installed only if not available system-wide.
These two steps can be done sequentially though it would be better to get the two of them at the same time.

[Jean-Pierre Flori]

If you open a tcket with the first step, I'll do the second one, and Jeroen or someone else can review it.

[Emmanuel Charpentier]

This is now Trac#21767. The unconditional installation works okay on top of 7.5beta0.

The conditional installation can probably be obtained with this bash snippet, at the top of spkg-install :

## Installation de curl...

## Look for a systemwide installation

LibExists=no
if $(pkg-config --exists libcurl) ; then
    LibExists=yes
    echo A systemwide version of libcurl exists. Testing it.
else
    echo "No systemwide curl installation (or not using pkg-config)."
fi

## Systemwide installation version check

VersOK=no
if test $LibExists = yes ; then
    if ! test "7.28.0" ">" $(pkg-config --modversion libcurl) ; then
    echo Systemwide installation of libcurl is enough to support R.
    VersOK=yes
    else
    echo Systemwide version of libcurl too old to support our R.
    fi
fi

## Checking for a systemwide executable

ExecOK=no
if test $VersOK = yes ; then
    if test X$(which curl)X = XX ; then
    echo No systemwide curl executable in PATH.
    else
    echo Systemwide curl executable found.
    ExecOK=yes
    fi
fi

if test $ExecOK = yes ; then
    echo No local installation required.
    exit 0
else
    echo Starting curl installation...
fi

The same tachnique can probably been used for xz. pcre will need a supplemental step (checking configuration).

What do you think ?

--
Emmanuel Charpentier

[John H Palmieri]

I have no opinion on whether this approach is on the whole a good idea, but I will point out that "which curl" is not as portable as "command -v curl". We had an issue a while ago with "which" not behaving properly on some platform -- maybe on Solaris "which blah" had a return status of 0 even if "blah" was not present? I forget. Anyway, if nothing else, try "command -v ..." instead of "which ...". (And would it be even better to try to run curl to make sure it functions properly, instead of just seeing if it's present?)

  John

[Volker Braun]

One issue with curl is that you sooner or later will need the root certificates to use it, namely when you start downloading from a https site. E.g. in hashstack I recently added the root certs:

https://github.com/hashdist/hashstack/issues/969

This issue already faces us with openssl but since we actually don't use it much we could avoid it so far. But when replacing curl in the PATH we are going to run into that issue.

I would be happy with adding curl (and up-to-date root certificates) to the list of system requirements for building Sage and avoid that can of worms.

PS: and no, you can't use the OS-provided certs, since Apple had to cook their own sauce once more.

[Emmanuel Charpentier]

Le mercredi 26 octobre 2016 00:42:01 UTC+2, John H Palmieri a écrit :

I have no opinion on whether this approach is on the whole a good idea, but I will point out that "which curl" is not as portable as "command -v curl". We had an issue a while ago with "which" not behaving properly on some platform -- maybe on Solaris "which blah" had a return status of 0 even if "blah" was not present? I forget.

Note that we don't use the return status, but the path expansion. I've no idea about the portability of both ; I note that which is a standard binary, whereas command is internal to bash. Pick your portability poison...
 

Anyway, if nothing else, try "command -v ..." instead of "which ...".

Easy to do.
 

(And would it be even better to try to run curl to make sure it functions properly, instead of just seeing if it's present?)

That's another can of worms. I'm trying to satisfy R's requirements ; the function of the installed package can be tested with the package test suite (sage -i -c...). Maintaining  the user's systemwide installation is out of Sage's scope.

The alternative is, of course, to install curl unconditionally. And the same could be said about xz and pcre, two other new R prerequisites...

--
Emmanuel Charpentier

[Emmanuel Charpentier]

Le mercredi 26 octobre 2016 03:12:04 UTC+2, Volker Braun a écrit :

One issue with curl is that you sooner or later will need the root certificates to use it, namely when you start downloading from a https site. E.g. in hashstack I recently added the root certs:

https://github.com/hashdist/hashstack/issues/969

This issue already faces us with openssl but since we actually don't use it much we could avoid it so far. But when replacing curl in the PATH we are going to run into that issue.

This issue is *distinct* from the one I'm trying to solve (iL e. satisfy R's prerequisites). Should't you open a relevant ticket ?

I would be happy with adding curl (and up-to-date root certificates) to the list of system requirements for building Sage and avoid that can of worms.

That would be a good solution, but a bit at ill ease with the "batteries included" motto. Ditto for possible addition of xz and pcre.

PS: and no, you can't use the OS-provided certs, since Apple had to cook their own sauce once more.

<Sigh...> BTW, what is the point of buying an expensive Mac for running Sage ?

--
Emmanuel Charpentier
 

[Jeroen Demeyer]

On 2016-10-25 23:52, Emmanuel Charpentier wrote:
> This is now Trac#21767 <https://trac.sagemath.org/ticket/21767>. The

> unconditional installation works okay on top of 7.5beta0.
>
> The conditional installation can probably be obtained with this bash
> snippet, at the top of spkg-install :

Unless there is a good reason to do otherwise, such checks are better
done in the top-level configure instead.

[Emmanuel Charpentier]

I'm not so sure : harder to do, harder to understand, not obvious at reading the spkg-install script. But cleaner, indeed.

BTW, I'm having second thoughts on the wisdom of installing conditionally to the current system's state. I'll open the discussion in another thread.

--
Emmanuel Charpentier

[Emmanuel Charpentier]

See this thread.

--
Emmanuel Charpentier

[Jean-Pierre Flori]

On Wednesday, October 26, 2016 at 9:24:35 AM UTC+2, Emmanuel Charpentier wrote:

Le mercredi 26 octobre 2016 09:18:06 UTC+2, Jeroen Demeyer a écrit :

On 2016-10-25 23:52, Emmanuel Charpentier wrote:
> This is now Trac#21767 <https://trac.sagemath.org/ticket/21767>. The
> unconditional installation works okay on top of 7.5beta0.
>
> The conditional installation can probably be obtained with this bash
> snippet, at the top of spkg-install :

Unless there is a good reason to do otherwise, such checks are better
done in the top-level configure instead.

I'm not so sure : harder to do, harder to understand, not obvious at reading the spkg-install script. But cleaner, indeed.

One good reason is that when two packages will depend on curl we don't need to duplicate code.

[Emmanuel Charpentier]

On 26/10/2016 09:59, Jean-Pierre Flori wrote:
>
>
> On Wednesday, October 26, 2016 at 9:24:35 AM UTC+2, Emmanuel Charpentier
> wrote:
>
> Le mercredi 26 octobre 2016 09:18:06 UTC+2, Jeroen Demeyer a écrit :
>
> On 2016-10-25 23:52, Emmanuel Charpentier wrote:
> > This is now Trac#21767 <https://trac.sagemath.org/ticket/21767
> <https://trac.sagemath.org/ticket/21767>>. The
> > unconditional installation works okay on top of 7.5beta0.
> >
> > The conditional installation can probably be obtained with
> this bash
> > snippet, at the top of spkg-install :
>
> Unless there is a good reason to do otherwise, such checks are
> better
> done in the top-level configure instead.
>
>
> I'm not so sure : harder to do, harder to understand, not obvious at
> reading the spkg-install script. But cleaner, indeed.
>
> One good reason is that when two packages will depend on curl we don't
> need to duplicate code.

??? I don't understand your point : the code being at the top of curl's
spkg-install script, it has not to be duplicated (nor executed twice :
curl would be marked as installed, and packages depending on it wouldn't
trigger it twice.

Putting something analogous in the main (Sage's) configure scriptt would
mean integrating curl in Sage itself. R being a standard package and
depending on curl, it would amount to the same thing. In which case,
"packaging" curl (i. e. create a distinct software entity) is nonsense ;
just add it to Sage's source tarball and, indeed, add the necessray
voodoo in configure (or, rather, to its sources, if any).

I'm not sure I'm following you...

Furthermore :

[Jean-Pierre Flori]

nah it was morning,I thought you  meant at top of the R spkg-install script.

[Jeroen Demeyer]

On 2016-10-26 09:24, Emmanuel Charpentier wrote:
> I'm not so sure : harder to do, harder to understand, not obvious at
> reading the spkg-install script. But cleaner, indeed.

Plus, it allows for explicit installation if you want to install it
anyway. It's nice for example that `./sage -i gcc` does install gcc,
regardless of whether you need it or not.

If the check would be in the spkg-install script of curl, then `./sage
-i curl` would *not* install curl.

[Volker Braun]

On Wednesday, October 26, 2016 at 2:40:42 AM UTC-4, Emmanuel Charpentier wrote:

This issue is *distinct* from the one I'm trying to solve (iL e. satisfy R's prerequisites). Should't you open a relevant ticket ?

Its not really a distinct issue, its a general caveat of compiling your own curl. And even if we (and/or R) don't use curl to download over ssl today, this will surely change in the future.

[Jean-Pierre Flori]

Hum, it seems to me that R wants curl with https support, look at
https://svn.r-project.org/R/trunk/m4/R.m4

"libcurl >= 7.28.0 library and headers are required with support for https"

[John H Palmieri]

On Tuesday, October 25, 2016 at 11:35:15 PM UTC-7, Emmanuel Charpentier wrote:

Le mercredi 26 octobre 2016 00:42:01 UTC+2, John H Palmieri a écrit :

I have no opinion on whether this approach is on the whole a good idea, but I will point out that "which curl" is not as portable as "command -v curl". We had an issue a while ago with "which" not behaving properly on some platform -- maybe on Solaris "which blah" had a return status of 0 even if "blah" was not present? I forget.

Note that we don't use the return status, but the path expansion. I've no idea about the portability of both ; I note that which is a standard binary, whereas command is internal to bash. Pick your portability poison...

I don't even remember if the return status was the issue. I would also point out that according to the web page http://pubs.opengroup.org/onlinepubs/9699919799/, "command" is listed as a standard shell utility, but not "which".

--
John

[Jean-Pierre Flori]

There is a curl-config utility.