distutils versus setuptools for the Sage library (was Re: Segregating development from production)

[William Stein]

On Fri, Mar 9, 2012 at 11:17 PM, Keshav Kini <kesha...@gmail.com> wrote:
> Jeroen Demeyer <jdem...@cage.ugent.be> writes:
>> On 2012-03-09 15:48, Keshav Kini wrote:
>>> I'm probably missing something obvious, but why do you need a repository
>>> in order to apply a patch to files? Doesn't `patch` work fine?
>>
>> We need the repository for exactly the same reason as we need a
>> repository for devel/sage.  When you are *developing* something, you
>> don't want to be restricted to using "patch".  If you are developing the
>> Sage build process (i.e. what the root repo does), you may want to do
>> that *before* building sage.
>
> You compare the need for a root repository to the need for the
> devel/sage repository, by which I guess you mean the current Sage
> library repository, not my proposed unified repository.
>
> But we do *not* run files directly from the devel/sage repository. They

The following is an important SIDE REMARK, which is not really related
to what you're discussing in this email, so I've changed the subject.

We should. The only reason we don't is because when I set things up
in 2004 (?) I think setuptools "python setup.py develop" either didn't
exist, or I had never heard of it. What *should* happen is that we
use setuptools instead of distutils, and the Sage library would live
100% and only at SAGE_ROOT/devel/sage/sage. Just to emphasize, the
Cython modules would then be built in place.

Making this change would (1) avoid an enormous amount of confusion
when people type "foo??" and see the source location, which is not the
path to the file they should edit, and (2) reduce the use of disk
space by several hundred megabytes.

I think one thing I liked about "python setup.py install" (which we
use now), was that there were 2-3 copies of every file, which made me
feel more secure at one point in time, when I was the only user of
Sage, and I didn't use any revision control system.

I did once spend "a few hours" trying to change Sage from distutils
(what we use now) to setuptools, and it is less trivial than one might
hope, but certainly do-able. It could be done as a several-day
project at a Sage Days.

I would like to make this change sometime in the next few months. If
anybody has any thoughts, feel free to share.

-- William

[Julien Puydt]

Le samedi 10 mars, William Stein a écrit:

> I would like to make this change sometime in the next few months. If
> anybody has any thoughts, feel free to share.

You already had my mind with your explanations, but the "reduce the use
of disk space by several hundred megabytes" totally stole my heart.

Snark on #sagemath

[Keshav Kini]

William Stein <wst...@gmail.com> writes:
> On Fri, Mar 9, 2012 at 11:17 PM, Keshav Kini <kesha...@gmail.com> wrote:
>> Jeroen Demeyer <jdem...@cage.ugent.be> writes:
>>> On 2012-03-09 15:48, Keshav Kini wrote:
>>>> I'm probably missing something obvious, but why do you need a repository
>>>> in order to apply a patch to files? Doesn't `patch` work fine?
>>>
>>> We need the repository for exactly the same reason as we need a
>>> repository for devel/sage.  When you are *developing* something, you
>>> don't want to be restricted to using "patch".  If you are developing the
>>> Sage build process (i.e. what the root repo does), you may want to do
>>> that *before* building sage.
>>
>> You compare the need for a root repository to the need for the
>> devel/sage repository, by which I guess you mean the current Sage
>> library repository, not my proposed unified repository.
>>
>> But we do *not* run files directly from the devel/sage repository. They
>
> The following is an important SIDE REMARK, which is not really related
> to what you're discussing in this email, so I've changed the subject.
>
> We should.

I just want to say that using `setup.py develop` is *not* at odds with
what I'm arguing for. In fact, it's very much aligned with what I want,
which is for our development repositories to have more freedom in terms
of where we put our code and how it is organized, which will facilitate
eventual consolidation into a single repository. So yes, I would be
quite happy to use `setup.py develop` with the Sage library. We already
use it for sagenb where it works great.

> The only reason we don't is because when I set things up
> in 2004 (?) I think setuptools "python setup.py develop" either didn't
> exist, or I had never heard of it. What *should* happen is that we
> use setuptools instead of distutils, and the Sage library would live
> 100% and only at SAGE_ROOT/devel/sage/sage. Just to emphasize, the
> Cython modules would then be built in place.
>
> Making this change would (1) avoid an enormous amount of confusion
> when people type "foo??" and see the source location, which is not the
> path to the file they should edit, and (2) reduce the use of disk
> space by several hundred megabytes.

Great! I for one wholly approve of this idea.

-Keshav

----
Join us in #sagemath on irc.freenode.net !

[Nicolas M. Thiery]

On Sat, Mar 10, 2012 at 06:59:56AM -0800, William Stein wrote:
> Making this change would (1) avoid an enormous amount of confusion
> when people type "foo??" and see the source location, which is not the
> path to the file they should edit

A big +1 to that change.

Another advantage is that it would not require to run `sage -b` after
each trivial change to a Python file. This will save me a few seconds
at each modify-test cycle, which will soon build up to a couple
minutes per days.

It's also much simpler to explain to newcomers (especially those using
a virtual machine where running sage -b is not immediate); we also
several times met the situation where running sage -b required a
compiler to be installed where it was not actually needed, which
forced us to download and install xcode. Which kind of ruined our
claim that "everybody could play around and edit the Sage sources".

Also that change might possibly even allow for "attaching" source
files from the Sage library, so that one could try new code without
even restarting Sage. That's not necessarily super robust, but would
save a lot of time in the easy cases; in case of doubt one can anyway
get back to a safe position by restarting Sage.

Cheers,
Nicolas
--
Nicolas M. Thi�ry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

[Nicolas M. Thiery]

On Sun, Mar 11, 2012 at 12:53:45PM +0100, Nicolas M. Thiery wrote:
> Another advantage is that it would not require to run `sage -b` after
> each trivial change to a Python file. This will save me a few seconds
> at each modify-test cycle, which will soon build up to a couple
> minutes per days.

Besides, this would solve the problem of unrelevant files not being
cleared up from the build directory when applying / unapplying patches
which create new files. I have just been hit by this, and this
confused Sphinx, and it was a pain to fix.

[Keshav Kini]

"Nicolas M. Thiery" <Nicolas...@u-psud.fr> writes:
> Besides, this would solve the problem of unrelevant files not being
> cleared up from the build directory when applying / unapplying patches
> which create new files. I have just been hit by this, and this
> confused Sphinx, and it was a pain to fix.

I was just about to say this. I wonder if switching to `setup.py
develop` would really clear this up completely, though - what about the
C extensions Cython builds?

[kcrisman]

> We should.  The only reason we don't is because when I set things up
> in 2004 (?) I think setuptools "python setup.py develop" either didn't
> exist, or I had never heard of it.   What *should* happen is that we
> use setuptools instead of distutils, and the Sage library would live
> 100% and only at SAGE_ROOT/devel/sage/sage.   Just to emphasize, the
> Cython modules would then be built in place.
>
> Making this change would (1) avoid an enormous amount of confusion
> when people type "foo??" and see the source location, which is not the
> path to the file they should edit, and (2) reduce the use of disk
> space by several hundred megabytes.

Wow, that would definitely make it be a lot easier to sometimes track
down certain things.

[Mike Hansen]

On Sat, Mar 10, 2012 at 6:59 AM, William Stein <wst...@gmail.com> wrote:
> I would like to make this change sometime in the next few months.  If
> anybody has any thoughts, feel free to share.

There's a patch at http://trac.sagemath.org/sage_trac/ticket/12659.
It should allow you have to both old (out-of-place) and new (in-place)
copies of the Sage library working side-by-side.

--Mike

[Keshav Kini]

Hmm. What is this? http://stackoverflow.com/a/6522905 Maybe it's worth
looking into using this new thing?

[Mike Hansen]

There is a patch at http://trac.sagemath.org/sage_trac/ticket/12659
which builds the Sage library in place just using distutils.

--Mike

> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org

[Keshav Kini]

Mike Hansen <mha...@gmail.com> writes:
> There is a patch at http://trac.sagemath.org/sage_trac/ticket/12659
> which builds the Sage library in place just using distutils.

Yup, I'm subscribed to the ticket :) I just found out about distutils2
and thought I'd mention it here.