Let's talk specifics (was Re: how we develop sage)

[Erik Bray]

On Thu, Apr 14, 2016 at 5:46 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> On 2016-04-14 17:38, Erik Bray wrote:
>>
>> Sage already has the problem of large
>> chunks of code that are effectively unmaintained and create a
>> maintenance burden on anyone serious about maintaining sage. Their
>> interfaces whither, and become inconsistent with the rest of the
>> package. It's dead weight.
>
> I disagree with the above. It's not necessarily a problem to have
> unmaintained-but-still-working code.

It is a problem to have unmaintained and maybe working but poorly
tested and poorly integrated code that nobody knows how to maintain
anymore and that doesn't meet the coding and interface standards of
the rest of the package.

My greatest joy as a software engineer is almost always deleting code
not adding it.

I think part of the larger problem of this discussion about
modularization is that nobody has made any concrete proposals yet.
You and others state that there are parts of Sage that are too
difficult to separate out because they're too deeply intertwined. But
nobody is necessarily proposing making cuts in those places. In some
cases maybe the extent of their circular dependency should be taken as
evidence of a need for refactoring. But I also believe you that there
are large parts where this is not the case. Those parts I believe
mostly make up the core package and shouldn't be broken up.

Perhaps some of us who support "modularization" as a general principle
should work together to discuss a specific proposal as to where to
make cuts and how to maintain this. Then it would be easier to
discuss the merits and problems with *specific* modifications, rather
than circular discussions about hypotheticals that may or may not be
relevant.

I'd certainly be interested in having that discussion with anyone else
interested. I don't know most of the Sage package well enough though
to know where to begin.

Thanks,
Erik

[mmarco]

Totally +1 for discussing specific proposals, although I don't have anything to propose right now.

 

I'd certainly be interested in having that discussion with anyone else
interested. I don't know most of the Sage package well enough though
to know where to begin.

In the other thread, I posted a quick estimation of how the dependency graph would look at module level inside sage the library. Maybe that could help to get a first impression about which modules could be treated separatedly without harming others. Short version is that there would be about 16 such modules. 

[Dima Pasechnik]

On Friday, April 15, 2016 at 1:34:57 PM UTC+1, Erik Bray wrote:

On Thu, Apr 14, 2016 at 5:46 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> On 2016-04-14 17:38, Erik Bray wrote:
>>
>> Sage already has the problem of large
>> chunks of code that are effectively unmaintained and create a
>> maintenance burden on anyone serious about maintaining sage.  Their
>> interfaces whither, and become inconsistent with the rest of the
>> package.  It's dead weight.
>
> I disagree with the above. It's not necessarily a problem to have
> unmaintained-but-still-working code.

It is a problem to have unmaintained and maybe working but poorly
tested and poorly integrated code that nobody knows how to maintain
anymore and that doesn't meet the coding and interface standards of
the rest of the package. 

sagenb is a good example here. A lot of it is a very old javascript code,

mixed up with twisted and its callbacks, and other partly outdated web design

things, that  perhaps noone actively involved in Sage project proper understands now.

Attempts to replace it completely with something newer  are stalled. It actually appears

to be a legacy that has to be maintained for long time, as there textbooks and courses

developed and relying on it...

There were attempts to improve it, which also failed, not the least by existence

of better alternatives (SMC and Jupyter nb).

My greatest joy as a software engineer is almost always deleting code
not adding it.

I think part of the larger problem of this discussion about
modularization is that nobody has made any concrete proposals yet.
You and others state that there are parts of Sage that are too
difficult to separate out because they're too deeply intertwined.  But
nobody is necessarily proposing making cuts in those places.  In some
cases maybe the extent of their circular dependency should be taken as
evidence of a need for refactoring. But I also believe you that there
are large parts where this is not the case.  Those parts I believe
mostly make up the core package and shouldn't be broken up.

the core problem is that the core package is actually very big.

It is probably 80% of Sage library.

And it has circular dependencies simply due to the fact that they 

follow circular mathematical dependencies of various mathematical

concepts (e.g. posets depend on graphs, graphs depend on posets, 

matroids depend on graphs and posets, yet graphs and posets also

depend on matroids; all the three depend on a lot of algebra, llinear and not,

and again the other way around, etc etc)

There a clear outliers, like finances, sagenb, graphics, and duplicates (functionality-wise), 

such as two different interefaces to GAP (pexpect and libGAP), to Maxima (pexpect vs library),

Cutting these out would already be a huge improvement...

(but e.g. completely replacing GAP with libGAP is quite a job...)

[Erik Bray]

Yeah, I saw that post. I haven't had time yet to really go over it
closely, but it will definitely help in performing a first pass. In
order to make it easier to find later, you wrote:

> there are 9 modules in the base of the dependency graph (that is, avery
> other module imports something from them, or from other modules that imports
> something on them, or so on). They are schemes, ext, docbuild, version, env,
> all_cmdline, all, all_notebook, and notebook

> over them there is a big set of modules that have circular dependencies
> between them. That is, they import something from the previous set, or from
> other module in this set. They are probability, data_structures, misc,
> combinat, server, quadratic_forms, homology, plot, functions, stats, matrix,
> rings, categories, algebras, calculus, arith, finance, symbolic, interfaces,
> numerical, modular, repl, groups, databases, libs, parallel, structure,
> geometry, modules, typeset, graphs, quivers, doctest, sets, gsl, monoids.
> Then there are 16 modules that import something from the previous two lists,
> but not from anything on this level. They are: games, tensor, matroids,
> knots, interacts, crypto, dev, logic, lfunctions, dynamics, sat,
> coding, game_theory, tests, sandpiles, media

> Finally, there is manifolds, that imports from tensor and from level (2).

> As I said, this is just a quick look (for instance, I didn't consider lazy
> imports). But I think it gives an idea of what is already moreless
> independent, and what is pretty much entangled with the rest of the
> components. One would be tempted to say that (2) consists of the "core" of
> the Sage library, whereas (1) are its dependencies and (3) and (4) would be
> "extra" modules that deppend on the core library. But again, this is just a
> first impression after a quick look.

Now, breaking up the existing sage library, which may or *may not* be
a good idea--I don't necessarily presuppose that it is, though I think
it's worth looking into in detail to determine that. But we also have
to discuss what to do better moving forward and there was also some
discussion toward the end of the other thread about how to make it
easier to develop packages that depend on sage but are not subpackages
thereof, and how to better advertise those packages. That's also
worth discussing here. See also my posts about how the Astropy
project manages "affiliated packages".

Thanks,
Erik

[Erik Bray]

I admit I obviously don't have a deep understanding of how sagenb
integrates with sage-the-library. I think a better thing to do with
it would be to determine how it currently depends on sage and maintain
those interfaces for some time to come so that it's still possible to
open and use existing sage notebooks. I agree it should be kept
functional, but folding it back into the main library seems like a
backwards approach to me :)

>> My greatest joy as a software engineer is almost always deleting code
>> not adding it.
>>
>> I think part of the larger problem of this discussion about
>> modularization is that nobody has made any concrete proposals yet.
>> You and others state that there are parts of Sage that are too
>> difficult to separate out because they're too deeply intertwined. But
>> nobody is necessarily proposing making cuts in those places. In some
>> cases maybe the extent of their circular dependency should be taken as
>> evidence of a need for refactoring. But I also believe you that there
>> are large parts where this is not the case. Those parts I believe
>> mostly make up the core package and shouldn't be broken up.
>
>
> the core problem is that the core package is actually very big.
> It is probably 80% of Sage library.
> And it has circular dependencies simply due to the fact that they
> follow circular mathematical dependencies of various mathematical
> concepts (e.g. posets depend on graphs, graphs depend on posets,
> matroids depend on graphs and posets, yet graphs and posets also
> depend on matroids; all the three depend on a lot of algebra, llinear and
> not,
> and again the other way around, etc etc)

That all sounds reasonable. I don't disagree with this--at the end
the "core" library may still be quite large.

> There a clear outliers, like finances, sagenb, graphics, and duplicates
> (functionality-wise),
> such as two different interefaces to GAP (pexpect and libGAP), to Maxima
> (pexpect vs library),
> Cutting these out would already be a huge improvement...
> (but e.g. completely replacing GAP with libGAP is quite a job...)

Yep, I think these are the best places to focus first, as well as a
lot of the other stuff in sage.interfaces.

What is the difficulty as it stands with completely switching over to
libGAP? I was under the (apparently false?) impression that that was
mostly a done deal by this point, despite the pexect interface still
being around.

Thanks,
Erik

[Erik Bray]

Another thing I'm considering--though I could be wrong about this--is
sage.ext.fast_callable. I only even found out about this by working
on fixing sage.ext.interpreters to build in Cygwin. (I've heard
mention of fast_callable before but nobody explained very well what it
is--the documentation is fine though).

I wish I had know about this sooner, and furthermore I wish it had
been usable and discoverable outside of sage. I've done a lot of the
same work myself, and have merely protyped some of the kind of
bytecode generation from expressions that it does--never had the time
to get that fully working though.

Currently sage.ext.fast_callable is written in such a way that depends
on some of sage's core data structures, but I'm not convinced it needs
to be that way. With some reworking and more extensibility I think it
could be redesigned in such a way that doesn't depend on sage, but
that sage can easily build on top of to add specific support for its
structures. A good example where this is already done is in how
ElementInterpreter extends PythonInterpreter to check that sage
Elements are in the correct parent domain (IIUC).

Eirk

[Dima Pasechnik]

well:

upstream$ tar tf sagenb-0.11.7.tar 

sagenb-0.11.7/

sagenb-0.11.7/webassets-0.11.1.tar.gz

sagenb-0.11.7/Flask-OpenID-1.2.5.tar.gz

sagenb-0.11.7/python-openid-2.2.5.zip

sagenb-0.11.7/itsdangerous-0.24.tar.gz

sagenb-0.11.7/zope.interface-4.1.3.tar.gz

sagenb-0.11.7/Flask-0.10.1.tar.gz

sagenb-0.11.7/install_order

sagenb-0.11.7/sagenb-0.11.7.tar.gz

sagenb-0.11.7/speaklater-1.3.tar.gz

sagenb-0.11.7/Flask-Babel-0.9.tar.gz

sagenb-0.11.7/Twisted-15.5.0.tar.bz2

sagenb-0.11.7/Flask-Silk-0.2.tar.gz

sagenb-0.11.7/pytz-2013b.zip

sagenb-0.11.7/Flask-AutoIndex-0.5.tar.gz

sagenb-0.11.7/Werkzeug-0.11.4.tar.gz

sagenb-0.11.7/Babel-2.2.0.tar.gz

sagenb-0.11.7/Flask-OldSessions-0.10.tar.gz

need I say more?

(of this, webassets it not used...)

IMHO it's still a good solid person-month of work, to clean it all up, to replace various glue

functions which are GAP-specific, but absent in libGAP, etc...

And libGAP itself still needs some fixes, as it does not work when one has to exchange a hell of a lot of data, IIRC.

(old GAP interface reads/writes files for such cases)

 

Dima

[Ralf Stephan]

Once there is a consensus about what the Sage core consists of, what you then want

is a way to signal developers downstream that

 * "this new revision of the Sage core does not change its interface"

 * "this new revision of the Sage core has (backwards-compatible) additions to its interface"

 * "this new revision of the Sage core removes parts of its interface"

Sounds familiar? Linux shared libraries use major/minor/micro version numbers

to load the newest still compatible version via libtool. The library developers just

need to take care to apply a specific version increment scheme. See e.g.

http://www.freesoftwaremagazine.com/articles/building_shared_libraries_once_usi
ng_autotools (the part about "The Libtool library versioning scheme" has comparisons

between different numbering schemes).

What I'm proposing is to apply such a scheme to allow clear signalling of changes

that were done in the newest Sage core version vs the earlier ones. This presupposes

of course that the core is defined.

Regards, 

[Erik Bray]

Great point--in addition to what may or may not need to be done with
the sage library itself, switching to a reliable version scheme is
perhaps even more important--a prerequisite to anything else.

Thanks,
Erik

[Dima Pasechnik]

in fact, it seems that things are moving here, see e.g. http://trac.sagemath.org/ticket/20276

 

 

Dima

[Jeroen Demeyer]

On 2016-04-15 14:34, Erik Bray wrote:
> I'd certainly be interested in having that discussion with anyone else
> interested. I don't know most of the Sage package well enough though
> to know where to begin.

Obviously, I'm also interested in that discussion.

[Jeroen Demeyer]

On 2016-04-15 14:34, Erik Bray wrote:

> It is a problem to have unmaintained and maybe working but poorly
> tested and poorly integrated code that nobody knows how to maintain
> anymore and that doesn't meet the coding and interface standards of
> the rest of the package.

There is a huge amount of code like that in the Sage distribution. That
is a fact which we cannot ignore.

[Vincent Delecroix]

I don't think that the question where to cut is meaningful in a first
time. Cutting more should not be a problem.

I would make an important distinction between:

1 - Core fonctionality that we may package in some way to be useful to
others. I think Cython, cysignals are meaningful examples. Though, these
two examples are developed outside of Sage.

2 - Optional functionality that may be useful to Sage users.

This is basically the distinction we currently have between a "Sage
core" (still made of packages) and "Sage optional packages". Of course
there could be much more that currently in both categories.

Examples of possible cutting of type 1 (Sage core):

- a lot in sage/libs (e.g. pari and the work of Jeroen to update
cypari [1], the gsl interface, interface to LP solvers, ...)
- fast_callable as Erik mentioned

Examples of possible cutting of type 2 (things that needs Sage core):

- graphs
- a large part of combinat
- games
- modular groups
- elliptic curves

I think that the development framework should not necessarily be the
same for both. I feel like we should keep "Sage core" as centralized as
possible concerning its development. Even though, each release of Sage
might also be a release of other subpackages.

Concerning the extras, I think that it should not be a problem to make
it very open. On one hand, some of them could still be centralized in
development with Sage core. But anybody should be able to have its own
package on its webpage as it already exists.

Hence for me we lack in order:

* a model that allows centralized development but modularized
distribution

* a patchbot that tests core and optional packages as well as their
interactions

* a nice howto (as already started by Chris)

* having a centralized way of testing for presence of extra packages
(a good step would be [#20382])

Starting cutting is just about people time and interest...

Cheers
Vincent

[1] https://github.com/jdemeyer/cypari
[#20382] http://trac.sagemath.org/ticket/20382

[mmarco]

I would consider graphs to be part of the core. There are many other modules that deppend on it. Maybe it could be split in two parts though.

I think one of the problems to be addressed is that we lack a system to have "external packages". We do have "optional packages", but that is another history, since they go through the usual development process. One has to write the corresponding scripts, open an account in trac, open a ticket, and wait for it to be reviewed, this stops a lot of people. We would need something simpler for people that just wrote a few functions to compute something related to their research, and want to share them with others.

[Dima Pasechnik]

On Friday, April 15, 2016 at 5:12:28 PM UTC+1, vdelecroix wrote:

I don't think that the question where to cut is meaningful in a first
time. Cutting more should not be a problem.

I would make an important distinction between:

  1 - Core fonctionality that we may package in some way to be useful to
others. I think Cython, cysignals are meaningful examples. Though, these
two examples are developed outside of Sage.

  2 - Optional functionality that may be useful to Sage users.

This is basically the distinction we currently have between a "Sage
core" (still made of packages) and "Sage optional packages". Of course
there could be much more that currently in both categories. 

IMHO you are  underestimate how well-connected Sage lib is.

 

Examples of possible cutting of type 1 (Sage core):

  - a lot in sage/libs (e.g. pari and the work of Jeroen to update
cypari [1], the gsl interface, interface to LP solvers, ...)

LP solvers are used by graphs, coding theory, what else?

 

  - fast_callable as Erik mentioned

perhaps.

 

Examples of possible cutting of type 2 (things that needs Sage core):

  - graphs
  - a large part of combinat  

graphs are used by combinat (posets!), and graphs use combinat.

Posets are used by polyhedral code, which is also used by combinat, 

and by various toric varieties stuff...

  - games
  - modular groups
  - elliptic curves

perhaps.

[mmarco]

LP solvers are used by graphs, coding theory, what else?

 

knots, geometry.cone, numerical.knapsack, numerical.optimize and combinat.integer_vector  (and for instance, matrix deppends on this).

[Vincent Delecroix]

On 15/04/16 14:18, mmarco wrote:
> I would consider graphs to be part of the core. There are many other
> modules that deppend on it. Maybe it could be split in two parts though.

I still consider that where to split should not be part of the current
discussion.

> I think one of the problems to be addressed is that we lack a system to
> have "external packages". We do have "optional packages", but that is
> another history, since they go through the usual development process. One
> has to write the corresponding scripts, open an account in trac, open a
> ticket, and wait for it to be reviewed, this stops a lot of people. We
> would need something simpler for people that just wrote a few functions to
> compute something related to their research, and want to share them with
> others.

+1. The need of modifying Sage source code to have an external package
is very bad. Moreover, it does not give any guarantee that a given
optional package is actually working (beyond the fact that some people
actually uses it). So this is about:

* how to completely externalize existence of optional packages (Python
packaging/library is one way already in use)

* how to integrate in the patchbot testing optional packages (see
#20182 in that direction)

But on the other hand, if Sage core has no idea about the list of
packages, how could we test them? Would make sense to have somewhere a
list of packages together with a classification along:
- link to tarball broken
- does not build
- does build but have not test suite or test suite fails
- does build and test suite succeeded

The current optional vs experimental reflects this a bit.

[Vincent Delecroix]

Sorry to have to stop the prose about your passion for LP solvers, but
you completely misunderstood my post (or possibly not even read it). My
proposition was to make the LP solvers part a "Sage core" packages.
Meaning that:
- it does not need Sage to live
- Sage needs it to live

But again, I think that the priority is to end up with a concrete howto
proposal. Not about which part should be split. (with that respect it
seems that you also did not read my post).

[mmarco]

My proposal would go in the direction of having three categories: optional, experimental, and external. Optional and Experimental would follow moreless what we have now, external programs that provide some functionality, and we keep them in our packaging system because we cannot rely on them being provided by the distro (or maybe we could, but we need some patching or specific version). They are more tied to "Sage the distribution" than to "Sage the library".

What I propose to call external packages would be something different. They would go in the spirit of sage/python code writen by sage users for their teaching/rsearch/whatever. In that sense, they would be in spirit closer to "Sage the library", but the Sage development team would make no promises about it (since it wouldn't go through the review process). They could be provided by any of the usual ways to install python modules (pip, easy_install...) but we could have some kind of database to do some automatic testing. We could even distribute them (as optional packages or calling them dfifferently). I think providing a template and howto for the people that wants to share this kind of code is a good start. A next step would be to maintain a database  and maybe include some automated way to install them.

[John H Palmieri]

Would it also be possible for them to be installed either in the main Sage directory or in the user's .sage directory, depending on how the installation was invoked and/or permissions? Pardon my ignorance, but how does Python deal with this?

  John

[Vincent Delecroix]

It is not only about Python but operating system distribution. Usually
on linux everything is managed by the distribution and you just have to
do "apt-get install python-nice-package" (or yum or whatever). This
results in a global installation usually in /usr/lib/pythonversion/.
Should be similar under FreeBSD. Though this needs the intervention of
some administrator.

You can also do this "out of the distro" using "pip".

If you want to add user packages you need to setup environment variables
PYTONPATH and/or PYTHONHOME [1] pointing to the appropriate directory.
Otherwise your python would not be able to find your libraries. To
install a module that way with pip

$ pip install -t DIR package

Concrete example

$ pip install -t /home/john/lib/ biopython
$ export PYTHONPATH=/home/john/lib/
$ python

and then biopython is available!

There is also the alternative possibility of having virtual environment
for even more isolation [2].

I am sure that there are more ways of doing things some of which I do
not know of.

Vincent

[1] https://docs.python.org/3.4/using/cmdline.html
[2] http://docs.python-guide.org/en/latest/dev/virtualenvs/

[Simon King]

On 2016-04-15, Vincent Delecroix <20100.d...@gmail.com> wrote:
> +1. The need of modifying Sage source code to have an external package
> is very bad.

I miss old-style packages, too. With them, it has been possible to just
install an spkg from a file; in contrast, new-style packages require to
change Sage sources (i.e., SAGE_ROOT/build/...) in order to install the
spkg.

Cheers,
Simon

[Dima Pasechnik]

On Friday, April 15, 2016 at 10:13:22 PM UTC+1, Simon King wrote:

On 2016-04-15, Vincent Delecroix <20100.d...@gmail.com> wrote:
> +1. The need of modifying Sage source code to have an external package
> is very bad.

I miss old-style packages, too.

we didn't get to see an spkg with a virus or with `rm -rf /`,  but surely it was just a question of time...

 

[mmarco]

That is a question that we have to address: if we allow this kind of external packages (or furthermore, move the devlopments of parts of sage to a different workflow)... what kind of control will we do about them? Will we make no promises about them at all? Have some kind of revision process before accepting them? If so, what exactly do we check?

The fact that all Sage code goes through a peer review process is an important point. So, if we split the development process in different modules... should we enforce such a review process in all the modules?

[William Stein]

On Fri, Apr 15, 2016 at 1:06 PM, Vincent Delecroix
<20100.d...@gmail.com> wrote:
[..]

> You can also do this "out of the distro" using "pip".
>
> If you want to add user packages you need to setup environment variables
> PYTONPATH and/or PYTHONHOME [1] pointing to the appropriate directory.
> Otherwise your python would not be able to find your libraries. To install a
> module that way with pip
>
> $ pip install -t DIR package
>
> Concrete example
>
> $ pip install -t /home/john/lib/ biopython
> $ export PYTHONPATH=/home/john/lib/
> $ python
>
> and then biopython is available!

You can also do

$ pip install --user biopython
$ python (or sage)

and then biopython is available. It puts things in ~/.local, and
you don't have to do anything with PYTHONPATH.

> There is also the alternative possibility of having virtual environment for
> even more isolation [2].
>
> I am sure that there are more ways of doing things some of which I do not
> know of.
>
> Vincent
>
> [1] https://docs.python.org/3.4/using/cmdline.html
> [2] http://docs.python-guide.org/en/latest/dev/virtualenvs/
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--
William (http://wstein.org)

[William Stein]

On Fri, Apr 15, 2016 at 4:49 PM, mmarco <mma...@unizar.es> wrote:
> That is a question that we have to address: if we allow this kind of
> external packages

It is not up to us to decide. Nobody can stop people from making
external Python packages that depend on Sage. Absolutely anybody with
any skills could read this page:

https://python-packaging.readthedocs.org/en/latest/minimal.html

and in about 20 minutes create a Python package that is visible on
pypi and which can be "pip installed" into your copy of Sage. (No
claims about it being useful.) As Volker has mentioned before,
everything is already in place. My hope is just that we make it
even easier than it already is, and actually everybody is doing that
right now :-)

> (or furthermore, move the devlopments of parts of sage to
> a different workflow)... what kind of control will we do about them? Will we
> make no promises about them at all? Have some kind of revision process
> before accepting them? If so, what exactly do we check?
>
> The fact that all Sage code goes through a peer review process is an
> important point. So, if we split the development process in different
> modules... should we enforce such a review process in all the modules?

All sage code that we distribute as part of the Sage library goes
through review.
That's not the case for "all sage code" though, in the sense of code
that depends on Sage.

It might be worth thinking about this like the difference between
"writing a paper and putting it on arxiv" and "publishing a paper in a
top journal"... and your questions are also extremely good questions
for the "publish a paper" setting. Erik Bray shared his experiences
about "affiliated packages" for his previous project...

-- William

>
> El viernes, 15 de abril de 2016, 23:40:05 (UTC+2), Dima Pasechnik escribió:
>>
>>
>>
>> On Friday, April 15, 2016 at 10:13:22 PM UTC+1, Simon King wrote:
>>>
>>> On 2016-04-15, Vincent Delecroix <20100.d...@gmail.com> wrote:
>>> > +1. The need of modifying Sage source code to have an external package
>>> > is very bad.
>>>
>>> I miss old-style packages, too.
>>
>>
>> we didn't get to see an spkg with a virus or with `rm -rf /`, but surely
>> it was just a question of time...
>>
>

[mmarco]

Sure, but I was thinking about something like a specific list of those pip packages that are, in some sense, tied to Sage. Where "tied" might mean several things, from "we keep a list of them in the Sage webpage" to "we ship them in Sage the distribution", with many possible intermediate points, that might or might not include the following:

• A list of these packages in the Sage web page.
• Instructions on how to write, package and distribuite them.
• Some kind of policy about style, reviewing and/or testing in order to be included in the list
• Include the doctests in these packages as part of our testuite.
• Include them in the Sage documentation
• ...

Is it possible, for instance, to get a list of all packages in PyPI that deppend on sage?

[Erik Bray]

On Fri, Apr 15, 2016 at 8:04 PM, mmarco <mma...@unizar.es> wrote:
> My proposal would go in the direction of having three categories: optional,
> experimental, and external. Optional and Experimental would follow moreless
> what we have now, external programs that provide some functionality, and we
> keep them in our packaging system because we cannot rely on them being
> provided by the distro (or maybe we could, but we need some patching or
> specific version). They are more tied to "Sage the distribution" than to
> "Sage the library".
>
> What I propose to call external packages would be something different. They
> would go in the spirit of sage/python code writen by sage users for their
> teaching/rsearch/whatever. In that sense, they would be in spirit closer to
> "Sage the library", but the Sage development team would make no promises
> about it (since it wouldn't go through the review process). They could be
> provided by any of the usual ways to install python modules (pip,
> easy_install...) but we could have some kind of database to do some
> automatic testing. We could even distribute them (as optional packages or
> calling them dfifferently). I think providing a template and howto for the
> people that wants to share this kind of code is a good start. A next step
> would be to maintain a database and maybe include some automated way to
> install them.

+1 I think this is in the right direction. And I think that some of
the existing leaf packages of the core sage-lib could be
re-distributed in this form. But as Vincent has repeatedly written it
would be more important to have this infrastructure and process in
place than it is in most cases to decide specific things to *remove*
from sage-lib. Even though I think there are things that should be
split out I could agree it's not as urgent to do so, and it should be
handled with great care.

This is Astropy's "database" of affiliated packages (similar to what
you're calling "external packages"):
https://github.com/astropy/astropy.github.com/blob/master/affiliated/registry.json

I've argued a few times (but without much action on my part) that it
should be more detailed, including summaries and separate detailed
descriptions of the packages' contents, links to their documentation
(which is sometimes different from their home page) so that their
objects.inv can be downloaded easily for inter-sphinx, and possibly
even the commands to run to build, install, and test those packages
(in most cases these are the same since most of these packages use a
template that provides all these features).

Erik

[Erik Bray]

On Sat, Apr 16, 2016 at 1:49 AM, mmarco <mma...@unizar.es> wrote:
> That is a question that we have to address: if we allow this kind of
> external packages (or furthermore, move the devlopments of parts of sage to
> a different workflow)... what kind of control will we do about them? Will we
> make no promises about them at all? Have some kind of revision process
> before accepting them? If so, what exactly do we check?
>
> The fact that all Sage code goes through a peer review process is an
> important point. So, if we split the development process in different
> modules... should we enforce such a review process in all the modules?

Should we enforce such a review process? I say no--many such external
packages would be developed by individual researchers or small teams
of researchers. Really the amount of support that can be given a
particular package varies widely depending on who's working on it, and
we don't want them to be a burden to the developers of the core
package.

But there should definitely be some standards for acceptance as an
"official" external package. I don't know exactly what those
standards should be since I'm new to this community, though I could
make a few (incomplete) suggestions.

If we want to incorporate patchbot testing against external packages I
think that's good, but if some particular external package is not
being maintained anymore it should have an explicit cap placed on its
max supported version of sage (assuming it breaks with some newer
version of sage) and should be dropped from testing. In the chance
someone finds such an unmaintained package useful to them they would
still have options such as:

* Take over maintenance of the package, even if only just long enough
to update it to work with newer versions of sage
* Run it with an older version of sage--this will be easier as sage
packaging improves. This will also be made easier by our efforts to
provide docker containers for sage. Although the process isn't as
formalized and automated as I want yet, the plan is to eventually
provide docker containers for most individual versions of sage.
* ...others...?

> El viernes, 15 de abril de 2016, 23:40:05 (UTC+2), Dima Pasechnik escribió:
>>
>>
>>
>> On Friday, April 15, 2016 at 10:13:22 PM UTC+1, Simon King wrote:
>>>
>>> On 2016-04-15, Vincent Delecroix <20100.d...@gmail.com> wrote:
>>> > +1. The need of modifying Sage source code to have an external package
>>> > is very bad.
>>>
>>> I miss old-style packages, too.
>>
>>
>> we didn't get to see an spkg with a virus or with `rm -rf /`, but surely
>> it was just a question of time...
>>
>

[Robert Bradshaw]

This has been a very interesting discussion... 

There are pros and cons to modularity, and it's worth noting that this can be done at various different levels. For example, where I work we have a single, monolithic codebase (very convenient) but at the same time have fairly strict, explicit dependencies, public APIs, and owners (modular). 

As for which direction Sage should move, more below...

+1

We should build the infrastructure to make it easy for packages to depend on Sage but not be part of Sage. Part of this is a cultural shift. But I think tooling could be a huge part of the process. One thing people forget is that CPU cycles are actually pretty cheap. (Initially I didn't know how up-to-date the patchbot would be, but was pleasantly surprised that, with the exception of right after a new release, a single machine was able to keep up and was still idle most of the time.) Simply making it easy for a project to run it's tests against every commit (or even pull request aka trac ticket) of Sage, as well as the latest release, could go a long way. We wouldn't promise to not break external packages, but at least we couldn't do so in ignorance which is 90% of the battle. 

Once this is healthy, we can think about moving existing, loosely-coupled, less-frequently-used modules out. 

Well defined dependencies and public APIs within "core" Sage would be a useful cleanup too... but that's another story. 

- Robert

[Kwankyu Lee]

This discussion is hardening the terms: Sage core and external packages. But from the point of view of the people developing the would-be external packages, the official term would better be 

Sage extension,

Sage library extension,

Sage library extension package,

or SLEP.

Or simply extension package. 

[William Stein]

Or

Python package (that depends on Sage)

>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

--
William (http://wstein.org)

[Erik Bray]

On Wed, Apr 27, 2016 at 5:05 PM, William Stein <wst...@gmail.com> wrote:
> On Wed, Apr 27, 2016 at 7:09 AM, Kwankyu Lee <ekwa...@gmail.com> wrote:
>> This discussion is hardening the terms: Sage core and external packages. But
>> from the point of view of the people developing the would-be external
>> packages, the official term would better be
>>
>> Sage extension,
>> Sage library extension,
>> Sage library extension package,
>> or SLEP.

Two easily confused with "extension module" in the sense of a Python
module written in C.

>> Or simply extension package.
>
> Or
>
> Python package (that depends on Sage)

Well remember there are two types of projects we're talking about here:

1) Projects that adhere to some yet-to-be-determined standards set out
by the Sage project, and that in return are promoted as extra, if not
more specialized products built on Sage, included in
Sage-the-distribution, and possibly used in continuous integration
with Sage as well. Those would deserve a special name "affiliated
package" or the like.

2) Packages that happen to "import sage" somewhere. Those I agree are
Python packages (that depend on sage).

Best,
Erik

[Erik Bray]

On Mon, May 2, 2016 at 2:52 PM, Erik Bray <erik....@gmail.com> wrote:
> On Wed, Apr 27, 2016 at 5:05 PM, William Stein <wst...@gmail.com> wrote:
>> On Wed, Apr 27, 2016 at 7:09 AM, Kwankyu Lee <ekwa...@gmail.com> wrote:
>>> This discussion is hardening the terms: Sage core and external packages. But
>>> from the point of view of the people developing the would-be external
>>> packages, the official term would better be
>>>
>>> Sage extension,
>>> Sage library extension,
>>> Sage library extension package,
>>> or SLEP.
>
> Two easily confused with "extension module" in the sense of a Python
> module written in C.

* Too, not "Two"; I already had the next paragraph buffered somewhere
but there was a stray memory copy error, possibly caused by a cosmic
ray :)