how we develop sage

[William Stein]

Hi,

This was a comment I just put on trac #965: "I would make a completely
separate python package, maybe called pysmalljac, which builds
smalljac and makes it usable from Python. It would be on github and
pypi. That's how most Sage development should be done. What a
monster I've created by following the Magma way of doing things
instead of the standard open source best practices..."

I am recommending to absolutely everybody I talk with about Sage
development that we switch from our current massive monolithic
centralized approach toward standard open source practices. Namely,
lots of smaller libraries, standard open source practices, etc. It
would be really valuable to have a thread on sage-devel about how to
more systematically support this.

Some thoughts:

- For now, work can be done that is valuable but doesn't have to
impact the current sage/trac workflow. For example, somebody might
create an awesome Python package that does BLAH and depends on the
core Sage library (what you get via "import sage" now).

- There are 77989 examples of Python packages at https://pypi.python.org/pypi

- This is pretty useful to read:
https://python-packaging.readthedocs.org/en/latest/

- In the beginning I put everything into one library, since it was
small. Then it started getting bigger and in 2006, 2007, etc. we
talked a lot about splitting up Sage, having smaller packages, etc.
There was a lot of momentum and it was pretty clear to all the CS-type
people involved in Sage (who knew about programming from more than
just Sage) that this is what we needed to do. People like Martin
Albrecht and David Harvey. Then those people all got jobs and
couldn't work on infrastructure aspects of Sage... and nothing
happened in this direction.

- PSage was an attempt at something like this where I could work
out how to build Cython code in parallel, etc., etc. using more modern
packaging. That was in about 2011, and then I stopped to work on SMC
fulltime for several years, hence work on psage stopped, since I can
only do one thing at a time.

-- William

--
William (http://wstein.org)

[William Stein]

From Drew: "+1. At the LMFDB workshop in Bristol last week we ended
up giving a lot of people accounts on the main LMFDB machine in
Warwick simply so they wouldn't have to go through the time-consuming
(and somewhat error prone) process of building Sage 7.1 on their
laptops and installing the additional packages that LMFDB requires,
even though in some cases this meant teaching people how to use vi so
they could edit files remotely when they needed to (!).

BTW, there will be a new release of smalljac in the not-too-distant
future that fully supports genus 3 curves (and does not require you to
hardwire the genus at compile time -- BTW, contrary to what it says on
trac server, the current version of smalljac (and this has been true
for at least 5 years) handles genus 1 and 2 simultaneously out of the
box, there is no need to recompile.

Drew"

--
William (http://wstein.org)

[Volker Braun]

On Tuesday, April 5, 2016 at 8:44:45 PM UTC+2, William wrote:

[...] toward standard open source practices.

You mean like in the Linux kernel, which uses a single monolithic git repository?

Really, modularization is not a useful goal in of itself. And it comes with its own sets of issues, see the left-pad fiasco last week when the nodejs clown boat caught fire.

You can do "import sage" right now, it works. The question is, what do you expect it to do. The basic dichotomy is

* either this imports something like the "undergrad curriculum", all kinds of rings and groups and plotting and ... (we are basically here, and this necessarily has a longish list of dependencies).

* or this imports some piece of Sage (integers?) that is super-fast to compile. But you want elliptic curves? Never heard of that, maybe if you install the sage-elliptic-curves (or maybe the competing and incompatible sage-riemann-surface). 

As long as the goal of "import sage" is to give you something like the feature set of Sage right now we don't benefit from modularization. That is just a tautology, the goal is just not a modular one. We'd just shoot ourselves in the foot if we split things into multiple interdependent packages that then must be upgraded in lockstep. 

It would still be useful to straighten out the Sage startup circular import game, have some core feature set (~= sage.rings) that must be importable (and usable) by itself. This could be shipped as a separate package even if its not a separate repository. It just requires tooling to keep dependencies straight. And as I mentioned before, Python3 has better namespace package support...

[William Stein]

On Wed, Apr 6, 2016 at 12:51 AM, Volker Braun <vbrau...@gmail.com> wrote:
> On Tuesday, April 5, 2016 at 8:44:45 PM UTC+2, William wrote:
>>
>> [...] toward standard open source practices.
>
>
> You mean like in the Linux kernel, which uses a single monolithic git
> repository?

I think you are being sarcastic. It's hard to tell on the internet.

> Really, modularization is not a useful goal in of itself. And it comes with
> its own sets of issues, see the left-pad fiasco last week when the nodejs
> clown boat caught fire.

There are very good reasons to supporting both modularization and much
more standard approaches to packaging. It's unbelievable to me that
somebody as technically smart as you and knowledgeable about sage and
software doesn't see this as obvious. Since -- based on responses --
almost nobody else in our community seems to get this either, I'll
just forge on and come up with the future approach to sage development
myself (even though I don't want to).

William

>
> You can do "import sage" right now, it works. The question is, what do you
> expect it to do. The basic dichotomy is
>
> * either this imports something like the "undergrad curriculum", all kinds
> of rings and groups and plotting and ... (we are basically here, and this
> necessarily has a longish list of dependencies).
>
> * or this imports some piece of Sage (integers?) that is super-fast to
> compile. But you want elliptic curves? Never heard of that, maybe if you
> install the sage-elliptic-curves (or maybe the competing and incompatible
> sage-riemann-surface).
>
> As long as the goal of "import sage" is to give you something like the
> feature set of Sage right now we don't benefit from modularization. That is
> just a tautology, the goal is just not a modular one. We'd just shoot
> ourselves in the foot if we split things into multiple interdependent
> packages that then must be upgraded in lockstep.
>
> It would still be useful to straighten out the Sage startup circular import
> game, have some core feature set (~= sage.rings) that must be importable
> (and usable) by itself. This could be shipped as a separate package even if
> its not a separate repository. It just requires tooling to keep dependencies
> straight. And as I mentioned before, Python3 has better namespace package
> support...
>

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--
William (http://wstein.org)

[Nicolas M. Thiery]

On Thu, Apr 07, 2016 at 03:19:14PM -0700, William Stein wrote:
> There are very good reasons to supporting both modularization and much

> more standard approaches to packaging. ... Since -- based on responses --

> almost nobody else in our community seems to get this either,

For the record, modularity is a major topic of the Sage Days we are
having right now (although the focus has been more about modularity at
the granularity of the Sage library w.r.t its dependencies than within
the Sage library):

https://wiki.sagemath.org/days77

Cheers,
Nicolas
--
Nicolas M. Thiéry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

[William Stein]

On Thu, Apr 7, 2016 at 3:41 PM, Nicolas M. Thiery
<Nicolas...@u-psud.fr> wrote:
> On Thu, Apr 07, 2016 at 03:19:14PM -0700, William Stein wrote:
>> There are very good reasons to supporting both modularization and much
>> more standard approaches to packaging. ... Since -- based on responses --
>> almost nobody else in our community seems to get this either,
>
> For the record, modularity is a major topic of the Sage Days we are
> having right now (although the focus has been more about modularity at
> the granularity of the Sage library w.r.t its dependencies than within
> the Sage library):
>
> https://wiki.sagemath.org/days77

Cool! If there's any summary of discussions or questions, etc., let me know.

I think a very useful sprint project for such a workshop would be to
make a Sage-friendly version of this:

https://python-packaging.readthedocs.org/en/latest/minimal.html

Some issues are:

- How to doctest code with "sage:" prompts that isn't in the sage
library (basically, you have to be sure to do "sage -t --force-lib
<package name>" and explicitly import from the Python library you're
making in the docstrings.

- Cython: relying on pxi/pxd files from the sage library.

- Discussion about how to import functionality from the Sage
library into a library.

There could be a package (hosted on pip) called "sagemath" that other
packages depend on. It could provide some helper functionality for
the above.

I make no claims to have thought anything through yet! I did
however spend an hour playing around with this problem yesterday,
which resulted in

https://github.com/williamstein/sage_modabvar

https://pypi.python.org/pypi/sage_modabvar/0.1

which is far from done. I just did a very tiny bit of initial work
on removing this "big" pure-Python modular abelian varieties package
from Sage and making it a standalone module.

Thanks,

William

>
> Cheers,
> Nicolas
> --
> Nicolas M. Thiéry "Isil" <nth...@users.sf.net>
> http://Nicolas.Thiery.name/
>

[Volker Braun]

On Friday, April 8, 2016 at 12:19:57 AM UTC+2, William wrote:

> You mean like in the Linux kernel, which uses a single monolithic git
> repository?
I think you are being sarcastic.

I'm only partially kidding. Why is the kernel not a collection of packages? Because nobody wants half a kernel, and because the kernel developers don't want to make promises about internal APIs. Similarly, I want a car but please leave out the engine and two of the four wheels said nobody ever. 

 

There are very good reasons to supporting both modularization

And there are reasons against as well. Its a tool, and with every tool there are pros and cons. The old adage about the right tool for the right job still holds. 

The only thing that I'm convinced is that going around and claiming that modularization is the solution of all our problems (without making a concrete proposal about what and how) is not productive. I agree that our current dependency installation system sucks (but are we even talking about that or do you mean the Sage library).

You seem to think that pypi/whl is a suitable venue for binary distribution. As I mentioned before, the author of whl doesn't think so for what would be our use case. Of course its trivial to publish a pure Python package on pypi. So a sagemath pypi package that just assumes that you have a dozen obscure math libraries (+headers) on you system to cythonize against? Is that what this thread is even about?

[Dima Pasechnik]

On Thursday, April 7, 2016 at 11:19:57 PM UTC+1, William wrote:

On Wed, Apr 6, 2016 at 12:51 AM, Volker Braun <vbrau...@gmail.com> wrote:
> On Tuesday, April 5, 2016 at 8:44:45 PM UTC+2, William wrote:
>>
>> [...] toward standard open source practices.
>
>
> You mean like in the Linux kernel, which uses a single monolithic git
> repository?

I think you are being sarcastic.  It's hard to tell on the internet.

> Really, modularization is not a useful goal in of itself. And it comes with
> its own sets of issues, see the left-pad fiasco last week when the nodejs
> clown boat caught fire.

There are very good reasons to supporting both modularization and much
more standard approaches to packaging.  It's unbelievable to me that
somebody as technically smart as you and knowledgeable about sage and
software doesn't see this as obvious.  Since -- based on responses --
almost nobody else in our community seems to get this either, I'll 
just forge on and come up with the future approach to sage development
myself (even though I don't want to). 

while better packaging would be great, and ability not to install (or uninstall) parts that one never uses

(e.g., as far as I am concerned, I don't need finances/, games/), I fear that "full modularisation" efforts in case of "core" Sage are akin to surgery attempts to separate siamese twins. 

I cannot imagine separating groups, NT, algebra, linear algebra, symbolics, calculus, combinatorics/graph theory; 

they are too much interdependent for this.

IMHO things like improving the build system are more urgent (and improving the build system

would lessen the feel that Sage is too big...)

[William Stein]

This thread is first and foremost about reducing the friction involved in making code that depends on the Sage distribution available to the world.    Based on feedback I get from users, this friction is a very, very significant problem to the growth of Sage. 

There are projects with healthy package ecosystems.  Examples include: tex, R, and Python.

Often people write code for sage that sits in some branch on trac used by almost nobody for years and years.  Just touching or working on that code involves way too much friction.  In many cases that code would be much better placed in its own separate Python library.  When it is fully up to snuff, we could make that Python library a standard part of the sage distribution of we want.     

Regarding your car analogy, sometimes it is far better if we can factor of some pieces of sage as separate libraries that we use, but which are also used outside of sage.  For example, cysignals (I think from Malb).  The broader community benefits and we also benefit since more people care about the code and work on it.  It's a win-win.   Remember that Cython used to be part of Sage and be called SageX... Making it separate was good for everybody. 

  - William

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--

Sent from my massive iPhone 6 plus.

[Nicolas M. Thiery]

On Thu, Apr 07, 2016 at 03:58:43PM -0700, William Stein wrote:
> Cool! If there's any summary of discussions or questions, etc., let me know.

We are taking notes in a couple of pads, including:

https://www.lri.fr/etherpad/p/sage-days77-packaging

This will end up after the Sage Days on the wiki or something similar
(need to check whether there already is a page for this).

[Volker Braun]

On Friday, April 8, 2016 at 1:26:07 AM UTC+2, William wrote:

This thread is first and foremost about reducing the friction involved in making code that depends on the Sage distribution available to the world.

Whats wrong with the obvious solution: make it a Python package (basically add setup.py) and then "sage -pip install https://github.com/vbraun/awesomepackage.git". Clearly we could have more documentation for how to write the setup.py or some package directory service. But really it is already a one-liner.

Regarding your car analogy, sometimes it is far better if we can factor of some pieces of sage as separate libraries that we use, but which are also used outside of sage.  For example, cysignals (I think from Malb).  The broader community benefits and we also benefit since more people care about the code and work on it.  It's a win-win.

Yes but cython and cysignals don't do anything for math directly, they are pure dependencies. By contrast, lets say we break out all elliptic curve stuff into a separate package. Just like with cysignals, this will remove all elliptic curve documentation from the Sage reference manual. Still a win-win? 

 

  Remember that Cython used to be part of Sage and be called SageX... Making it separate was good for everybody. 

  - William

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscribe@googlegroups.com.

To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[Volker Braun]

Correction: "sage -pip install git+https://github.com/vbraun/awesomepackage.git"

[William Stein]

On Thursday, April 7, 2016, Volker Braun <vbrau...@gmail.com> wrote:

On Friday, April 8, 2016 at 1:26:07 AM UTC+2, William wrote:

This thread is first and foremost about reducing the friction involved in making code that depends on the Sage distribution available to the world.

Whats wrong with the obvious solution: make it a Python package (basically add setup.py) and then "sage -pip install https://github.com/vbraun/awesomepackage.git". Clearly we could have more documentation for how to write the setup.py or some package directory service. But really it is already a one-liner.

 That is exactly what I'm proposing.  However...

 

Regarding your car analogy, sometimes it is far better if we can factor of some pieces of sage as separate libraries that we use, but which are also used outside of sage.  For example, cysignals (I think from Malb).  The broader community benefits and we also benefit since more people care about the code and work on it.  It's a win-win.

Yes but cython and cysignals don't do anything for math directly, they are pure dependencies. By contrast, lets say we break out all elliptic curve stuff into a separate package. Just like with cysignals, this will remove all elliptic curve documentation from the Sage reference manual. Still a win-win? 

1. Not necessarily true.

2. Is it so bad that the Cython docs aren't in the sage reference manual?  

The sage reference manual is way too big already.  Building it is a high source of friction and complaints by users.

 

 

  Remember that Cython used to be part of Sage and be called SageX... Making it separate was good for everybody. 

  - William

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--
Sent from my massive iPhone 6 plus.

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[Florent Hivert]

Hi,

On Thu, Apr 07, 2016 at 05:36:21PM -0700, William Stein wrote:
> The sage reference manual is way too big already. Building it is a
> high source of friction and complaints by users.

It's worth (and also fun) to point out there that the second major topic at
Sage Days 77 here after packaging and modularization et Sphinx and the
compilation of the doc. We are still at cleaning the mess before going
forward, but some experiment show that there is a good hope to have a better
doc build system.

Cheers,

Florent

[Kwankyu Lee]

It would be nice if

(1) Sage library consists of core + optional parts

(2) An optional part is dependent on core + other optional parts

(3) Optional parts to install can be selected at build time.

(4) An optional part has its own documentation, but accessible through a single interface in Sage.

For example, people may develop an optional part for function fields in a separate (GitHub) repository and discuss on a separate (Google) group.

Perhaps this functionality can be provided by just "sage -pip install ..." as suggested above.

[Francois Bissey]

> On 8/04/2016, at 14:13, Kwankyu Lee <ekwa...@gmail.com> wrote:
>
> (3) Optional parts to install can be selected at build time.

Or Optional parts can be installed separately after installing the core
part. Very pythonic that way.

François

[Volker Braun]

On Friday, April 8, 2016 at 2:36:23 AM UTC+2, William wrote:

Whats wrong with the obvious solution: make it a Python package (basically add setup.py) and then "sage -pip install https://github.com/vbraun/awesomepackage.git". Clearly we could have more documentation for how to write the setup.py or some package directory service. But really it is already a one-liner.

 That is exactly what I'm proposing

But then there is nothing to do on the Sage side, this already works and is totally standard. Documentation for how to package your own Python code can easily be found online.

 

Yes but cython and cysignals don't do anything for math directly, they are pure dependencies. By contrast, lets say we break out all elliptic curve stuff into a separate package. Just like with cysignals, this will remove all elliptic curve documentation from the Sage reference manual. Still a win-win? 

1. Not necessarily true.

It is if you want to have a uniform document with cross references working. Much of the memory usage of the current docbuild is the huge intersphinx data, and no amount of modularization would make that smaller. That is not to say that docbuilding can't be improved, but certainly not by spreading out the documentation sources.

 

2. Is it so bad that the Cython docs aren't in the sage reference manual?

As I said before,  cython and cysignals don't do anything for math directly so they are a special case. In fact, they shouldn't be in the main Sage docs even if they were part of Sage.

 

[Jeroen Demeyer]

On 2016-04-08 00:58, William Stein wrote:
> - Cython: relying on pxi/pxd files from the sage library.

Please push for https://github.com/cython/cython/pull/483 if you care
about this. Cython upstream doesn't seem to "get" that we need this patch.

[Sébastien Labbé]

On Tuesday, April 5, 2016 at 8:44:45 PM UTC+2, William wrote:

Hi,

This was a comment I just put on trac #965: "I would make a completely
separate python package, maybe called pysmalljac, which builds
smalljac and makes it usable from Python.  It would be on github and
pypi.  That's how most Sage development should be done.  What a
monster I've created by following the Magma way of doing things
instead of the standard open source best practices..."

I am recommending to absolutely everybody I talk with about Sage
development that we switch from our current massive monolithic
centralized approach toward standard open source practices.  Namely,
lots of smaller libraries, standard open source practices, etc.   It
would be really valuable to have a thread on sage-devel about how to
more systematically support this.

Some thoughts:

  - For now, work can be done that is valuable but doesn't have to
impact the current sage/trac workflow.  For example, somebody might
create an awesome Python package that does BLAH and depends on the
core Sage library (what you get via "import sage" now).

  - There are 77989 examples of Python packages at https://pypi.python.org/pypi

  - This is pretty useful to read:
https://python-packaging.readthedocs.org/en/latest/

Jeroen suggests also this link which seems to him more up to date:

https://packaging.python.org/en/latest/distributing/
 

[Luca De Feo]

> This thread is first and foremost about reducing the friction involved in making code that depends on the Sage distribution available to the world.

> Based on feedback I get from users, this friction is a very, very significant problem to the growth of Sage.

Let me recall something that comes up often, but that I haven't seen yet in this thread nor at the Sage days. Just two weeks ago I was said: "We are going to make our student write code in Magma, because we want something future proof. We've used Sage a couple of times before, but each time Sage updates break our code in <1yr, and it takes too much work to know why and update our code".

Let me stress that these are highly skilled researchers who develop amazing software and have a very deep knowledge about programming. One of them develops a library that is a core piece of infrastructure for Magma, and which is responsible for a good fraction of the benchmarks where Magma crushes Sage. If we could make these people stick to Sage, I can only imagine benefits for us.

I have this experience myself, though, being more involved in Sage, I make the effort to patch my code to work with the latest stable. At some points in time, it has even been impossible to have code that works both with the latest stable and the latest beta (and/or the version in SMC). This is something that should only happen at major version bumps, if ever.

Of course, this is not easy to solve, but having a more modular distribution (at least for some meaning of "modular") might help.

--
Luca

[Jeroen Demeyer]

On 2016-04-08 13:26, Luca De Feo wrote:
> This is something that should only happen at major version bumps, if ever.

That's easily solved: the next stable release of Sage should be Sage 8,
then Sage 9 and so on...

We do have a 1-year deprecation policy, but of course it's not enforced
(and I wouldn't know how to enforce that). If the author and reviewer of
a ticket don't care about deprecation, then stuff will break.

[Dima Pasechnik]

On Friday, April 8, 2016 at 12:26:22 PM UTC+1, Luca De Feo wrote:

> This thread is first and foremost about reducing the friction involved in making code that depends on the Sage distribution available to the world.
> Based on feedback I get from users, this friction is a very, very significant problem to the growth of Sage.

Let me recall something that comes up often, but that I haven't seen yet in this thread nor at the Sage days. Just two weeks ago I was said: "We are going to make our student write code in Magma, because we want something future proof. We've used Sage a couple of times before, but each time Sage updates break our code in <1yr, and it takes too much work to know why and update our code".

Let me stress that these are highly skilled researchers who develop amazing software and have a very deep knowledge about programming. One of them develops a library that is a core piece of infrastructure for Magma, and which is responsible for a good fraction of the benchmarks where Magma crushes Sage. If we could make these people stick to Sage, I can only imagine benefits for us.

mind you, Magma group invites such people for paid months-long visits, where they work on implementing these algorithms in close contact with Magma core devs.

If only Sage could do this too... :-)

 

I have this experience myself, though, being more involved in Sage, I make the effort to patch my code to work with the latest stable. At some points in time, it has even been impossible to have code that works both with the latest stable and the latest beta (and/or the version in SMC). This is something that should only happen at major version bumps, if ever.

Of course, this is not easy to solve, but having a more modular distribution (at least for some meaning of "modular") might help.

or it might make things worse, as the functionality becomes a function of a multitude of versions of different parts...

But there is a way to make code work in Sage as upgrades happen - making it a part of Sage :-)

Thus I don't buy this "future-proof" statement (well, of course, Sage may fold if sufficient number of right people get hit by a bus - but the same is true for Magma)

 

--
Luca

[Simon King]

Hi Luca,

On 2016-04-08, Luca De Feo <de...@lix.polytechnique.fr> wrote:
> Let me recall something that comes up often, but that I haven't seen yet in
> this thread nor at the Sage days. Just two weeks ago I was said: "We are
> going to make our student write code in Magma, because we want something
> future proof. We've used Sage a couple of times before, but each time Sage
> updates break our code in <1yr, and it takes too much work to know why and
> update our code".

+1

I was seriously fed up when my research software was broken by backwards
incompatible ostensibly "internal" changes --- repeatedly within one year---,
which hempered my work on a funded project to the extent that I couldn't
successfully finish.

I considered quitting Sage for that reason. Mainly I didn't quit because
in the *past* Sage used to be an ideal platform for my purposes, and while it
was moving forward it didn't break too much of existing code --- thus,
maintanance of my package and persistence of data was much less of a
hastle than it has been in the past one or two years. Since it was OK
till perhaps two years ago, I still hope Sage will be able to revive its
strength.

> I have this experience myself, though, being more involved in Sage, I make
> the effort to patch my code to work with the latest stable. At some points
> in time, it has even been impossible to have code that works both with the
> latest stable and the latest beta (and/or the version in SMC). This is
> something that should only happen at major version bumps, if ever.

+1

> Of course, this is not easy to solve, but having a more modular
> distribution (at least for some meaning of "modular") might help.

-1.

My impression was that what I perceive as "decay" of Sage was caused by
changing Sage's development model. Changing the whole architecture would
create even more of an overhead, or rather a swamp in which Sage would
disappear.

Sage should build the car and not re-invent the wheel --- and much less
Sage should re-invent Sage.

Best regards,
Simon

[Erik Bray]

On Tue, Apr 5, 2016 at 8:44 PM, William Stein <wst...@gmail.com> wrote:
> Hi,
>
> This was a comment I just put on trac #965: "I would make a completely
> separate python package, maybe called pysmalljac, which builds
> smalljac and makes it usable from Python. It would be on github and
> pypi. That's how most Sage development should be done. What a
> monster I've created by following the Magma way of doing things
> instead of the standard open source best practices..."
>
> I am recommending to absolutely everybody I talk with about Sage
> development that we switch from our current massive monolithic
> centralized approach toward standard open source practices. Namely,
> lots of smaller libraries, standard open source practices, etc. It
> would be really valuable to have a thread on sage-devel about how to
> more systematically support this.

Sorry for the vapid response for now, but +1 to this principle.

[Simon King]

Hi Jeroen,

Or rather if the release manager believes that a change is "internal"
and doesn't need deprecation, then stuff will break.

Cheers,
Simon

[Johan S. R. Nielsen]

> But there is a way to make code work in Sage as upgrades happen - making it
> a part of Sage :-)
> Thus I don't buy this "future-proof" statement (well, of course, Sage may
> fold if sufficient number of right people get hit by a bus - but the same
> is true for Magma)

It has something to do with the friction that William mentioned, though:
researchers barely has the time and motivation to implement their
algorithms, but they don't want to afterwards go through the - for
new-comers - rather arduous trac/ticket/reviewing process. One reason is
probably that they don't want to mature the doc and API of their code
sufficiently for it to enter sage-proper. I used to maintain a coding
theory library on bitbucket for exactly this reason.

Their work could be more available if Sage wasn't so monolithic, or at
least, if there was more of a culture of having small Sage packages
floating around. If you want their not-completely-nicely-documented
code, install their package into your Sage and be at your own risk.

What Luca is talking about poses a problem for that strategy though:
their code will work with only a few versions of Sage before breaking.
Note that I don't personally share this experience: in my own coding
theory library for Sage, no function was (AFAIK) broken in the last 5
years due to a Sage update (except for my own bugs, and some stuff
related to non-standard packaging).

But of course: if research developers 1) don't care enough about their
own code to polish it enough for going through our review process, and
2) don't care enough to fix it when incompatibilities with Sage occurs;
then is it really code we should be caring about? Won't it quickly rot
on github anyway? The research code going into Magma is exactly polished
during those visits to Magma HQ that Dima mentions.

Best,
Johan

[Erik Bray]

On Wed, Apr 6, 2016 at 9:51 AM, Volker Braun <vbrau...@gmail.com> wrote:
> On Tuesday, April 5, 2016 at 8:44:45 PM UTC+2, William wrote:
>>
>> [...] toward standard open source practices.
>
>
> You mean like in the Linux kernel, which uses a single monolithic git
> repository?

No piece of user-mode software is so special that it should look to
the Linux kernel as a good project to emulate.

> Really, modularization is not a useful goal in of itself. And it comes with
> its own sets of issues, see the left-pad fiasco last week when the nodejs
> clown boat caught fire.

That kind of issue is not so common though which is why it was such a
fiasco. If this happened all the time you wouldn't even hear about it.

[As an aside I wouldn't characterize an entire developer community as
a 'clown boat']

[Luca De Feo]

> if research developers 1) don't care enough about their own code to polish it enough for going through our review process

I don't agree with Dima that putting their code into Sage is what researchers should do, no matter what.

I do care a lot about my research code, and I know some of it has no place inside Sage. Maintaining it outside of Sage is not just carelessness or laziness, it's a design choice.

Luca

[Simon King]

Hi Dima,

On 2016-04-08, Dima Pasechnik <dim...@gmail.com> wrote:
> mind you, Magma group invites such people for paid months-long visits,
> where they work on implementing these algorithms in close contact with
> Magma core devs.
> If only Sage could do this too... :-)

Sage has SageDays, and Sage has a rather responsive community. I recall
that in the past, the only serious issue in cooperative projects in Sage
was the time difference between Europe/Africa, Americas and Asia. But
that could be solved by staying up late.

So, I think it *is* possible to implement algorithms in close contact
with Sage core devs.

> or it might make things worse, as the functionality becomes a function of a
> multitude of versions of different parts...

+1

And *making* it modular in the first place or even just *coping* with
modularisation is something that doesn't lie within the core competences
of (most) Sage developers, IMHO.

> But there is a way to make code work in Sage as upgrades happen - making it
> a part of Sage :-)

Indeed that's the approach that I now take with my optional group
cohomology spkg. Thereby making it modular, just to give that concept a try.
Splitting off one third-party component, upgrading to the current
upstream version; rebasing a second third party component on top of the
upgraded first third party component (that's the main obstacle); and
eventually rebasing my own code on top of it, putting it into the Sage
library rather than keeping it in one spkg together with the other two
components.

But it costs me resources that I'd prefer to spend in a more productive
way --- and *nothing* came out of it yet! The attempt to modularise and
upgrade one upstream source to cooperate with a more recent version of
another upstream source takes longer than creating the original version of
the spkg!

> Thus I don't buy this "future-proof" statement

+1

> (well, of course, Sage may
> fold if sufficient number of right people get hit by a bus - but the same
> is true for Magma)

Perhaps that's the hidden purpose of SageDays. Let's make a conspiracy
theory out of it ;-)

Cheers,
Simon

[Erik Bray]

On Tue, Apr 5, 2016 at 8:44 PM, William Stein <wst...@gmail.com> wrote:

> Some thoughts:
>
> - For now, work can be done that is valuable but doesn't have to
> impact the current sage/trac workflow. For example, somebody might
> create an awesome Python package that does BLAH and depends on the
> core Sage library (what you get via "import sage" now).

FWIW in the Astropy project we have a concept called affiliated
packages [1]. The idea behind this is that if someone says "Hey, I
need <relatively specialized> functionality for my work, can you add
that to Astropy?" And we say, "No, because none of the Astropy core
developers understand this subject well enough to maintain the code,
and it's perhaps a little too specialized. But you could implement it
as an affiliated package."

We require for affiliated packages that they depend on Astropy and use
Astropy's core features, such as common data structures, file format
handling, time and coordinate transformations, etc. etc. This ensures
that all affiliated packages interoperate well with each other through
the core Astropy features (in analogy to sage these might be common
representations of various core mathematical objects).

The benefits of being an "astropy affiliated package", in addition to
promising a certain level of interface commonality as described above,
also include being advertised through Astropy. We're also working on
the ability to better discover the functionality provided by different
affiliated packages such as a cross-documentation search. That's
still an open issue though.

And sometimes affiliated packages may be merged into the Astropy core
package as well. This has't been done often yet, but it has been done
and will be done again. For example the "gwcs" package is intended to
be an Astropy core package, but it's still very experimental and
volatile so it's better to develop it as a separate project for now,
rather that merge it directly into the Astropy core and then be left
being basically unable to make stable releases until the "gwcs"
feature is ready (or being forced to make releases with big warning
flags around incomplete features).

I thinks a model similar to this could work well for sage, and also
free its core developers from having to maintain huge piles of already
poorly-supported code (some "affiliated package" projects die out,
but it has no impact on the Astropy core).

Erik

[1] http://www.astropy.org/affiliated/

[Johan S. R. Nielsen]

> I don't agree with Dima that putting their code into Sage is what
> researchers should do, no matter what.
>
> I do care a lot about my research code, and I know some of it has no place
> inside Sage. Maintaining it outside of Sage is not just carelessness or
> laziness, it's a design choice.

For sure - and you were not in the group of people I meant, since you
clearly stated you cared enough to rebase your code.

Changes to Sage so that external packages like yours break is
unfortunate. But I'm questioning how much we should shape our
development methodology around research projects whose developers
*don't* care enough to fix their code when it breaks. And it will break
once in a while, as long as it is not part of Sage proper.

How often do you feel that changes in Sage breaks your code?

Best,
Johan

[Simon King]

Hi Johan,

On 2016-04-08, Johan S R Nielsen <santa...@gmail.com> wrote:
> How often do you feel that changes in Sage breaks your code?

As I stated before, several times in one year (that year being 2013 or 2014,
I am not sure), after it was more or less stable since 2009.

Cheers,
Simon

[Dima Pasechnik]

On Friday, April 8, 2016 at 1:45:00 PM UTC+1, Luca De Feo wrote:

> if research developers 1) don't care enough about their own code to polish it enough for going through our review process

I don't agree with Dima that putting their code into Sage is what researchers should do, no matter what.

 

I merely said that this appears to work. I never said that everything should go there.

On the other hand, you mention a core Magma library that helps it beat Sage on benchmarks, I imagine such a library would be a nice fit into core Sage.

 

[Luca De Feo]

> How often do you feel that changes in Sage breaks your code?

Looking at my log (this project https://github.com/defeo/ss-isogeny-software)

- Sage 6.2: module renamed in Pari/GP interface (without deprecation, as far as I recall)
- Sage 6.4: frankly, I don't know, but possibly a change in the finite fields implementation
- Sage 6.7: changes in cython, not really Sage's fault, I guess
- Sage 6.7: changes in the finite fields implementation (with proper deprecation, this time)
- Sage 6.10: changes in richcmp (did not get deprecation warnings, but the change concerns an internal function)
- Sage 7.1: affected by #19941 (properly deprecated, note that 7.1 is stable right now, but not yet in SMC)

It may be argued that I'm taking risks because I sometimes fiddle with "the internals" of Sage, but one other problem of Sage is that it does not define clearly what's the public API and what's internal.

Luca

[William Stein]

+1 -- this "one other problem of Sage is that it does not define
clearly what's the public API and what's internal." is a HUGE problem
-- and we're not aware of it because "user code" is typically private.

If there were a large body of pip-installable packages, which are user
code, this would help *define* what the public API of Sage really is,
and also give us a much larger body of code to test against before
making new releases.

As an example, when working on

https://github.com/williamstein/sage_modabvar

the first thing I noticed in that from sage-6.10 to sage-7.x

import sage.rings.arith

completely BROKE. It became "import sage.arith".

This is (who knows?) a change in the public api of the core sage
library. Maybe it's a gratituous one, and in this case one I'm
puzzled by why this change was made. (I wrote sage.rings.arith in the
first place, and it was meant to be some misc arith needed to
implement rings. Why move it to its own top level module like that?)

By writing Python packages that depend on "the core sage library" we
will be forced to define what the core library is and stabilize it.
If you guys want Sage to survive/thrive, this **MUST HAPPEN**. The
core sage library should -- like with Python -- be the place where
code, which has been tested/used by people and got stable, goes to
stay stable (at least API-wise). Like the standard Python library.

William

>
> Luca

>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--

William (http://wstein.org)

[Luca De Feo]

> I merely said that this appears to work. I never said that everything should go there.

Sorry for misunderstanding you.

> On the other hand, you mention a core Magma library that helps it beat Sage on benchmarks, I imagine such a library would be a nice fit into core Sage.

 
It would be lovely, but unfortunately the licence forbids us to distribute it :( But we're drifting away from the subject.

[William Stein]

On Fri, Apr 8, 2016 at 1:00 AM, Volker Braun <vbrau...@gmail.com> wrote:
> On Friday, April 8, 2016 at 2:36:23 AM UTC+2, William wrote:
>>>
>>> Whats wrong with the obvious solution: make it a Python package
>>> (basically add setup.py) and then "sage -pip install
>>> https://github.com/vbraun/awesomepackage.git". Clearly we could have more
>>> documentation for how to write the setup.py or some package directory
>>> service. But really it is already a one-liner.
>>
>>
>> That is exactly what I'm proposing
>
>
> But then there is nothing to do on the Sage side, this already works and is
> totally standard. Documentation for how to package your own Python code can
> easily be found online.

(Volker, I really hate to disagree with you, since I respect you
*very* highly.) However, this is where you are wrong. Just because
it is technically possible to do something and documented how to do so
online, doesn't mean there is nothing to do on the sage side.
Software, Sage, etc. is all about people, what they endorse, what
examples they set, etc. It's a leadership thing. The very thing that
makes you say "there is nothing to do" is what tells me "there is a
lot to do".

>
>
>>>
>>> Yes but cython and cysignals don't do anything for math directly, they
>>> are pure dependencies. By contrast, lets say we break out all elliptic curve
>>> stuff into a separate package. Just like with cysignals, this will remove
>>> all elliptic curve documentation from the Sage reference manual. Still a
>>> win-win?
>>
>> 1. Not necessarily true.
>
>
> It is if you want to have a uniform document with cross references working.

I continue to disagree.

> Much of the memory usage of the current docbuild is the huge intersphinx
> data, and no amount of modularization would make that smaller.

I disagree. If the elliptic curves docs aren't in Sage at all (say),
then it certainly
isn't going to make that stuff larger.

> That is not
> to say that docbuilding can't be improved, but certainly not by spreading
> out the documentation sources.
>
>>
>> 2. Is it so bad that the Cython docs aren't in the sage reference manual?
>
>
> As I said before, cython and cysignals don't do anything for math directly
> so they are a special case. In fact, they shouldn't be in the main Sage docs
> even if they were part of Sage.

And for 99.99% of sage users who don't know an elliptic curve from an
"elliptical curve" there is no gain by having the elliptic curves in
the same daunting reference manual as plotting functions.

>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--

William (http://wstein.org)

[Jeroen Demeyer]

On 2016-04-08 18:43, William Stein wrote:
> the first thing I noticed in that from sage-6.10 to sage-7.x
>
> import sage.rings.arith
>
> completely BROKE.

That's totally exaggerated.

It didn't completely BREAK, it still works with a nice deprecation warning:

Importing is_prime from here is deprecated. If you need to use it,
please import it directly from sage.arith.all
See http://trac.sagemath.org/19879 for details.

Saying that stuff BREAKS when people take the effort to make proper
deprecation is not encouraging those people to care about deprecations.

(disclaimer: I made that change)

[Volker Braun]

On Friday, April 8, 2016 at 6:50:19 PM UTC+2, William wrote:

> But then there is nothing to do on the Sage side, this already works and is
> totally standard. 

Just because

it is technically possible to do something and documented how to do so
online, doesn't mean there is nothing to do on the sage side.

What I meant was: there is nothing that we need to implement to make it work. As I also mentioned, it would surely be good to have examples and documentation.

 

> Much of the memory usage of the current docbuild is the huge intersphinx
> data, and no amount of modularization would make that smaller.
I disagree.  If the elliptic curves docs aren't in Sage at all (say),
then it certainly
isn't going to make that stuff larger.

But not smaller either, unless you get rid of the intersphinx data. The reference manual is already broken up into smaller chunks that are processed independently. Breaking it up more is never going to give us a significant improvement. We can only make sphinx more efficient or build shittier docs. And there you are IMHO just throwing out the baby with the bathwater. Remove (or making inaccessible) large chunks of our documentation and/or disabling hyperlinks surely saves a lot of time and ram when building the documentation. But why? If you don't care about the documentation then just compile Sage without it in the first place ("make build"). 

[Volker Braun]

On Friday, April 8, 2016 at 6:43:46 PM UTC+2, William wrote:

this "one other problem of Sage is that it does not define
clearly what's the public API and what's internal.

IMHO thats just not true; What you get on the commandline (i.e. from sage.all import *) is public and the rest is not. If thats not enough (and really nobody ever asked) we could mark extra imports as public, e.g. by adding special sage.foo.public packages.

If there were large body of pip-installable packages, which are user

code, this would help *define* what the public API of Sage really is,
and also give us a much larger body of code to test against before
making new releases.

What API design school is that? You dump code on users and whoever manages to build the most convoluted contraption out of that will determine the future direction of the project ;-) Where is the leadership there? Who is going to handle the testing for each ticket, are you going to do that yourself? 

[William Stein]

On Friday, April 8, 2016, Volker Braun <vbrau...@gmail.com> wrote:

On Friday, April 8, 2016 at 6:43:46 PM UTC+2, William wrote:

this "one other problem of Sage is that it does not define
clearly what's the public API and what's internal.

IMHO thats just not true; What you get on the commandline (i.e. from sage.all import *) is public and the rest is not. If thats not enough (and really nobody ever asked) we could mark extra imports as public, e.g. by adding special sage.foo.public packages.

Why does nobody ever ask?

 

If there were large body of pip-installable packages, which are user
code, this would help *define* what the public API of Sage really is,
and also give us a much larger body of code to test against before
making new releases.

What API design school is that? You dump code on users and whoever manages to build the most convoluted contraption out of that will determine the future direction of the project ;-) Where is the leadership there? Who is going to handle the testing for each ticket, are you going to do that yourself? 

I don't care about design schools.   I would much rather be aware of how sage-dependent code is actually being used in the wild than to sit in school blissfully ignorant of how sage is really being used. 

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--

Sent from my massive iPhone 6 plus.

[Dima Pasechnik]

On Friday, April 8, 2016 at 7:40:02 PM UTC+1, William wrote:

On Friday, April 8, 2016, Volker Braun <vbrau...@gmail.com> wrote:

On Friday, April 8, 2016 at 6:43:46 PM UTC+2, William wrote:

this "one other problem of Sage is that it does not define
clearly what's the public API and what's internal.

IMHO thats just not true; What you get on the commandline (i.e. from sage.all import *) is public and the rest is not. If thats not enough (and really nobody ever asked) we could mark extra imports as public, e.g. by adding special sage.foo.public packages.

Why does nobody ever ask?

 

If there were large body of pip-installable packages, which are user
code, this would help *define* what the public API of Sage really is,
and also give us a much larger body of code to test against before
making new releases.

testing against sufficiently large body of code which is not maintained by a project is a perfect way to make

sure that no new releases are made by the project, ever.

 

What API design school is that? You dump code on users and whoever manages to build the most convoluted contraption out of that will determine the future direction of the project ;-) Where is the leadership there? Who is going to handle the testing for each ticket, are you going to do that yourself? 

I don't care about design schools.   I would much rather be aware of how sage-dependent code is actually being used in the wild than to sit in school blissfully ignorant of how sage is really being used. 

I thought that with SMC you have a near-perfect opportunity to see what Sage users use in the wild...

And perhaps, perhaps, the 1st thing would be to get a single-user SMC frontend available as a pip-installable package, so that sagenb can retire, at last?

Dima

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscribe@googlegroups.com.

To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[Volker Braun]

On Friday, April 8, 2016 at 8:40:02 PM UTC+2, William wrote:

Why does nobody ever ask?

I'd like to think thats because there is simply no need for obscure imports when using Sage; We have public constructors for basic mathematical objects, and computations are then done using methods of said objects. Unless you muck in the Sage internals (=you want to patch Sage) you *shouldn't* have to import stuff thats not already global on the commandline, how else are users supposed to use it?

What API design school is that? You dump code on users and whoever manages to build the most convoluted contraption out of that will determine the future direction of the project

I don't care about design schools

I disagree, IMHO designing interfaces carefully is much more important than (and should come before) banging on the keyboard.

 

[William Stein]

On Friday, April 8, 2016, Dima Pasechnik <dim...@gmail.com> wrote:

On Friday, April 8, 2016 at 7:40:02 PM UTC+1, William wrote:

On Friday, April 8, 2016, Volker Braun <vbrau...@gmail.com> wrote:

On Friday, April 8, 2016 at 6:43:46 PM UTC+2, William wrote:

this "one other problem of Sage is that it does not define
clearly what's the public API and what's internal.

IMHO thats just not true; What you get on the commandline (i.e. from sage.all import *) is public and the rest is not. If thats not enough (and really nobody ever asked) we could mark extra imports as public, e.g. by adding special sage.foo.public packages.

Why does nobody ever ask?

 

If there were large body of pip-installable packages, which are user
code, this would help *define* what the public API of Sage really is,
and also give us a much larger body of code to test against before
making new releases.

testing against sufficiently large body of code which is not maintained by a project is a perfect way to make

sure that no new releases are made by the project, ever.

 

False.  You are simply arguing for ignorance. How we chose to use the results of such testing is up to us to decide. 

 

 

What API design school is that? You dump code on users and whoever manages to build the most convoluted contraption out of that will determine the future direction of the project ;-) Where is the leadership there? Who is going to handle the testing for each ticket, are you going to do that yourself? 

I don't care about design schools.   I would much rather be aware of how sage-dependent code is actually being used in the wild than to sit in school blissfully ignorant of how sage is really being used. 

I thought that with SMC you have a near-perfect opportunity to see what Sage users use in the wild...

SMC does inform my frustration with the current limitations  on Sage development. 

 

And perhaps, perhaps, the 1st thing would be to get a single-user SMC frontend available as a pip-installable package, so that sagenb can retire, at last?

SMC is not a Python program.

William

 

Dima

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--
Sent from my massive iPhone 6 plus.

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[Dima Pasechnik]

On Saturday, April 9, 2016 at 12:17:21 AM UTC+1, William wrote:

On Friday, April 8, 2016, Dima Pasechnik <dim...@gmail.com> wrote:

On Friday, April 8, 2016 at 7:40:02 PM UTC+1, William wrote:

On Friday, April 8, 2016, Volker Braun <vbrau...@gmail.com> wrote:

On Friday, April 8, 2016 at 6:43:46 PM UTC+2, William wrote:

this "one other problem of Sage is that it does not define
clearly what's the public API and what's internal.

IMHO thats just not true; What you get on the commandline (i.e. from sage.all import *) is public and the rest is not. If thats not enough (and really nobody ever asked) we could mark extra imports as public, e.g. by adding special sage.foo.public packages.

Why does nobody ever ask?

 

If there were large body of pip-installable packages, which are user
code, this would help *define* what the public API of Sage really is,
and also give us a much larger body of code to test against before
making new releases.

testing against sufficiently large body of code which is not maintained by a project is a perfect way to make

sure that no new releases are made by the project, ever.

 

False.  You are simply arguing for ignorance. How we chose to use the results of such testing is up to us to decide.  

No, I am arguing against splitting Sage into independent fifedoms. 

And I have very good reasons for this.

Hell, even a model where the kernel is relatively small,

and there is a largish collection of independent packages (cf. GAP),  functions with a lot of friction between core devs and the rest, with some package authors often endlessly pedalling back kernel changes, kernel development happening in a fork, etc etc is not something one would look forward too. A large part of work for the release manager is being a politician, not

a developer, etc etc.

And the reason for this is trivial - many package authors don't like to touch their code, once it is written.

Which is understandable, but counter-productive for kernel development.

 

 

What API design school is that? You dump code on users and whoever manages to build the most convoluted contraption out of that will determine the future direction of the project ;-) Where is the leadership there? Who is going to handle the testing for each ticket, are you going to do that yourself? 

I don't care about design schools.   I would much rather be aware of how sage-dependent code is actually being used in the wild than to sit in school blissfully ignorant of how sage is really being used. 

I thought that with SMC you have a near-perfect opportunity to see what Sage users use in the wild...

SMC does inform my frustration with the current limitations  on Sage development. 

 

I am afraid we see a conflict of interest here.

It is in interests of SMC that Sage is very, very stable... Frozen.

 

And perhaps, perhaps, the 1st thing would be to get a single-user SMC frontend available as a pip-installable package, so that sagenb can retire, at last?

SMC is not a Python program.

and because of this, it cannot get python bindings? 

 

Dima

William

 

Dima

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscribe@googlegroups.com.

To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--
Sent from my massive iPhone 6 plus.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscribe@googlegroups.com.

To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[William Stein]

On Sat, Apr 9, 2016 at 2:49 AM, Dima Pasechnik <dim...@gmail.com> wrote:
>> SMC does inform my frustration with the current limitations on Sage
>> development.
>
> I am afraid we see a conflict of interest here.
> It is in interests of SMC that Sage is very, very stable... Frozen.

I want Sage to be (massively) easier to develop and use. I want Sage
usage (and development) to grow beyond its relatively small initial
user group, where it has been stuck since 2010. I think the best
way forward is to trust users and developers further, and support them
with better tools to do development, rather than viewing them as not
worthy of such (" independent fifedoms.... a lot of friction"). My
whole goal with Sage was to give all users as much flexibility and
power as possible, not to centralize authority too much (like with
Magma).

Feedback from users makes it clear they need a stable foundation on
which to build their work, and that -- as much as we imagine we are
providing one -- Sage is not, and things really have got worse (see
Simon King's message). I base this on the feedback in this thread,
and the endless feedback I get from users, and so on.

My proposal is simple: it would be very valuable to support (both
culturally and technically) people creating standalone Python packages
that depend on Sage, developed using current standard open source
practices, which are at this point in pretty good shape.

That's it.

I will continue to recommend to everybody I talk with about Sage

development that we switch from our current massive monolithic

centralized approach toward standard open source practices, as I
mentioned at the beginning of this thread. I hope I have more time
in the future to make this easier for people. I appreciate all the
feedback in this thread.

-- William

[Dima Pasechnik]

On Saturday, April 9, 2016 at 3:24:27 PM UTC+1, William wrote:

On Sat, Apr 9, 2016 at 2:49 AM, Dima Pasechnik <dim...@gmail.com> wrote:
>> SMC does inform my frustration with the current limitations  on Sage
>> development.
>
> I am afraid we see a conflict of interest here.
> It is in interests of SMC that Sage is very, very stable... Frozen.

I want Sage to be (massively) easier to develop and use.   I want Sage
usage (and development) to grow beyond its relatively small initial
user group, where it has been stuck since 2010.      I think the best
way forward is to trust users and developers further, and support them
with better tools to do development, rather than viewing them as not
worthy of such (" independent fifedoms.... a lot of friction").    My

While better tools are great, in effect you propose to create just this, independent fifedoms, 

however nice your modularisation proposal might sound.

How this (does not) work(s) in practice one can see to an extent with GAP, or with Python.

I already elaborated upon the former; for the latter, here is a concrete example, 

related to Sage (it broke Sage on OSX at some point),

with a pip-installable package, https://cryptography.io.

They only build with Xcode on OSX (and cannot with gcc), or at least it was the case, see

https://github.com/pyca/cryptography/issues/1924, which obviously broke Sage support. 

And the response was, well, how an independent fifedom would respond to a request not

on its agenda, with "well, hmm, we'll fix it perhaps some time in the future in some way, but 

we would not even look at your trivial 3-line patch, thanks."

Yet another (Cython) example was provided by Jeroen in this thread.

 

whole goal with Sage was to give all users as much flexibility and
power as possible, not to centralize authority too much (like with
Magma).

Feedback from users makes it clear they need a stable foundation on
which to build their work, and that -- as much as we imagine we are
providing one -- Sage is not, and things really have got worse (see
Simon King's message).    I base this on the feedback in this thread,
and the endless feedback I get from users, and so on.

My proposal is simple:  it would be very valuable to support (both
culturally and technically) people creating standalone Python packages
that depend on Sage, developed using current standard open source
practices, which are at this point in pretty good shape.

That's it.

I will continue to recommend to everybody I talk with about Sage
development that we switch from our current massive monolithic
centralized approach toward standard open source practices, as I
mentioned at the beginning of this thread.  

There is no successful computer algebra project (of the scale of Sage, or, say, 1/5th of Sage)

that follows what I think you call,  for some reason, "standard open source practices", while effectively meaning

projects like pip and npm (as if Linux kernel is not following "standard OS practices"). 

Moreover, Sage modularisation attempts in the past have not led to anything useful. 

I'd rather see you being much more concrete and constructive in your proposal.

Dima

[Jeroen Demeyer]

On 2016-04-09 16:23, William Stein wrote:
> Feedback from users makes it clear they need a stable foundation on
> which to build their work

And you think having *many* packages with interdependencies which are
independently managed will actually make it more *stable*? Sounds like
the contrary to me...

I believe you that there is a problem to be fixed here. However, I
absolutely do not understand why modularization of the Sage library
would fix that problem.

> My proposal is simple: it would be very valuable to support (both
> culturally and technically) people creating standalone Python packages
> that depend on Sage, developed using current standard open source
> practices, which are at this point in pretty good shape.

It's already possible to make Python packages for Sage (it's just Python
packages). Same thing here: I do not see how modularization of the Sage
library would fix this.

Maybe some small components which are of independent interest (see
cysignals for example) can be split off, but I doubt this is a good idea
for the mathematical part of Sage.

Jeroen.

[Erik Bray]

On Fri, Apr 8, 2016 at 1:35 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> On 2016-04-08 13:26, Luca De Feo wrote:
>>
>> This is something that should only happen at major version bumps, if ever.
>
>
> That's easily solved: the next stable release of Sage should be Sage 8, then
> Sage 9 and so on...
>
> We do have a 1-year deprecation policy, but of course it's not enforced (and
> I wouldn't know how to enforce that). If the author and reviewer of a ticket
> don't care about deprecation, then stuff will break.

Deprecation really should be enforced by policy. Use a decorator when
marking a function or class (or specific feature thereof) deprecated,
and include in the decoration the *version* in which it was marked
deprecated.

When working on prepping releases search for code that has been marked
deprecated for a long time and remove it. But definitely not too
soon. Don't advertise an interface as stable and then move it around
between X.Y releases without a deprecation process. Sage-the-library,
or much of it, is grown up enough (in terms of stability) that it
should definitely be doing this. Enforcing this should be part of the
release manger's job (but everyone should help by being vigilant).

Best,
Erik

[Jeroen Demeyer]

On 2016-04-11 14:29, Erik Bray wrote:
> Don't advertise an interface as stable and then move it around
> between X.Y releases without a deprecation process. Sage-the-library,
> or much of it, is grown up enough (in terms of stability) that it
> should definitely be doing this.

Of course we should be doing this, and in many cases we actually do. We
have a 1-year deprecation policy in Sage. Still, it's easy to forget
this and some people might be unaware.

> Enforcing this should be part of the
> release manger's job (but everyone should help by being vigilant).

I cannot speak for the current release manager, but I certainly wouldn't
want to check every merged ticket to check for things which should be
deprecated.

[Erik Bray]

On Sun, Apr 10, 2016 at 6:29 AM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> On 2016-04-09 16:23, William Stein wrote:
>>
>> Feedback from users makes it clear they need a stable foundation on
>> which to build their work
>
>
> And you think having *many* packages with interdependencies which are
> independently managed will actually make it more *stable*? Sounds like the
> contrary to me...
>
> I believe you that there is a problem to be fixed here. However, I
> absolutely do not understand why modularization of the Sage library would
> fix that problem.

It absolutely would help. The question, which needs to be considered
and discussed seriously, is where the cuts should be made. You're
probably right that there are certain parts of Sage that cannot be
reasonably cleaved apart in any way that's readily maintainable. But
there are other places where cuts are absolutely appropriate. The
questions we need to be asking are what are the "core" functionalities
of sage-lib, and what are optional features that are built on top of
the "core" functionalities.

>> My proposal is simple: it would be very valuable to support (both
>> culturally and technically) people creating standalone Python packages
>> that depend on Sage, developed using current standard open source
>> practices, which are at this point in pretty good shape.
>
>
> It's already possible to make Python packages for Sage (it's just Python
> packages). Same thing here: I do not see how modularization of the Sage
> library would fix this.
>
> Maybe some small components which are of independent interest (see cysignals
> for example) can be split off, but I doubt this is a good idea for the
> mathematical part of Sage.

The "mathematical" part of sage is pretty broad and needs to be defined.

See also my previous message regarding Astropy affiliated-packages.
Almost all affiliated-packages constitute "astronomy code", but that
doesn't mean they belong in the Astropy "core" package, at least not
initially. But having a family of official affiliated packages does a
lot mold and constrain development of the core package,. This includes
helping to define what constitutes the "stable API" (because in
exchange for affiliated packages using the "core" library, the core
library is guaranteeing a certain level of stability between versions.

Sage, unfortunately, hasn't made many pacts in this regard, so it's
hard to know exactly where to begin. I would probably start with the
standard convention for all Python which is that anything that is
*not* preceded by an underscore is de-facto "public", while anything
preceded by an underscore is de-facto "private". If this hasn't been
historically well thought out, then oh well. We're stuck with any
"public" interfaces being, well, public, and that means if we want to
change or get rid of them they have to go through a deprecation
process. For sanity's sake some exceptions can be made. For example:

1) If a "private" interface is explicitly known to be used by
external code, and is used in a manner which doesn't break things for
the internal code, then it can be "upgraded" to a public interface (by
explicitly adding a non-underscored version of that interface, and
deprecating the formerly private interface). In other words, if a
"private" interface was being used by external code, that demonstrates
that there may have been a need for the interface to be exposed in the
first place (that or somebody was doing something wrong, which if you
guarantee that private interfaces are truly private then you're free
to tell that person that their code is wrong and that it's only their
fault if it breaks :)

2) There may be code--especially entire modules--that do not use
underscored names, and yet are not explicitly for public consumption.
For example one might have a "foo.bar.utils" module containing utility
functions that are only used within the "foo.bar" module, and were
never really meant for "public" use. This is fairly common and
annoying to change every time it's encountered. So the best thing to
do is clearly mark in the "foo.bar.utils" module docstring that it's
for internal use only, and make sure to explicitly exclude it from the
published API docs so that users won't tend to find it. This is
imperfect but better than nothing.

3) Other exceptions can certainly be made on a case-by-case basis.
Having a policy for the general case and sticking to it as best as
possible, while not requiring a "foolish consistency", is better than
having no policy at all.

This is not something that can be ignored. Doing so will (and
probably already has) lead to a downward spiral of poor design.

Best,
Erik

[Erik Bray]

I have done this as release manager of Astropy. It's no doubt very
time consuming to check every merged pull request, though I do have a
checklist for what I check for, and urge other maintainers with
merge-to-master rights to go down it as well. Some are better than
others about this and I'm more likely to scrutinize some commits more
closely than others depending on the committer's past history of
checking the policies :) Also some automated checks are put in place
which really help.

Anyways, it helps to be able to *pay* someone to do this grunt work...
:) It's definitely a lot for a volunteer to do.

Best,
Erik

[Jeroen Demeyer]

On 2016-04-11 15:00, Erik Bray wrote:
> Also some automated checks are put in place
> which really help.

Can you elaborate on this?

[Volker Braun]

On Monday, April 11, 2016 at 3:00:12 PM UTC+2, Erik Bray wrote:

It's no doubt very time consuming to check every merged pull request

 

Thats a great idea and scales really well when the project merges 5 branches a day. It will also ensure that there are always enough volunteers to serve as release manager.

[Volker Braun]

On Monday, April 11, 2016 at 2:57:16 PM UTC+2, Erik Bray wrote:

Sage, unfortunately, hasn't made many pacts in this regard

Sage does have a very clear way of making symbols available on the commandline, namely via accompanying all.py files. We can either use that to define our public api (which it de facto already is when you use Sage interactively), or go through every module and sprinkle around underscores. Only one of these two is realistic.

This needs to be complemented with automated testing. On the Sage side, so that the public api does not accidentally change. Equally important, we need a mechanism for consumers of the api to avoid fragile dependencies on private structures. This could be via unit testing, a linter, or an import meta hook.

[Nils Bruin]

On Monday, April 11, 2016 at 10:46:59 AM UTC-7, Volker Braun wrote:

On Monday, April 11, 2016 at 2:57:16 PM UTC+2, Erik Bray wrote:

Sage, unfortunately, hasn't made many pacts in this regard

Sage does have a very clear way of making symbols available on the commandline, namely via accompanying all.py files. We can either use that to define our public api (which it de facto already is when you use Sage interactively), or go through every module and sprinkle around underscores. Only one of these two is realistic.

Are you proposing to define the public API as just the functions that are available on the command line by default? It sounds attractive, but I think it's not workable. For instance, in

sage: k.<a>=GF(5^3)
sage: k.primitive_element()
a

I think we do have to consider `primitive_element` as part of the public API (otherwise, how do people get a primitive element in a finite field using only the public API?) So that would leave the sprinkling around underscores as the only realistic option ... Would you care to reconsider?

[Volker Braun]

On Monday, April 11, 2016 at 8:29:06 PM UTC+2, Nils Bruin wrote:

Are you proposing to define the public API as just the functions that are available on the command line by default?

And all the (non-underscored) methods on objects that you can construct starting from globals, of course. Sorry if that wasn't clear. I.e. basically everything you can do in interactive use, excluding manual imports.

[Vincent Delecroix]

including

sage: sage.combinat.permutation.left_action_product([2,1,3,4], [3,1,2])

Basically all of Sage can be reached from the global namespace without
manual import because modules are already loaded...

[Volker Braun]

Excluding accessing sage.* or the module import cache directly

There are various language-lawyering ways of expressing this; I appreciate your contribution to find a bulletproof formulating but I think its intuitively clear what is meant.

[Erik Bray]

Sorry, I can't tell if this was meant to be sincere or not. Because
I'm being quite sincere. Sage isn't the biggest project in the
world--it can be managed. The problem currently is that it's on an
all-volunteer basis, and that's hard. And I respect that. But I don't
think having an increasingly unmanageable project is going to attract
*more* volunteers. Another reason to look for places to trim the fat,
so to speak.

>> Sage, unfortunately, hasn't made many pacts in this regard
>
> Sage does have a very clear way of making symbols available on the
> commandline, namely via accompanying all.py files. We can either use that to
> define our public api (which it de facto already is when you use Sage
> interactively), or go through every module and sprinkle around underscores.
> Only one of these two is realistic.

Why only one? One thing that too many people in this discussion seem
to assume is that this has to be all or nothing. You wrote "every
module", but why not "some modules", or "the most abused modules"?
Or, "on a case-by-case basis when evidence of misuse turns up in bug
reports"? Because an incremental approach is better than doing
nothing at all. An all.py file is fine too though and certainly helps
as a starting point. We could say "don't use anything that's not
listed there". But certainly when it comes to class methods, perhaps
more than anything else, consistent use of underscores is a must.
Module names, and functions within modules are sometimes a little
looser, and better handled through __all__ or some equivalent thereof.

> Equally important, we need
> a mechanism for consumers of the api to avoid fragile dependencies on
> private structures. This could be via unit testing, a linter, or an import
> meta hook.

Not exactly sure what you would use an import hook for in this case,
but it sounds like overkill. It sounds instead like a slight cultural
shift is needed--that people writing code for sage need to understand
what they're writing Python, and should follow the standard
best-practices for Python. They should understand what is and isn't
for private use, and that they can't rely on private interfaces. Unit
testing + linter rules for all code sounds like a good idea though!

Best,
Erik

[Erik Bray]

This is a perfect example of why class methods are especially critical
to have well-defined public/private visibility. There's no
__all__-like mechanism for classes, unfortunately (the closest I can
think of is __dir__).

Thanks,
Erik

[Erik Bray]

On Tue, Apr 12, 2016 at 11:19 AM, Erik Bray <erik....@gmail.com> wrote:
> On Mon, Apr 11, 2016 at 7:36 PM, Volker Braun <vbrau...@gmail.com> wrote:
>> On Monday, April 11, 2016 at 3:00:12 PM UTC+2, Erik Bray wrote:
>>>
>>> It's no doubt very time consuming to check every merged pull request
>>
>>
>> Thats a great idea and scales really well when the project merges 5 branches
>> a day. It will also ensure that there are always enough volunteers to serve
>> as release manager.
>
> Sorry, I can't tell if this was meant to be sincere or not. Because
> I'm being quite sincere. Sage isn't the biggest project in the
> world--it can be managed. The problem currently is that it's on an
> all-volunteer basis, and that's hard. And I respect that. But I don't
> think having an increasingly unmanageable project is going to attract
> *more* volunteers. Another reason to look for places to trim the fat,
> so to speak.

As an aside after reading this over again, I apologize if this came
across as not valuing the work you do as release manager. I know it's
a time-intensive and often thankless task, especially when it's being
done on a volunteer basis. I just wanted to be clear on that.

It's certainly *nobody's* fault that sage is the way it is, and not
everything about it is bad either (I think the existing issue workflow
is great for the most part!). I just think it's a big enough project
that it can't afford not to be developed more carefully going
forward...

Best,
Erik

[Pedro Cruz]

(1) Sage developers could be better informed of what are they are going to affect if sage, as an "author tool",  send usage statistics about each "sage-lib" related function:

- Author should put sage in "send usage statistics mode"

- The same author would be warned if any of those functions starts to be deprecated.

- And maybe, if authors freely send files to be kept and seen as "testing examples" like in "sage -t <author-example>.sage"

But after a new deprecation warning, I'm thinking of implement one of this solutions to my code:

(2) A tool to automatically change the deprecated "simplify_exp" for "canonicalize_radical" inside sage files;

(3) Create my own local "simplify_exp" that mimics original "simplify_exp" because here authors enjoy applying "simplify_exp" to exponential expressions.

Pedro

(a developer of a package that uses sagemath and will be in pypi as suggested)

sexta-feira, 8 de Abril de 2016 às 17:43:46 UTC+1, William escreveu:

On Fri, Apr 8, 2016 at 9:33 AM, Luca De Feo <de...@lix.polytechnique.fr> wrote:
>> How often do you feel that changes in Sage breaks your code?
>
> Looking at my log (this project
> https://github.com/defeo/ss-isogeny-software)
>
> - Sage 6.2: module renamed in Pari/GP interface (without deprecation, as far
> as I recall)
> - Sage 6.4: frankly, I don't know, but possibly a change in the finite
> fields implementation
> - Sage 6.7: changes in cython, not really Sage's fault, I guess
> - Sage 6.7: changes in the finite fields implementation (with proper
> deprecation, this time)
> - Sage 6.10: changes in richcmp (did not get deprecation warnings, but the
> change concerns an internal function)
> - Sage 7.1: affected by #19941 (properly deprecated, note that 7.1 is stable
> right now, but not yet in SMC)
>
> It may be argued that I'm taking risks because I sometimes fiddle with "the
> internals" of Sage, but one other problem of Sage is that it does not define

> clearly what's the public API and what's internal.

+1 -- this "one other problem of Sage is that it does not define
clearly what's the public API and what's internal." is a HUGE problem
-- and we're not aware of it because "user code" is typically private.

If there were a large body of pip-installable packages, which are user

code, this would help *define* what the public API of Sage really is,
and also give us a much larger body of code to test against before
making new releases.

As an example, when working on

  https://github.com/williamstein/sage_modabvar

the first thing I noticed in that from sage-6.10 to sage-7.x

   import sage.rings.arith

completely BROKE.  It became "import sage.arith".

This is (who knows?) a change in the public api of the core sage
library.  Maybe it's a gratituous one, and in this case one I'm
puzzled by why this change was made.  (I wrote sage.rings.arith in the
first place, and it was meant to be some misc arith needed to
implement rings.  Why move it to its own top level module like that?)

By writing Python packages that depend on "the core sage library" we
will be forced to define what the core library is and stabilize it.
If you guys want Sage to survive/thrive, this **MUST HAPPEN**.   The
core sage library should -- like with Python -- be the place where
code, which has been tested/used by people and got stable, goes to
stay stable (at least API-wise).  Like the standard Python library.

William

>
> Luca

>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to sage-devel+...@googlegroups.com.

> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--

William (http://wstein.org)

[kcrisman]

This has been an interesting thread.  In the end, I think that some (or a lot) of Sage's attractiveness to end users goes away if it becomes a bunch of possibly-updated packages that might or might not work with a current version of Sage.  I always found the "with(plots)" syntax (or whatever it was) in Maple very frustrating, and that is presumably a 'core' package; having random stuff suddenly not work (let me be clear, because it was left behind by Sage core) would be even more so - as has been pointed out several times here.

Sage users (and potential ones) I speak with want more than just the "basic" functionality, because they want something they can use throughout the curriculum and in their own research.  There are other (good) tools for those who truly won't be doing anything beyond calculus.

Now it's true that some material in Sage probably could have been in separate packages, as it's quite specialized - likely a lot of the sage-combinat stuff, the designs stuff, modular forms stuff (elliptic curves are actually more popular, I think).  But then there's the opposite problem of finding out how to enforce that a package must compile with the most recent Sage.  This is R's model, but R tends to have a very different type of package, one that implements something relatively narrow.  Also, we don't have the auto-testing resources of R.

Given that, I'd definitely rather have the kitchen sink, including that stuff, in Sage - as a *user*, not developer.  I know that whatever is in Sage will stay in it, and it will all be there.

The problems that Luca and Simon are (rightly) pointing out, in my view, are not solved by more modularity - if anything, the problems were because they were not part of 'standard' Sage, though I of course am not suggesting they should have been part of it.  With the kind of auto-testing R has, at least Simon's problems could have been immediately noted and avoided (as a full optional package); Luca's list of breakages is impressive indeed, and a better way of indicating what is "API" and what isn't would have been very useful there, and would be useful now.  But you can have a permissive or non permissive set of publicly available stuff with or without lots of separate packages. 

As an example of what happens with the package system, consider several Maxima packages (which shall remain unnamed) which don't work well with other Maxima commands/flags/packages (no doubt rjf would say we are using them improperly, which may be true).  Well, they're separate packages, with their own maintainers, and I don't know that anyone beyond them takes responsibility, and fixing lots and lots of hard-to-track-down bugs once you've put in the initial effort is very daunting and time-consuming.  So they kind of languish, I think - not that some Sage bugs don't too, but there is less likelihood that someone else will take the time to work on them if it's "just a package", perhaps with its own separate web affiliation.

(I'm sure the same thing applies to many user-contributed Mma and Maple packages, but I don't listen in on their ecosystems - I mean upgrades breaking them, that is.   They are fairly monolithic, though?)

To be clear, I'm only talking about the core Sage library; if people can find a way to make the other stuff more like sage-on-Gentoo without making it really, really hard to use on any setup other than the most popular distros/most bleeding-edge Mac OS, that is great.  

I also don't have a problem with things like psage or any other such packages (as I think William is proposing in this thread for much of the functionality), but the experience thus far has been that the successful such things are eventually merged in Sage proper, rather than being contingent.  Think of how much time has been spent on managing the sage-combinat queue or branch or whatever to make sure it always works ...   *Given our constraints on testing*, building the car was the right idea then, and it's still the right idea now.  

And I would rather have the car that can drive everywhere - for me, for my students, for my colleagues - against the car that needs fancy upgrades that are often not available on my model.  Volker's original comment still holds: "As long as the goal of "import sage" is to give you something like the feature set of Sage right now we don't benefit from modularization. That is just a tautology, the goal is just not a modular one. We'd just shoot ourselves in the foot if we split things into multiple interdependent packages that then must be upgraded in lockstep."

[William Stein]

On Tue, Apr 12, 2016 at 7:11 AM, kcrisman <kcri...@gmail.com> wrote:
> This has been an interesting thread. In the end, I think that some (or a
> lot) of Sage's attractiveness to end users goes away if it becomes a bunch
> of possibly-updated packages that might or might not work with a current
> version of Sage. I always found the "with(plots)" syntax (or whatever it
> was) in Maple very frustrating, and that is presumably a 'core' package;
> having random stuff suddenly not work (let me be clear, because it was left
> behind by Sage core) would be even more so - as has been pointed out several
> times here.

I am a little annoyed, since this completely misunderstands my proposal.

1. My proposal was to make it easier for people to develop new code
independently of the core sage library. This in itself has nothing
to do with taking away from the existing library or in fact changing
it in any way (at least initially).

2. Moreover, to directly address your concern, if 3d plotting (say)
were split off as a separate Python package/library, that does *not*
imply that when you download and install Sage, or start it, that you
can even tell the difference. It doesn't mean that the normal visible
public API of Sage changes
at all. Why do you think otherwise? We would just include that
Python library as a standard package, just like the hundred other
standard packages. Volker has mentioned several times how Python
enables doing this sort of thing pretty easily already.

> Sage users (and potential ones) I speak with want more than just the "basic"
> functionality, because they want something they can use throughout the
> curriculum and in their own research. There are other (good) tools for
> those who truly won't be doing anything beyond calculus.
>
> Now it's true that some material in Sage probably could have been in
> separate packages, as it's quite specialized - likely a lot of the
> sage-combinat stuff, the designs stuff, modular forms stuff (elliptic curves
> are actually more popular, I think). But then there's the opposite problem
> of finding out how to enforce that a package must compile with the most
> recent Sage. This is R's model, but R tends to have a very different type
> of package, one that implements something relatively narrow. Also, we don't
> have the auto-testing resources of R.

1. There can and should exist packages that depend on the core Sage
library and have a relatively narrow focus. Why not?

2. Maybe we don't have the same auto-testing resources as R does
today. Who is to say we won't in a year or two?

I'm planning for a future where the Sage project *does* have
resources, and where it is possible to hire 2+ people fulltime to
maintain Sage, like what ODK is doing *right now*. That's what we
should be aiming for. We have it right now (due to ODK) with Erik and
Jereon, at least, and if SageMath Inc succeeds, then it will be
possible to continue and grow this.

>
> The problems that Luca and Simon are (rightly) pointing out, in my view, are
> not solved by more modularity - if anything, the problems were because they
> were not part of 'standard' Sage, though I of course am not suggesting they
> should have been part of it.

Their problems were partly caused by us not supporting and encouraging
the creation of code outside of standard Sage. We should be doing
that 1000x what we are doing now. Right now, we as a community (not
me, but certainly many others) are shockingly discouraging and
negative toward any code that isn't officially in the core sage
library. I think this situation is really baffling to see for a lot
of outsiders.

> As an example of what happens with the package system, consider several
> Maxima packages (which shall remain unnamed) which don't work well with
> other Maxima commands/flags/packages (no doubt rjf would say we are using
> them improperly, which may be true). Well, they're separate packages, with
> their own maintainers, and I don't know that anyone beyond them takes
> responsibility, and fixing lots and lots of hard-to-track-down bugs once
> you've put in the initial effort is very daunting and time-consuming. So
> they kind of languish, I think - not that some Sage bugs don't too, but
> there is less likelihood that someone else will take the time to work on
> them if it's "just a package", perhaps with its own separate web
> affiliation.

Just because some people have zero funding and are bad at packaging
doesn't mean that the mainstream standard mature approach to open
source software development, as exemplified in many ecosystems now (R,
Pypi, npm, Debian, etc.) is broken.

> (I'm sure the same thing applies to many user-contributed Mma and Maple
> packages, but I don't listen in on their ecosystems - I mean upgrades
> breaking them, that is. They are fairly monolithic, though?)
>
> To be clear, I'm only talking about the core Sage library; if people can
> find a way to make the other stuff more like sage-on-Gentoo without making
> it really, really hard to use on any setup other than the most popular
> distros/most bleeding-edge Mac OS, that is great.
>
> I also don't have a problem with things like psage or any other such
> packages (as I think William is proposing in this thread for much of the
> functionality), but the experience thus far has been that the successful
> such things are eventually merged in Sage proper, rather than being
> contingent.

Extreme selection bias. Our community strongly forces things to fail
if they aren't "merged in Sage proper". You then look at that and
conclude things that succeed do so because they are merged in Sage
proper.

> Think of how much time has been spent on managing the
> sage-combinat queue or branch or whatever to make sure it always works ...
> *Given our constraints on testing*, building the car was the right idea
> then, and it's still the right idea now.

Cars are built out of parts.

> And I would rather have the car that can drive everywhere - for me, for my
> students, for my colleagues - against the car that needs fancy upgrades that
> are often not available on my model. Volker's original comment still holds:
> "As long as the goal of "import sage" is to give you something like the
> feature set of Sage right now we don't benefit from modularization. That is
> just a tautology, the goal is just not a modular one. We'd just shoot
> ourselves in the foot if we split things into multiple interdependent
> packages that then must be upgraded in lockstep."

Modularization -- one of the most basic and important ideas in
software engineering -- is not shooting ourselves in the foot.

-- William

[kcrisman]

 It doesn't mean that the normal visible 
public API of Sage changes 
at all.   Why do you think otherwise?

Well, reading this thread made me think that.  Because I don't see how we enforce all those "other pieces" working nicely together, so some would (perhaps quite quickly) drop by the wayside.  You are right, as I apparently didn't make clear, that it would be even better to have people more easily able to have separate packages that are quite narrow - and I think that a testing framework like R has would probably suffice for this (as long as breaking such packages was a blocker for release!).  (There's also the CRAN versus RForge etc. question...) But I don't see how this is possible with the *current* Sage.

> Cars are built out of parts. 

Yes, parts that are, despite parts suppliers not being the same as the car companies, those parts have high tolerances and highly controlled supply chains.  (Or so my brief experience with a parts supplier long ago suggests.)  All your examples of "mature open source development" are not the same in their goal as Sage, which apparently aims to have all of mathematics at its fingertips (in the way that Maple and Mathematica, though presumably not Magma or Matlab, seem to).  That's a lot harder to disentangle than the dozen or more R packages I've had occasion to use are, and presumably also the case in non-mathematical software.  Modularity in Sage and resolving our import fiasco isn't the same thing as saying that every folder in Sage has to be a separate project, which is what you seem to be implying; though since I misunderstood the very first point, maybe I'm wrong about that too.

As an example, take game_theory.  This is exactly the kind of "narrow" thing one might think would survive better as a modular bit that could be imported.  Which one of the programs in question it relies on (gambit) is, in terms of Python, of course.  Yet there is a lot of opportunity with being inside of "real" Sage for them with interconnections with all kinds of other combinatorial and plotting and other stuff that they would be hard put to keep current without being in the Sage library.  Having access to all that other math makes the game_theory stuff more powerful.  I am skeptical that math can be pared down to just a few pieces that everything else fairly loosely holds fast to.

But like I said, if you or ODK or someone else can make this vision happen, as long as the end user still gets the kitchen sink without having to do some wacky import yoga or use possibly-broken packages, I'm not complaining!  "I also don't have a problem with things like psage or any other such packages" - and if you're right that my view on that is selection bias, I would be very happy to be wrong.  (Well, not really that happy, because people have worked hard and Sage has had weird upgrades that broke their stuff.  But you know what I mean.)  Bring on the automated testing!  :-)

[William Stein]

On Tue, Apr 12, 2016 at 12:50 PM, kcrisman <kcri...@gmail.com> wrote:
>> It doesn't mean that the normal visible
>> public API of Sage changes
>> at all. Why do you think otherwise?
>
>
> Well, reading this thread made me think that. Because I don't see how we
> enforce all those "other pieces" working nicely together, so some would
> (perhaps quite quickly) drop by the wayside.

If 3d plotting were in a separate python library we would enforce it
working in a new release in precisely the same way we do so now with
standard packages. Install and test it. However, by developing it
as a separate library, we can clarify how it depends on the rest of
sage (and vice versa). We might also be able to make it available
outside Sage, and it could suddenly be of huge value to the Python
world. In fact, technically the sage 3d plotting has a huge amount
of cool highly optimized and useful code for 3d graphics, e.g., the
marching cubes implicit plot code, etc., and beautifully written
Cython 3d plotting code by Robert Bradshaw. It is a *really stupid*
and sad waste that this 3d functionality is *only* available in Sage?
Why not in Sympy or elsewhere? As best I can tell it is because of
fear like you seem to have...

It's all my fault originally, but I think I've learned something about
software engineering during the last few years... and I think we're
doing it wrong.

> You are right, as I apparently
> didn't make clear, that it would be even better to have people more easily
> able to have separate packages that are quite narrow - and I think that a
> testing framework like R has would probably suffice for this (as long as
> breaking such packages was a blocker for release!).

It may or may not be, depending on the package and why it broke.
That's up to us to decide.

William


--
William (http://wstein.org)

[Volker Braun]

On Tuesday, April 12, 2016 at 9:59:40 PM UTC+2, William wrote:

We might also be able to make it available
outside Sage, and it could suddenly be of huge value to the Python
world.

In other words: If we change our mission to "the most useful collection of libraries" then we should split up the sage library because it would be very useful to other projects.

[William Stein]

Sigh....

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--

[Michael Orlitzky]

On 04/12/2016 03:58 PM, William Stein wrote:
>
> It's all my fault originally, but I think I've learned something about
> software engineering during the last few years... and I think we're
> doing it wrong.
>

Don't be too hard on yourself. The fact that every function in Sage is
documented, tested, and reviewed is a bigger "pro" than the packaging is
a "con." In that respect Sage is a shining beacon in a sea of garbage.
The technical problems are relatively easy to solve.

Making modularization possible can only be a good thing. We don't have
to force a development model on anyone. If people want to add their code
to Sage so that it gets doctested in future releases, great. If they
want to maintain it as a separate package, great. If Sage becomes so
modular that developers can depend on a specific version of
sage-the-library, then breakage should be minimized (as long as we don't
make crazy changes in minor releases).

[R. Andrew Ohana]

On Tue, Apr 12, 2016 at 12:50 PM, kcrisman <kcri...@gmail.com> wrote:

 It doesn't mean that the normal visible 
public API of Sage changes 
at all.   Why do you think otherwise?

Well, reading this thread made me think that.  Because I don't see how we enforce all those "other pieces" working nicely together, so some would (perhaps quite quickly) drop by the wayside.  You are right, as I apparently didn't make clear, that it would be even better to have people more easily able to have separate packages that are quite narrow - and I think that a testing framework like R has would probably suffice for this (as long as breaking such packages was a blocker for release!).  (There's also the CRAN versus RForge etc. question...) But I don't see how this is possible with the *current* Sage.

> Cars are built out of parts. 

Yes, parts that are, despite parts suppliers not being the same as the car companies, those parts have high tolerances and highly controlled supply chains.  (Or so my brief experience with a parts supplier long ago suggests.)  All your examples of "mature open source development" are not the same in their goal as Sage, which apparently aims to have all of mathematics at its fingertips (in the way that Maple and Mathematica, though presumably not Magma or Matlab, seem to).

This also seems to be the crux of why some Sage developers are not interested in breaking things up (as opposed to why they actively do not want to break things up) -- namely that Sage should be a monolithic application just like the Maple and Mathematica, and that is all it should be.

While in the early days a lot of Sage's functionality was simply wrapping other open source libraries, now there is quite a bit that can only be found in Sage's library -- it seems to me only like the right thing to do to make it so that others in the open source community could easily benefit from our work just like we have benefited from theirs.

This doesn't conflict with our mission statement, as it is still perfectly possible to create a interactive shell with tons of functionality from a collection of libraries (again, Sage started as this).

As for the technical concerns of having a modular library, it seems the biggest worry is that it will be too hard to orchestrate all of the inter-dependent components of Sage. At this point, I'm not too concerned on this front -- many projects in the past have solved these sorts of issues and I don't think Sage is special in this regard. I would want to see a more formal technical proposal (something along the lines of a SEP) before I would feel comfortable criticizing such a proposal.

 

That's a lot harder to disentangle than the dozen or more R packages I've had occasion to use are, and presumably also the case in non-mathematical software.  Modularity in Sage and resolving our import fiasco isn't the same thing as saying that every folder in Sage has to be a separate project, which is what you seem to be implying; though since I misunderstood the very first point, maybe I'm wrong about that too.

As an example, take game_theory.  This is exactly the kind of "narrow" thing one might think would survive better as a modular bit that could be imported.  Which one of the programs in question it relies on (gambit) is, in terms of Python, of course.  Yet there is a lot of opportunity with being inside of "real" Sage for them with interconnections with all kinds of other combinatorial and plotting and other stuff that they would be hard put to keep current without being in the Sage library.  Having access to all that other math makes the game_theory stuff more powerful.  I am skeptical that math can be pared down to just a few pieces that everything else fairly loosely holds fast to.

But like I said, if you or ODK or someone else can make this vision happen, as long as the end user still gets the kitchen sink without having to do some wacky import yoga or use possibly-broken packages, I'm not complaining!  "I also don't have a problem with things like psage or any other such packages" - and if you're right that my view on that is selection bias, I would be very happy to be wrong.  (Well, not really that happy, because people have worked hard and Sage has had weird upgrades that broke their stuff.  But you know what I mean.)  Bring on the automated testing!  :-)

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--

Andrew

[Luca De Feo]

Let me share some more thoughts on the interplay between modularization and the review process. I'm afraid I'm going to be long... TL;DR: modularization is good to keep Sage going forward.

Our review process discourages patchbombs, and for a reason. For one, no one wants to review patchbombs. But, most importantly, it is too risky to drop a patchbomb into the next release, having the doctests and a couple of human eyes as the only safety guarantee.

The consequence of this is that we modify old code by carefully *sprinkling* diffs here and there, breaking up major changes into minor, innocent looking patches. I know something about it, I'm doing it right now for the Pari interface. There is nothing fundamentally bad about this, and, eventually, old code gets fully rewritten and the codebase gets improved. Only, this is time-consuming, and, by definition, it leads to *patchy* code.

What I want to highlight is that non-modularization + review process discourage major rewrites too much. Some examples:
- How many modules in Sage haven't moved to the category framework yet (assuming that the move would make sense for the module)?
- With all the respect I owe to the huge work done by William, John, and the other contributors to elliptic curves in Sage, I've found myself many times thinking "This design is so bad! I want to rewrite it from scratch".

On the opposite site, take the sage-coding project: it is making huge leaps forwards, designing non-trivial mathematical interfaces at an impressive pace. How do they manage it? By writing from scratch, at times even by writing outside of Sage. They have no (or, rather, few) pre-existing interfaces to cater for, and they can simply drop small tickets with half-working code because no other code is depending on them. Most of Sage's code was developed in bursts like this one. Only, after some time, you may realize that there is a better design for what you've written some years before, and that's when you're stuck with the problem of redesigning while maintaining old code at the same time.

Constantly rewriting from scratch is silly, and that's why the review process is there. But, never rewriting parts of a code that is 10 years old is also silly. By making it impossible to drop patchbombs, we're slowing down progress. If we made Sage more modular, if it was easy to build code *outside* of Sage and then progressively bring it inside Sage once it is mature, then rewriting whole modules would become much easier.

Let's say I'm set up on rewriting the elliptic curve modules, and that I've gathered a small team of enthusiasts to help me. Here's my choices right now:
- Announce my plan on sage-devel, seek consensus, maybe write a white paper, modify sage.schemes.elliptic_curves little by little with small patches. This would likely keep breaking the public API at each minor release until the rewrite is done. If we keep faith and get to the end, we would end up with *patchy* code.
- Announce my plan on sage-devel, seek consensus, start development in a sage.schemes.elliptic_curves.new package, inside-review the small tickets. When the package is ready, go back to sage-devel, propose to replace the old package with the new one, wait a couple of years. Eventually the rewrite will be accepted, old code will break, the only way for users to keep the old interface will be to stop upgrading Sage. Or, most likely, the rewrite will stay in .new, it will be forgotten, and rot there for the rest of its life.

Now, suppose that elliptic curves were a *core* Sage module, shipped with Sage, but which you can uninstall nevertheless. Here's how I would go: Start the new elliptic module outside of Sage, play with it, redesign it a few times. When it starts getting serious, write about it on sage-devel, seek consensus, document the public API. When the module gets stable enough, start advertising it, tell people they can try it out with `sage -i defeo/elliptic_curves`. When the module has gained some traction, go back to sage-devel and propose to replace the core package with it. Have more people look at it, reach feature parity with the core package, wait some minor releases. Eventually the replacement gets accepted, a lot of warnings are issued to the users, the new code goes in at the next *major* release, old code can still run (at least for some time) by doing `sage -u elliptic_curves` and `sage -i elliptic_curves_old`, I become the main maintainer of the elliptic curves core package, John is happy because he does not have to do maintenance anymore. Or maybe the replacement is not accepted, then I write an incendiary email on sage-devel saying all Sage devs are fascists, and I'm quitting the community, and tell all my colleagues to `sage -u elliptic_curves` and `sage -i defeo/elliptic_curves`. The community has lost a (bad) developer, but no code was lost in the process :)

I understand that some people here are scared of dispersion: what if we end up with a dozen, half-baked, badly maintained, concurrent elliptic curves modules? Wouldn't it be better to make all these developers contribute to Sage directly? This is a legitimate fear, and a realistic scenario if the community is not properly organized. Better tools can only lead to better code if they are properly used. I hope that my exposé gave you an impression of how we could use more modularization for good, and will make you embrace that vision.

Technical details are coming, as soon as I'm finished writing the report on Sage days 77.

Luca

[kcrisman]

Well, reading this thread made me think that.  Because I don't see how we enforce all those "other pieces" working nicely together, so some would (perhaps quite quickly) drop by the wayside.  You are right, as I apparently didn't make clear, that it would be even better to have people more easily able to have separate packages that are quite narrow - and I think that a testing framework like R has would probably suffice for this (as long as breaking such packages was a blocker for release!).  (There's also the CRAN versus RForge etc. question...) But I don't see how this is possible with the *current* Sage.

> Cars are built out of parts. 

Yes, parts that are, despite parts suppliers not being the same as the car companies, those parts have high tolerances and highly controlled supply chains.  (Or so my brief experience with a parts supplier long ago suggests.)  All your examples of "mature open source development" are not the same in their goal as Sage, which apparently aims to have all of mathematics at its fingertips (in the way that Maple and Mathematica, though presumably not Magma or Matlab, seem to).

This also seems to be the crux of why some Sage developers are not interested in breaking things up (as opposed to why they actively do not want to break things up) -- namely that Sage should be a monolithic application just like the Maple and Mathematica, and that is all it should be.

While in the early days a lot of Sage's functionality was simply wrapping other open source libraries, now there is quite a bit that can only be found in Sage's library -- it seems to me only like the right thing to do to make it so that others in the open source community could easily benefit from our work just like we have benefited from theirs.

This doesn't conflict with our mission statement, as it is still perfectly possible to create a interactive shell with tons of functionality from a collection of libraries (again, Sage started as this).

As for the technical concerns of having a modular library, it seems the biggest worry is that it will be too hard to orchestrate all of the inter-dependent components of Sage. At this point, I'm not too concerned on this front -- many projects in the past have solved these sorts of issues and I don't think Sage is special in this regard. I would want to see a more formal technical proposal (something along the lines of a SEP) before I would feel comfortable criticizing such a proposal.

This is all a good summary, thank you.  If someone feels it is worth the energy to do all this, that sounds fine; I would just be concerned that it wouldn't be realistic (it would remain possible, I suppose) to do it without giving up the "monolithic" version for those (which is a lot) who would want it. 

> Why not in Sympy or elsewhere?     As best I can tell it is because of fear like you seem to have...

Well, and the nontrivial question of when/where it's going to happen, which will by necessity invoke an opportunity cost with respect to other things that people who know how to do that (a small set) are doing.  But it's certainly more a practical objection than a principled one, and in any case I fear to wear out my welcome since I have not been able to actively work on Sage this semester, or year, really. :-(

[mmarco]

As for the technical concerns of having a modular library, it seems the biggest worry is that it will be too hard to orchestrate all of the inter-dependent components of Sage. At this point, I'm not too concerned on this front -- many projects in the past have solved these sorts of issues and I don't think Sage is special in this regard. I would want to see a more formal technical proposal (something along the lines of a SEP) before I would feel comfortable criticizing such a proposal.

 

That is my feeling exactly right now: I see good arguments for the "going modular" idea, and also good arguments against it. Without a specific proposal of how this idea should be materialized in practice, it is difficult for me to get a position about wheather the pros outweight the cons or not.

One idea that has come to my mind by reading this thread is the gentoo metapackages. They are packages that do nothing in reality, but deppend on a bunch of other packages. That way, if you want to install, say, the KDE desktop, you just install the KDE metapackage, and it will pull all the modular parts of KDE as dependencies. So translated to our setting, we could have a Sage "metapackage", that enforces to install all the parts that conform "Sage the software". Maybe we would need a better packaging system than what we have now though.

And of course, it remains to see in how many different packages the sage library can be splitted. If there are circular dependencies everywhere, at the end we would end with just one package for the whole sage library. Maybe we should start by taking a look at the dependency graph of the different modules to see exactly where are the possible places for splitting.

But again, until we come to a concrete proposal, this is pretty much hand waving.

[Jeroen Demeyer]

On 2016-04-12 21:58, William Stein wrote:
> If 3d plotting were in a separate python library we would enforce it
> working in a new release in precisely the same way we do so now with
> standard packages. Install and test it.

Which is very much a pain right now with SageNB. I find it a bad thing
that SageNB is a separate project while it's still tested within Sage
itself. It's in maintenance mode right now but it breaks doctests now
and then if something in Sage changes. That is really annoying because a
trivial commit now involves making a new SageNB release. Not fun!
I would in fact advocate to merge SageNB back into Sage proper.

My conclusion is that we need to treat those packages either as truly
separate packages or as truly part of Sage. Anything in between (like
SageNB currently) is bad.

[Jeroen Demeyer]

On 2016-04-05 20:44, William Stein wrote:
> I am recommending to absolutely everybody I talk with about Sage
> development that we switch from our current massive monolithic
> centralized approach toward standard open source practices. Namely,
> lots of smaller libraries, standard open source practices, etc.

Does anybody happen to know how Jupyter coped with this? About 1 year
ago, they split themselves up from a single IPython project to about 10
separate projects. They might have some valuable experience to share.

[Jeroen Demeyer]

Luca, I read your post but I don't really buy your argument that
"rewriting an entire module" is easier with a modular Sage. I think it's
equally hard in all cases, modular or not.

I think the reason that the coding people are moving forward quickly is
mainly because there is an enthousiastic group just doing stuff. If we
could get a few people like that together for elliptic curves, maybe do
a Sage Days around it, I'm sure we could make as much progress within
the current Sage development model. I have a few elliptic-curve related
tickets lying around, but just no time to work on them.

Jeroen.

[Simon King]

Hi!

On 2016-04-13, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> I think the reason that the coding people are moving forward quickly is
> mainly because there is an enthousiastic group just doing stuff.

Aren't there further examples? IIRC, the matroid code was (or still is?)
developed separately, and then made its way into Sage proper.

Question to people with computer engineering experience: Shouldn't we
distinguish between *creating* functionality in a modular way (such as:
creating matroid code as an optional package) and *maintaining* it in a
modular way?

My impression with my group cohomology package seems to suggest to me
that maintaining a separate package is more difficult than creating it
in the first place, and I regret that I didn't start earlier with
binding it more tightly to the Sage code, so that regressions could be
detected and fixed more easily.

Best regards,
Simon

[mmarco]

That sounds like an important point. Definitely creating software and maintaining it are two different tasks that might require different workflows.

[mmarco]

Just to take a look at how much different modules deppend on each other right now, I did some quick analysis by looking at the import statements of the .py files in the sage library.

The results of this quick experiment are as follows:

1. there are 9 modules in the base of the dependency graph (that is, avery other module imports something from them, or from other modules that imports something on them, or so on). They are schemes, ext, docbuild, version, env, all_cmdline, all, all_notebook, and notebook
   
2. over them there is a big set of modules that have circular dependencies between them. That is, they import something from the previous set, or from other module in this set. They are probability, data_structures, misc, combinat, server, quadratic_forms, homology, plot, functions, stats, matrix, rings, categories, algebras, calculus, arith, finance, symbolic, interfaces, numerical, modular, repl, groups, databases, libs, parallel, structure, geometry, modules, typeset, graphs, quivers, doctest, sets, gsl, monoids.
   
3. Then there are 16 modules that import something from the previous two lists, but not from anything on this level. They are: games,  tensor,  matroids,  knots,  interacts,  crypto,  dev,  logic,  lfunctions,  dynamics,  sat,  coding,  game_theory,  tests,  sandpiles,  media
   
4. Finally, there is manifolds, that imports from tensor and from level (2).
   

As I said, this is just a quick look (for instance, I didn't consider lazy imports). But I think it gives an idea of what is already moreless independent, and what is pretty much entangled with the rest of the components. One would be tempted to say that (2) consists of the "core" of the Sage library, whereas (1) are its dependencies and (3) and (4) would be "extra" modules that deppend on the core library. But again, this is just a first impression after a quick look.

[Erik Bray]

On Tue, Apr 12, 2016 at 4:11 PM, kcrisman <kcri...@gmail.com> wrote:
> This has been an interesting thread. In the end, I think that some (or a
> lot) of Sage's attractiveness to end users goes away if it becomes a bunch
> of possibly-updated packages that might or might not work with a current
> version of Sage. I always found the "with(plots)" syntax (or whatever it
> was) in Maple very frustrating, and that is presumably a 'core' package;
> having random stuff suddenly not work (let me be clear, because it was left
> behind by Sage core) would be even more so - as has been pointed out several
> times here.
>
> Sage users (and potential ones) I speak with want more than just the "basic"
> functionality, because they want something they can use throughout the
> curriculum and in their own research. There are other (good) tools for
> those who truly won't be doing anything beyond calculus.
>
> Now it's true that some material in Sage probably could have been in
> separate packages, as it's quite specialized - likely a lot of the
> sage-combinat stuff, the designs stuff, modular forms stuff (elliptic curves
> are actually more popular, I think). But then there's the opposite problem
> of finding out how to enforce that a package must compile with the most
> recent Sage. This is R's model, but R tends to have a very different type
> of package, one that implements something relatively narrow. Also, we don't
> have the auto-testing resources of R.
>
> Given that, I'd definitely rather have the kitchen sink, including that
> stuff, in Sage - as a *user*, not developer. I know that whatever is in
> Sage will stay in it, and it will all be there.

Hi,

I think these are some thoughtful comments, but I also think this is
partly missing the point. This discussion isn't (necessarily) about
how Sage is packaged and presented for the average user. One can
certainly put together a metapackage containing all the bells and
whistles and optional dependencies that one would ever (and never)
need.

So when bundling a SageMath "product" for users, there's not so much
harm in throwing in the kitchen sink (generally, I don't think). I
know William doesn't want to cut users off from functionality.

I interpreted William's original message as being about how Sage is
*developed*, and the current kitchen sink model is I think
unsustainable for development. System integration can be hard, but at
a certain point it's less hard than developing an overly large single
project, IMO.

Best,
Erik

[Erik Bray]

On Tue, Apr 12, 2016 at 7:33 PM, William Stein <wst...@gmail.com> wrote:
> On Tue, Apr 12, 2016 at 7:11 AM, kcrisman <kcri...@gmail.com> wrote:
>> This has been an interesting thread. In the end, I think that some (or a
>> lot) of Sage's attractiveness to end users goes away if it becomes a bunch
>> of possibly-updated packages that might or might not work with a current
>> version of Sage. I always found the "with(plots)" syntax (or whatever it
>> was) in Maple very frustrating, and that is presumably a 'core' package;
>> having random stuff suddenly not work (let me be clear, because it was left
>> behind by Sage core) would be even more so - as has been pointed out several
>> times here.
>
> I am a little annoyed, since this completely misunderstands my proposal.
>
> 1. My proposal was to make it easier for people to develop new code
> independently of the core sage library. This in itself has nothing
> to do with taking away from the existing library or in fact changing
> it in any way (at least initially).
>
> 2. Moreover, to directly address your concern, if 3d plotting (say)
> were split off as a separate Python package/library, that does *not*
> imply that when you download and install Sage, or start it, that you
> can even tell the difference. It doesn't mean that the normal visible
> public API of Sage changes
> at all. Why do you think otherwise? We would just include that
> Python library as a standard package, just like the hundred other
> standard packages. Volker has mentioned several times how Python
> enables doing this sort of thing pretty easily already.

Okay, and this is what I get for not reading further downthread before
replying--I see we're in agreement here.

Erik

[Erik Bray]

Sorry to keep bringing up Astropy here--it by no means does everything
perfectly--but it is a good analog to Sage in many ways due to general
similarities between scientific software communities.

Anyways one thing that Astropy provides for Affiliated Packages is a
"package template" [1] that allows all Affiliated Packages to be built
and packaged more or less the same way. They aren't *required* to use
the package template, but the vast majority do because it's
convenient--it sets them up with the same testing and documentation
tools that the Astropy core package uses it.

And even more importantly (and the reason I bring this up) is that it
includes templates for configuring their package to be built and
tested on continuous integration platforms like Travis [2] and
AppVeyor [3]. The default templates include Astropy as a build/test
dependency, and include separate build job configurations for using
both the latest stable release as well as the latest version from git.
This means that affiliated package authors typically find out quickly
if a change in Astropy breaks their code too. This doesn't happen
that often due to the slow deprecation process, but it does happen
(probably most commonly if some import changes). We can then work
with the maintainers of the individual affiliated packages to resolve
the issue--whether it's putting the right astropy-version-specific
code in their package, or backing out changes or providing better
backward-compatibility in Astropy.

Unit testing and continuous integration really make this a mostly
painless process, and by providing package authors with the tools to
do these things easily we promote best practices.

Best,
Erik

[1] https://github.com/astropy/package-template
[2] https://travis-ci.org/
[3] https://www.appveyor.com/

[Erik Bray]

The Jupyter people can be more specific I'm sure, but it mostly comes down to:

1) Unit tests
2) Continuous integration
3) Package templates, to reduce reinventing the wheel with every package
4) Communication

[Jeroen Demeyer]

A really important point (which so far hasn't been addressed) is how to
deal with breakages of affiliated packages. I see two ways:

1. If I make a change to some core package, it is my responsibility to
ensure that no affiliated package breaks.

2. It is the responsibility of the affiliated package to fix breakages.

I think both of the above have problems:

With 1, the author of the core package might not have access to the
affiliated package: how do you force your affiliated package to accept a
needed change? This is the current situation with Sage and SageNB.

With 2, breakages will need to fixed after the fact (after the change to
the core package has been merged already), leading to occasional
breakages for end users. This is the current situation with Sage
optional packages.

[kcrisman]

I think these are some thoughtful comments, but I also think this is

Thanks.

 

partly missing the point.  This discussion isn't (necessarily) about
how Sage is packaged and presented for the average user.  One can
certainly put together a metapackage containing all the bells and
whistles and optional dependencies that one would ever (and never)
need.

I think I more or less already replied to this that I have no philosophical objection to the proposed project, just some practical concerns, so I won't repeat myself, especially since it seems that you are among the practical resources who might be brought to bear on solving those practical concerns.  Which the whole community, I think, would look forward to - *especially* concerning Windows :) so thanks in advance!

PS I just saw Jeroen's most recent update, and that is a good summary of the practical problem.  Sage is not like other software where a breakage (even a trivial one) just means that package doesn't work; it could affect a lot of other stuff, especially the "kitchen sink".

[Erik Bray]

On Thu, Apr 14, 2016 at 3:21 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> A really important point (which so far hasn't been addressed) is how to deal
> with breakages of affiliated packages. I see two ways:

These are good questions.

> 1. If I make a change to some core package, it is my responsibility to
> ensure that no affiliated package breaks.
>
> 2. It is the responsibility of the affiliated package to fix breakages.

Those are the most obvious but there are other ways. A variant on #2
is that it's the responsibility of the affiliated package's maintainer
to perform integration testing of their package and report issues that
result from upstream changes. But whose responsibility it is to "fix
breakages" may depend on the particular issue.

I agree with the overall premise though.

> I think both of the above have problems:
>
> With 1, the author of the core package might not have access to the
> affiliated package: how do you force your affiliated package to accept a
> needed change? This is the current situation with Sage and SageNB.
>
> With 2, breakages will need to fixed after the fact (after the change to the
> core package has been merged already), leading to occasional breakages for
> end users. This is the current situation with Sage optional packages.

I agree, mostly. Both of these *are* problems.

Astropy doesn't do #1--that is, there is nothing that tests Astropy
core changes against all affiliated packages (rather, the affiliated
packages run their own tests against the latest Astropy core--the
inverse). I think there has been some discussion of adding affiliated
package tests to Astropy's main contiguous integration matrix and I
think that would be a good idea. It would also enable better
integration testing between affiliated packages (some already do
this--for example APLpy [1] depends on WCSAxes [2]--both are Astropy
affiliated packages and use (different) functionality from the Astropy
core library.

Implementing #1 would seem to require addressing problem #1. However,
this needs to be addressed either way--if some changes are made in the
core package that break affiliated packages then the affiliated
package *may* have to be updated regardless of how the breakage was
discovered. If truly nobody can maintain an affiliated package
anymore it might die. And that's a problem since it might mean loss of
functionality for users, but it's a less bad problem then having
unmaintained code sloshing around in the main package (and if a
package goes unmaintained it probably didn't have many, if any users
beyond its original author anyways). It should be pinned to require
the last versions of its dependencies that it did work with so that it
can be easily installed in a virtualenv if absolutely needed, and then
be sent out to pasture.

Problem #2 does result specifically from requiring affiliated packages
to do their own integration testing. That's why having more
integration testing for the core package would be good too. This is
rarely a huge showstopper though. These issues are almost always
caught in development and don't affect anyone but the affiliated
package author, and usually not for long, as their package will
normally still work with the latest stable version. In *some* cases
the development versions of an affiliated package will depend on the
development version of the core package, but in most of those cases
the affiliated package author is working closely with whoever is
working on the dependent features in the core package (or often are
the same person(s)).

Finally, in the rare cases where a release gets out that breaks
integration it can be addressed quickly by a patch release. Currently
Sage doesn't do patch releases, but it should. Especially if there is
a move toward increased modularization. But that's more a separate
issue.

Erik

[1] https://aplpy.github.io/
[2] http://wcsaxes.readthedocs.org/en/latest/

[Erik Bray]

On Thu, Apr 14, 2016 at 3:26 PM, kcrisman <kcri...@gmail.com> wrote:
>>
>> I think these are some thoughtful comments, but I also think this is
>
>
> Thanks.
>
>>
>> partly missing the point. This discussion isn't (necessarily) about
>> how Sage is packaged and presented for the average user. One can
>> certainly put together a metapackage containing all the bells and
>> whistles and optional dependencies that one would ever (and never)
>> need.
>
>
> I think I more or less already replied to this that I have no philosophical
> objection to the proposed project, just some practical concerns, so I won't
> repeat myself, especially since it seems that you are among the practical
> resources who might be brought to bear on solving those practical concerns.
> Which the whole community, I think, would look forward to - *especially*
> concerning Windows :) so thanks in advance!

Got it. I think they're all valid concerns that need to be addressed.
I (or preferably someone else, but I can do it if no one else does)
should go back over this thread and summarize the points and
counter-points.

IMO most of the counter-points are fine to bring up but mostly consist
of specific examples of where something went wrong somewhere one time.
I think nobody is claiming that any development philosophy will lead
to a problem-free process.

> PS I just saw Jeroen's most recent update, and that is a good summary of the
> practical problem. Sage is not like other software where a breakage (even a
> trivial one) just means that package doesn't work; it could affect a lot of
> other stuff, especially the "kitchen sink".

I disagree that Sage is all that special. Or at least, I don't
believe there's any need for it to be, whether or not it is currently.

Best,
Erik

[Francesco Biscani]

On 14 April 2016 at 15:51, Erik Bray <erik....@gmail.com> wrote:

I disagree that Sage is all that special.  Or at least, I don't
believe there's any need for it to be, whether or not it is currently.

If the past is any indication, you will find some cultural resistance about this point in the Sage community. Sometimes it seems like Sage lives in a reality distortion field in which normal software development practices do not apply :)

William's change of heart about modularization is a relatively recent development, which - I speculate - might have been brought forward by his increased involvement with the Python (and non-Python) ecosystems while working on SMC.

FWIW, I think you are doing the right thing and wish you all the best luck in the effort :)

Cheers,

  Francesco.

[Michael Orlitzky]

On 04/14/2016 09:21 AM, Jeroen Demeyer wrote:
> A really important point (which so far hasn't been addressed) is how to
> deal with breakages of affiliated packages. I see two ways:
>
> 1. If I make a change to some core package, it is my responsibility to
> ensure that no affiliated package breaks.
>
> 2. It is the responsibility of the affiliated package to fix breakages.
>

It would be have to be (2), and that would be the distinction between an
affiliated package (one that depends on Sage) and one that gets added to
Sage itself. Both should still be an option: people get less upset if
they make a choice and it bites them in the ass than they would get if
you forced the same choice (that they would have made anyway) on them.

Breakages will still occur, but they can be managed. If it becomes
possible to package Sage without all of third-party software, then it
becomes possible for the people who depend on Sage to depend on a
particular version (and enforce that with pip, or a package manager, or
whatever). That's not possible now because Sage hasn't been packaged
most places. Then all we need is a little discipline to not break things
too badly between releases.

To do that, we have to figure out what the "public API" is, and how much
we're allowed to change it. Major versions (e.g. sage-6.x to sage-7.x)
should be "anything goes." With the current versioning scheme, minor
releases should be things like build-system changes, method additions,
bug fixes, and dependency updates. (This needs more careful thought than
I've given it.) The release manager would have to decide which is which,
and maintain two separate branches. The current "develop" branch
corresponds to "the next major release", and we would need a new branch
for "the next minor release" containing only those tickets that don't
break the public API.

Nobody can write code depending on a library that is guaranteed to work
forever with all versions of that library. But "anything can change at
any time" isn't much of a promise to those people. Having a more
semantic versioning scheme will make things a lot less painful. And the
option to commit straight into Sage itself is there for the people who
are willing to give up some control for stability.

[Jeroen Demeyer]

On 2016-04-14 15:46, Erik Bray wrote:
> If truly nobody can maintain an affiliated package
> anymore it might die. And that's a problem since it might mean loss of
> functionality for users

A logical conclusion from the above that it's a simply a bad idea to
split up Sage into separate packages unless somebody really cares about
maintaining each of those packages. I care about cysignals since I wrote
most of it, but if nobody cared we should not have split it up.

The converse should also be true: if a package is no longer actively
worked on, it should be folded back into Sage core. I think this would
currently be a good idea for SageNB.

Jeroen.

[Erik Bray]

On Thu, Apr 14, 2016 at 5:25 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> On 2016-04-14 15:46, Erik Bray wrote:
>>
>> If truly nobody can maintain an affiliated package
>> anymore it might die. And that's a problem since it might mean loss of
>> functionality for users
>
>
> A logical conclusion from the above that it's a simply a bad idea to split
> up Sage into separate packages unless somebody really cares about
> maintaining each of those packages. I care about cysignals since I wrote
> most of it, but if nobody cared we should not have split it up.

I think we're dealing with different premises because that's not the
logical conclusion I come to. Sage already has the problem of large
chunks of code that are effectively unmaintained and create a
maintenance burden on anyone serious about maintaining sage. Their
interfaces whither, and become inconsistent with the rest of the
package. It's dead weight.

cysignals is a different story altogether, since it was split out for
a different reason--that it has functionality not particular to sage
and not intricately tied to other parts of sage. It's worth splitting
out because it can benefit the wider Python community (something most
of sage-devel seems to think is on another plane of existence ;) and
in turn cysignals (and sage) can benefit from that.

matplotlib started out as part of a package of EEG data analysis tools
(I think it was always developed as a separate library but I'm not
sure--regardless that was the motivation behind it). If it never grew
beyond the specific domain that motivated it it would no longer exist.

> The converse should also be true: if a package is no longer actively worked
> on, it should be folded back into Sage core. I think this would currently be
> a good idea for SageNB.

Why? I thought the goal was to deprecate sagenb? Easier to do if
it's separate--eventually you say "Okay, the last version of sage that
sagenb works with is X.Y" then let it whither and die.

[Jeroen Demeyer]

On 2016-04-14 17:38, Erik Bray wrote:
> Sage already has the problem of large
> chunks of code that are effectively unmaintained and create a
> maintenance burden on anyone serious about maintaining sage. Their
> interfaces whither, and become inconsistent with the rest of the
> package. It's dead weight.

I disagree with the above. It's not necessarily a problem to have
unmaintained-but-still-working code.

> Why? I thought the goal was to deprecate sagenb?

At long as SageNB is still supposed to work (even if deprecated),
somebody has to do some minimal maintenance to keep it working.
Currently, it seems that nobody wants to do this. If SageNB would become
again part of Sage, then we can do this minimal maintenance as part of
normal Sage development.

[William Stein]

On Thursday, April 14, 2016, Francesco Biscani <blues...@gmail.com> wrote:

On 14 April 2016 at 15:51, Erik Bray <erik....@gmail.com> wrote:

I disagree that Sage is all that special.  Or at least, I don't
believe there's any need for it to be, whether or not it is currently.

If the past is any indication, you will find some cultural resistance about this point in the Sage community. Sometimes it seems like Sage lives in a reality distortion field in which normal software development practices do not apply :)

William's change of heart about modularization is a relatively recent development, which - I speculate - might have been brought forward by his increased involvement with the Python (and non-Python) ecosystems while working on SMC.

I have had a strong interest in this since when I started psage over 6 years ago.  However I'm only now starting to complain loudly and repeatedly just because I'm seeing such a huge wasted opportunity.   Let's move out of the reality distortion field and apply normal software development practices!!!!!!

 

FWIW, I think you are doing the right thing and wish you all the best luck in the effort :)

Cheers,

  Francesco.

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--

[Jeroen Demeyer]

On 2016-04-14 20:35, William Stein wrote:

> However I'm only now starting to complain loudly and
> repeatedly just because I'm seeing such a huge wasted opportunity.

Instead of complaining, why don't you put together a more concrete and
technical proposal of what you actually want? Because this thread is
getting really long with a low fact/opinion ratio. Surely by far not
enough facts to convince me.

I also see a lot of discussion in this thread about general issues with
Sage and Sage development which are unrelated to modularisation. So,
when you do write that proposal, be sure to stick to problems which are
really caused by the monolithic nature of Sage.

I can believe that there are certain parts of Sage which could be split
up (you gave the example of 3D plotting and cysignals), but I think
that's only realistic for the non-mathematical parts of Sage. So I
wonder if you think that, say, elliptic curves should be a separate
package too?

[William Stein]

On Thu, Apr 14, 2016 at 12:23 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> On 2016-04-14 20:35, William Stein wrote:
>>
>> However I'm only now starting to complain loudly and
>> repeatedly just because I'm seeing such a huge wasted opportunity.
>
>
> Instead of complaining, why don't you put together a more concrete and
> technical proposal of what you actually want? Because this thread is getting
> really long with a low fact/opinion ratio. Surely by far not enough facts to
> convince me.
>
> I also see a lot of discussion in this thread about general issues with Sage
> and Sage development which are unrelated to modularisation. So, when you do
> write that proposal, be sure to stick to problems which are really caused by
> the monolithic nature of Sage.

Just to be clear: I personally won't be writing any such thing anytime
in the near future.

I guess my conclusion for now from this thread is not that I'm trying
to convince you of anything. However, as a said before, my
recommendation to anybody considering writing code for Sage is, to
quote Francesco Biscani, to "move out of the reality distortion field
and apply normal software development practices." Here's a concrete
example of this happening:

https://github.com/haochenuw/sage_modabvar

> I can believe that there are certain parts of Sage which could be split up
> (you gave the example of 3D plotting and cysignals), but I think that's only
> realistic for the non-mathematical parts of Sage. So I wonder if you think
> that, say, elliptic curves should be a separate package too?

I'm not lowering the fact/opinion ratio further...

William


--
William (http://wstein.org)

[Vincent Delecroix]

On 14/04/16 16:32, William Stein wrote:
> On Thu, Apr 14, 2016 at 12:23 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
>> On 2016-04-14 20:35, William Stein wrote:
>>>
>>> However I'm only now starting to complain loudly and
>>> repeatedly just because I'm seeing such a huge wasted opportunity.
>>
>>
>> Instead of complaining, why don't you put together a more concrete and
>> technical proposal of what you actually want? Because this thread is getting
>> really long with a low fact/opinion ratio. Surely by far not enough facts to
>> convince me.
>>
>> I also see a lot of discussion in this thread about general issues with Sage
>> and Sage development which are unrelated to modularisation. So, when you do
>> write that proposal, be sure to stick to problems which are really caused by
>> the monolithic nature of Sage.
>
> Just to be clear: I personally won't be writing any such thing anytime
> in the near future.
>
> I guess my conclusion for now from this thread is not that I'm trying
> to convince you of anything. However, as a said before, my
> recommendation to anybody considering writing code for Sage is, to
> quote Francesco Biscani, to "move out of the reality distortion field
> and apply normal software development practices." Here's a concrete
> example of this happening:
>
> https://github.com/haochenuw/sage_modabvar

Or

https://github.com/abelfunctions/abelfunctions
https://github.com/coulbois/sage-train-track

These packages are nearly impossible to found from the sagemath website!

Vincent

[William Stein]

On Thu, Apr 14, 2016 at 12:41 PM, Vincent Delecroix

Chris -- who wrote abelfunctions -- is a Univ of Wash grad student I
know. I recently ran into him and he told me that he had spent years
writing this package as a standard Python package depending on sympy
(mainly) and not Sage. However, it was getting quite frustrating to
finish abelfunctions, due to him needing things that are in Sage (I
think things like interval arithmetic?). He didn't want to develop
his code in a way that depended on Sage, because he wanted it to be
more widely available, and he clearly didn't see the value in having
the code packaged as part of Sage for only a few small things. This
is a great example of how our centralized monolithic approach is
overall not the best for the community at large.

I've cc'd Chris in case he wants to clarify this, since I might not
understand.

William

>
> Vincent

>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--

William (http://wstein.org)

[Chris Swierczewski]

(Disclaimer: I haven't yet read this entire thread when writing my response.)

Chris -- who wrote abelfunctions -- is a Univ of Wash grad student I
know.  I recently ran into him and he told me that he had spent years
writing this package as a standard Python package depending on sympy
(mainly) and not Sage.  However, it was getting quite frustrating to
finish abelfunctions, due to him needing things that are in Sage (I
think things like interval arithmetic?).   He didn't want to develop
his code in a way that depended on Sage, because he wanted it to be
more widely available, and he clearly didn't see the value in having
the code packaged as part of Sage for only a few small things.    This
is a great example of how our centralized monolithic approach is
overall not the best for the community at large.

I've cc'd Chris in case he wants to clarify this, since I might not
understand.

I remember discussions of this "modularized Sage" proposal way back when I was an undergraduate so I don't think it's anything new. In fact, hearing William talk about these issues back then is, in part, what prompted me to design my Sage package the way I did.

That being said, I can only really comment on my own experience developing Abelfunctions (http://github.com/abelfunctions/ableufncitons). As William mentioned, I started writing Abelfunctions such that only depended on very common core Python packages. Anaconda is very popular in the applied mathematics community and many of the key scientific computing packages I needed (e.g. Numpy, Scipy, networkx) are included in Anaconda as well as in Sage. This design seemed like a good idea at the time since my code could reach as many people as possible.

However, I slowly realized that Sympy's limitations in *efficiently* performing exact arithmetic with polynomial roots would prevent me from completing my software. After attempting to basically write my own version of QQbar in Sympy I made the very difficult decision, after four+ years of my Ph.D. writing Sympy workarounds, to completely rewrite my code "using Sage". I'm glad I did even though it added time to my Ph.D. --- a very small amount of time compared to the amount wasted trying to get Sympy to do what I wanted. So I am definitely glad that something like Sage is around to provide those open source tools I needed.

I didn't pursue writing Abelfunctions into Sage or even making Abelfunctions an "optional" or "experimental" listed spkg for several reasons:

1. It is still in very active development --- My quick (read: several month) total rewrite of Abelfunctions to use Sage reintroduced bugs that I squashed in the Python version. This is somewhat evidenced in the GitHub issues page. Because I'm constantly fixing and adding things as I finish my thesis it seems bothersome to have to push a new spkg after every commit. (Sure, I can write a script to try and do that.)

2. It is a niche product --- The mathematical communities I work with know about Abelfunctions and they have spread the word of its existence to those who might be interested. That's great. But I never assumed that it should be a standard part of Sage. Sage already takes too much time to compile so why bloat it with something that <1% of Sage users would ever touch? (Though, my implementation of Puiseux series in Abelfunctions is now making it's way into the Sage codebase.)

3. GitHub is my resume --- Headhunters have seen that I write code. I doubt they scan trac.sagemath as much as they do GitHub. Being able to put my code up on GitHub has made it clearer to others and to myself what my contribution to the math community has been. Not that I'm against sharing in an open-source project but to have Abelfunctions Trac tickets buried in the mound of other Sage issues didn't seem like a clear way to get people to work on this small project at the time.

Overall, developing Abelfunctions as an "external" Sage package has been pretty effective. I have some minor complaints, including the fact that the standard "$ sage setup.py install" doesn't seem to work on all platforms and trying to get TravisCI to download and install Sage before being able to test my code is a bit of a pain. If people who want to use it are having trouble finding it then it's my fault, not Sage's, since I suck at advertising myself. (Writing this post about my own code was difficult enough.) I think if enough people write packages this way we will see what works and what doesn't thus putting the whole "let's refactor Sage to make it modular" into a more concrete setting.

--

Chris

[kcrisman]

> These packages are nearly impossible to found from the sagemath website!

Chris -- who wrote abelfunctions -- is a Univ of Wash grad student I
know.  I recently ran into him and he told me that he had spent years

This is a good point; we could use more infrastructure for supporting the advertising of packages of this kind.
 

writing this package as a standard Python package depending on sympy
(mainly) and not Sage.  However, it was getting quite frustrating to
finish abelfunctions, due to him needing things that are in Sage (I
think things like interval arithmetic?).   He didn't want to develop
his code in a way that depended on Sage, because he wanted it to be
more widely available, and he clearly didn't see the value in having
the code packaged as part of Sage for only a few small things.    This

is a great example of how our centralized monolithic approach is
overall not the best for the community at large.

I don't know whether this relevant, but it reminds me of people who write code depending on Mathematica (for which there are many such packages, not least of which knot stuff)?   Obviously it won't end up in Mma proper, but people will depend upon Mma.  And such packages definitely seem to get a heavy amount of use.