Indexing Sage external packages (was: Giving back to the community)

[Samuel Lelièvre]

Dear sage-devel,

This is a follow-up to a discussion on sage-support [0] in which

the original poster asks a question which I will summarize as:

  Having written some amount of Sage code for a project,

  how can I best package it and share it with the community?

Among the answers, Vincent suggests turning the code into a pip-installable

Python package for Sage, and points to a sample such package [1], made last

week at Sage Days 79 precisely to help people do that.

Vincent also points to the wiki page [2], which consists in a list

of external packages for Sage, some pip-installable, some not.

A bit later in the discussion, William writes:

> It would be great if somebody could create some sort of index

> of such packages, which we could link to (or include) on sagemath.org.

>

> This might eventually involve using some sort of tagging

> (or searching) of https://pypi.python.org/pypi and/or github.

> For now, this could just be a Github wiki page, which gets updated

> as we become aware of packages, and which we link to from sagemath.org.

>

> Thoughts? 

I'm moving the discussion to sage-devel.

The pages "SageMath external packages" [2] and "Code sharing workflow" [3]

were created in May 2016 following discussions on various Sage mailing lists

and at Sage Days 77 about modularization and packaging of Sage.

The page [2] was a preliminary attempt at "some sort of index of such

packages", and has a link to "Packages on PyPI matching 'sagemath'" [4],

which is intended as "using some sort of tagging or searching of PyPI".

It would certainly be nice to have a similar tagging or searching of GitHub,

bitbucket, and other code repositories, to detect external packages for Sage.

Probably the list at [2] should be turned into a better format, maybe a json

or yaml file or some appropriate database format, and be made accessible and

visible at some page on sagemath.org.

Entries in a database of Sage packages might have the following fields:

- package name

- alternate names, if applicable

- author(s)

- maintainer(s)

- short description

- long description

- keywords

- link to package home page

- link to code repository

- link to documentation

- license

- date package was created

- type of package (spkg, pip-installable, ...)

- relevant Sage trac tickets,

- discussions on sage-devel or other media

- maintenance status

- ...

We might also check how other communities index external packages,

for instance Astropy, GAP, Julia, ...

[0] https://groups.google.com/d/msg/sage-support/xsDt7TUjZkM/iOSBXfK7CgAJ

[1] https://github.com/nthiery/sage_sample

[2] https://wiki.sagemath.org/SageMathExternalPackages

[3] https://wiki.sagemath.org/CodeSharingWorkflow

[4] https://pypi.python.org/pypi?%3Aaction=search&term=sagemath&submit=search

Best, Samuel

[Vincent Delecroix]

Thanks Samuel for poping this discussion. on sage-devel.

I have only have random remarks:

- Among mathematical softwares that offer external packages the example
of R is particulary impressive (9598 packages) [1].

- Relying on Python packaging and PyPI might be too limited for Sage at
some point (tricky dependencies, Sage recompilation needed)

- We want to test these optional packages against each sage release and
for that purpose we need a kind of proper database (and extend the
patchbot functionalities)

- the wiki is a good start for a small manually handled database

Vincent

[1] https://cran.r-project.org/

[Jeroen Demeyer]

On 2016-11-28 20:14, Vincent Delecroix wrote:
> - Relying on Python packaging and PyPI might be too limited for Sage at
> some point (tricky dependencies, Sage recompilation needed)

I would say it's limited only because it deals with Python packages, not
other libraries. I don't quite get what you mean with "tricky
dependencies, Sage recompilation needed".

[Kwankyu Lee]

Hi,

I am thinking of developing an external package in the direction of  "experimental feature branches" among the work flows described in https://wiki.sagemath.org/CodeSharingWorkflow. Examples for this work flow, I think, could be found in  the numerous track tickets with milestone "sage-feature" or type "task". We may enhance the support for this work flow and these "feature" trac tickets by providing a separate repository. I envision the following:

We would have a "features trac" similar to the current development trac. A ticket in the features trac keeps a feature branch (Git) that provides a feature or extended functionality that can be merged to the Sage core at the user's build time. The ticket is not reviewed (in the development trac sense) and its branch is not supposed to be merged to Sage proper. The user can select a feature set at his/her build time. Then "make-sage-with feature_set" will fetch the feature branches from the features trac and merge them with the master branch of Sage core and start to make in the usual way. 

The features trac can provide

1. Description about the feature

2. info about authors or maintainers. 

3. Info about the latest Sage release with which the feature works. We may have "feature-bots" for doctesting.

4. info about dependencies, that is, other features that this feature depends on.

5. link to the host at which actual development occurs, like Github repos.

6. branch name in a standardized format. Example: "feature/klee/rings/super_field" where "klee" is the author's id.

Remarks:

R1. The feature branch should contain source code and documentation, of course. The documentation may have links toward the documentation of Sage core but not vice versa. After build, the user will have a single documentation as usual.

R2. The features in the features trac are either orthogonal or competent to other features in their functionality.

R3. Some parts of the present Sage library may be turned into features. For example, we may have "feature/sage/modular/abvar". 

What do you think? I don't know about technical difficulties in implementing this infrastructure. So would you comment?

[Jeroen Demeyer]

On 2016-12-07 14:52, Kwankyu Lee wrote:
> What do you think?

Why do this? I guess this will mostly lead to bitrotting code that would
better be merged into Sage.

[William Stein]

"According to the CRAN website, Currently, the CRAN package repository
features 5601 available packages. According to RDocumentation.org
there are 11794 R packages on CRAN, Bioconductor and GitHub at the
moment (November 20th 2016). Number of packages is rising with each
day."

R is the defacto standard in statistics and has millions of users.

In contrast, Sage has like 5 third party packages, and maybe 50K
users, and is certainly not the defacto standard for math.

Have some faith in people to be able to actually maintain code on their own.
I would way, way rather have 5000 external sage packages, with only
1000 well maintained, then 0. I applaud all efforts to give people
the tools and support they need to express themselves.

-- William


>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--
William (http://wstein.org)

[Kwankyu Lee]

On Wednesday, December 7, 2016 at 4:41:32 PM UTC+1, Jeroen Demeyer wrote:

Why do this? I guess this will mostly lead to bitrotting code that would
better be merged into Sage.

It seems a nice way to share code that enhances Sage in special areas, with least "friction". Users (and developers) don't want to have a behemoth Sage with all the code merged.

[William Stein]

There is also a massive amount of code people write themselves, which
could be generally useful, which they never really share at all
because we don't have an organized endorsed way of doing so. I hear
from tons of people about this often at the Sage booth at the Joint
Math meetings...

[Travis Scrimshaw]

If it is an independent package, then there should be minimal bit rot. It doesn't strictly force a minimal version of Sage onto the users and could potentially be easy to install for Sage users who do not want to mess with git/branches.

Best,
Travis

[William Stein]

On Wed, Dec 7, 2016 at 1:42 PM, Travis Scrimshaw <tsc...@ucdavis.edu> wrote:
>
>
> On Wednesday, December 7, 2016 at 9:41:32 AM UTC-6, Jeroen Demeyer wrote:
>>
>> On 2016-12-07 14:52, Kwankyu Lee wrote:
>> > What do you think?
>>
>> Why do this? I guess this will mostly lead to bitrotting code that would
>> better be merged into Sage.
>
>
> If it is an independent package, then there should be minimal bit rot.

The existence of many such packages will also motivate us to define
the external API of Sage more clearly.

> It
> doesn't strictly force a minimal version of Sage onto the users and could
> potentially be easy to install for Sage users who do not want to mess with
> git/branches.
>
> Best,
> Travis
>

[Erik Bray]

On Wed, Dec 7, 2016 at 7:41 PM, William Stein <wst...@gmail.com> wrote:
> On Wed, Dec 7, 2016 at 7:41 AM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
>> On 2016-12-07 14:52, Kwankyu Lee wrote:
>>>
>>> What do you think?
>>
>>
>> Why do this? I guess this will mostly lead to bitrotting code that would
>> better be merged into Sage.
>
> "According to the CRAN website, Currently, the CRAN package repository
> features 5601 available packages. According to RDocumentation.org
> there are 11794 R packages on CRAN, Bioconductor and GitHub at the
> moment (November 20th 2016). Number of packages is rising with each
> day."
>
> R is the defacto standard in statistics and has millions of users.
>
> In contrast, Sage has like 5 third party packages, and maybe 50K
> users, and is certainly not the defacto standard for math.
>
> Have some faith in people to be able to actually maintain code on their own.
> I would way, way rather have 5000 external sage packages, with only
> 1000 well maintained, then 0. I applaud all efforts to give people
> the tools and support they need to express themselves.

With our efforts at improving sage packaging, it will also be
possible, for example, for people to easily install sage into
development environments (e.g. via apt-get), which means they can
(finally!) use sage, for example, in free continuous integration
systems like Travis-CI and do development and testing like normal open
source projects.

At the end of the day the only thing that makes sage "special" is its
broken packaging (and to a lesser extent its poor guarantee of a
stable API). If that were fixed it would be fine for people to
develop third party packages against sage, and if they don't maintain
it that's their problem, not sage's. In the rest of the world, people
understand this.

[Paul Leopardi]

Thanks Samuel,
It probably would also be useful to have a bibliography type field.

i have been developing Sage and Python code for use in my exploration of bent functions and their strongly regular Cayley graphs as described in my recent (today) talk;
Classifying bent functions by their Cayley graphs, using Sage, 40 ACCMCC, 2016.
The code (and manuscript) repository is at  https://github.com/penguian/Boolean-Cayley-graphs
There is also a public SageMathCloud directory at
https://cloud.sagemath.com/projects/80f4c9e7-8a37-4f59-82e7-aa179ec0b652/files/Boolean-Cayley-graphs/
with extra SageMathCloud worksheets.

I am in the process of updating the code and especially the comments to conform to SageMath expectations, and porting some of the Sage code to Python, where sensible,
but some will remain as Sage code. If I create a PyPI package, what am i supposed to do with my Sage code that is outside of the boolean_cayley_graphs Python package directory? Also, how can I ask for people to review my code short of trying to submit it into Sage proper?
All the best,Paul