Would it be a good idea to start working on a module on Population Dynamics?

[Nikhil Pappu]

Hey guys! I am new here and I am trying to get used to SageMath and the community. I was wondering if it would be a good idea to start working on a module on mathematical models of Population Dynamics.

[Dima Pasechnik]

On Saturday, July 22, 2017 at 10:04:39 AM UTC+1, Nikhil Pappu wrote:

Hey guys! I am new here and I am trying to get used to SageMath and the community. I was wondering if it would be a good idea to start working on a module on mathematical models of Population Dynamics.

What do you think are tools provided by Sagemath, but missing in the more generic generic Python universe, that are necessary for the task?

[Nikhil Pappu]

SageMath is a fully featured math library so I believe it must have extensive modules on Calculus and Statistics which are necessary for the task. A lot of generic Python math frameworks I am familiar with are included in SageMath so that is a huge plus.

I found modules on Applied Mathematics like Game Theory, Coding Theory, Finance etc in SageMath so I thought a module on Population Dynamics would be a useful addition. Correct me if I am wrong.

I want to study more on the topic and start working on creating a module (a simplistic one at first). Can you tell me if this is a good idea?

[William Stein]

Nikhil, 

Thanks – could you do this Google search

 "population dynamics" python

study the results, and report back?   

William

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--

-- William Stein

[Nikhil Pappu]

William,

These papers are quite fascinating :

ftp://ftp.ntua.gr/mirror/python/pycon/papers/populationsim.html

http://jasss.soc.surrey.ac.uk/16/1/8.html

This package studies population dynamics of mosquitoes : https://pypi.python.org/pypi/albopictus/0.3

I found some small implementations and case studies but didn't find many extensive packages with user friendly API's. 

[Nikhil Pappu]

Also a lot of the implementations I came across were highly specific and I feel that there can be a wider and more general module.

On Saturday, July 22, 2017 at 11:11:14 AM UTC-5, William wrote:

[Robert Dodier]

On 2017-07-22, Dima Pasechnik <dim...@gmail.com> wrote:

> What do you think are tools provided by Sagemath, but missing in the more
> generic generic Python universe, that are necessary for the task?

Not OP, but we are dealing with proposals of this flavor over in the
Maxima project and I'd like to put in my 2 cents here.

As to the special benefit of Sage here, generic Python can provide
numerical solutions but it's weak on symbolic representations --
essentially one would have to reinvent that wheel. I think the
combination of symbolic and numerical calculations is a huge win.

So I think such a project is a great idea. However, I wonder whether it
should be absorbed into Sage proper. What I've settled on, after
repeated proposals over in Maxima, is that such projects should be
managed separately, by their authors, hosted at Github or whatever, but
also there should be a simple, convenient way to install and load it
into Sage. I think there exist such mechanisms for R, Python, Ruby,
etc. -- maybe Sage can use Python's with or without modification.

For what it's worth. I am interested to hear how such things are handled
in Sage. It seems like a problem that needs to be solved in some form
in various projects -- how to create an ecosystem without bogging down
the main project.

best

Robert Dodier

[Dominique Laurain]

I fully agree with Robert Dodier : sagemath cannot include the multi-purpose packages round around the computer world, but the question is "If a package cannot be integrated into sagemath community code, does it exist a clear process for other communities (physics, geometry, statistics,..) to link with sage in order to make core sagmath + extrapackages, be runnable on SMC-sagecell-standalone sagemath...".

Process means : what are recommended code management (github ?..) ? what are requirements or dependencies (python2 or python 3) ? exist sand-box to run in protected mode, core sagemath+extra packages ?..

I do euclidean geometry code with sagemath, and I have wondered many times, if my code could be shared with the sagemath community. After learning about the sagemath dev process (issues,peer review and so on) I resigned for entered into it because of the needed extra-work (documentation, examples, doctests..) which is very time consuming. I am now more in "for future times, I will set a github project and make it public and evolving".

Dominique

[Dominique Laurain]

@Nikhil : 

Sorry to disappoint you , but I was more "fascinated" by Conway's game of life modeling.

What William did'nt point out, that the "work" you are hoping for,..,you can do it yourself.

If you have a sagemath cloud account, you simply install your package in one of your project...then start to code (GUI, doc, worksheets,...).Good luck.

Help for installing python package is somewhere near : https://github.com/sagemathinc/cocalc/wiki/Programming

Dominique

[mforets]

for module development and distribution, i think that the sage_sample project is the way to go! it explains how to do continuous integration with any sage (stable) version, and it allows installing your module in a *very* convenient way (just do: $ sage -pip install --upgrade -v git+https://github.com/... ).

however, i don't know of a "central" place linking to sage-related projects, that can be explored by keywords/topics...

[Marc Masdeu]

I'm just back from Sage Days 87 in Burlington, VT, where I gave a talk on how to deal with this issue, precisely using sage_sample (you can find the slides here https://www.slideshare.net/mmasdeu/distributing-sage-python-code-the-right-way).

Participants seemed to buy the solution. I think that for this to work it is essential that at least two things happen very quickly:

1) A good number already existing projects migrate to this method. I have contributed two projects and there is a handful of others, but there is lots of code out there that is "distributed" in various ad-hoc ways.

2) Proper documentation is put in the www.sagemath.org website so that anyone looking for this (like the OP) find it at first sight.

Also, Samuel Lelièvre was thinking of how to compile a list of such packages. The idea was to have a simple form with predefined fields so that code developers could add themselves to the list. This would then be populated (manually?) on to some official section in sagemath.org. Hopefully he'll be able to add more on this.

[Lee Worden]

Nikhil, you may find this module useful, which includes a class for population dynamics models that are implemented by ODEs: https://github.com/worden-lee/SageDynamics. The structure is intended to be extensible to more forms of dynamical systems and population dynamics models. It uses Sage's equation solver and other features, but I've kept it as a python module maintained separately from sage (I'm open to suggestions about better ways).

Lee Worden

[Dominique Laurain]

@Marc : good work and good ideas.. :-)

Why using the "unstable" http://k8s.sagemath.org (see page 5)  and why no "public" access to worksheets (darmonpoints,...) ?

Dominique.

[William Stein]

On Mon, Jul 24, 2017 at 12:16 PM Dominique Laurain <dominique...@orange.fr> wrote:

@Marc : good work and good ideas.. :-)

Why using the "unstable" http://k8s.sagemath.org (see page 5)

It’s a custom server with tons of CPU bought with a grant setup for sage days and sage dev; we had projects really maxing it out, and using 100’s of GB’s of disk, etc. 

 and why no "public" access to worksheets (darmonpoints,...) ?

I think we would be happy to make things public.   I’ve cc’d David Roe, who is organizing.   Personally, I think it would be better though to copy all content that should be public to a normal cocalc.com project, and make it public there, since that way the URL will be valid forever (since the intention with CoCalc is immortallity, but k8s.sagemath.org is more of a one-off university hosted setup). 

Dominique.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--

-- William Stein

[Marc Masdeu]

> I think we would be happy to make things public.   I’ve cc’d David Roe, who is organizing.   Personally, I think it would be better though to copy all content that

> should be public to a normal cocalc.com project, and make it public there, since that way the URL will be valid forever (since the intention with CoCalc is

> immortallity, but k8s.sagemath.org is more of a one-off university hosted setup). 

I understand that this is still discussing page 5 of the slides. I'm sorry for the confusion, these slides were meant to be used at SD87 and the reason they had links on them was so that I could click on them during the presentation in case I lost the tabs in the browser. The "demo" on k8s consisted in creating an empty project and installing the darmonpoints package through pip, to show how seamless all of it was. So there is nothing really interesting on that project, really (not even the darmonpoints code, since I uninstalled the package as part of the demo, too).

Again, what would really be useful is to document very thoroughly the sage_sample project (the pull requests should be reviewed, too ;-) ), and possibly changing its name to something more google-friendly (e.g. sage-package-template, sagemath-package-template, sage-package-boilerplate,...).

[William Stein]

In case it was never mentioned here, there is also a large list of
external sage-dependent packages here:
https://wiki.sagemath.org/SageMathExternalPackages

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--

William (http://wstein.org)

[Travis Scrimshaw]

I disagree with the conclusion of Marc's slides; in particular that it is the "right way". In fact, I am going to be bold and propose that only adding new code like that is the wrong way for developing Sage. Following the logic, we should take all of the code in Sage, put in into (separate) 3rd party packages, and have Sage be the bindings between some of the base dependencies. However, these different parts of Sage can have very non-trivial dependencies and local doctests do not necessarily have. Moreover, there is absolutely no quality control measures put in place: anyone can put code in a 3rd party package with little-to-no documentation or doctests. So code is likely to break, not work, and/or be unusable. That really is the only advantage I can see to having a 3rd party package: you do not have to do Sage's review process.

Now, for distributing code with the eventual goal to merge into Sage, this is a good methodology. However, it still does not replace a good trac ticket with code that should be there for people to review. Additionally, if you do all of the stuff to bring it to Sage's standards, how much harder is it to submit it as a trac ticket for integration into Sage? If it languishes in needs review, then users can use it (and hopefully one of them will eventually review it, but that seems unlikely). We also do not have the infrastructure to do something like R with package management. IDK what QC measures R has in place to fairly assess their process.

In case it was never mentioned here, there is also a large list of
external sage-dependent packages here:
https://wiki.sagemath.org/SageMathExternalPackages

Case in point, how many of those external packages actually work with 8.0? Is anyone actually testing them? SageManifolds has now been fully integrated into Sage proper, so it probably should not be on the list. I am suspect that a number of them will work on 8.0 due to internal changes in Sage. Train tracks are now in the process of getting merged into Sage.
 

[Vincent Delecroix]

You are making several assertions

1) Sage source code is of good quality
2) We do not test "external packages"
3) Adding and maintaining source code in Sage is simple

Let me discuss them in order.

In my opinion, the most powerful part of Sage comes from its third party
libraries (such as NTL, flint, pari or singular). The Python user
contributed code is often slow and of bad quality compared to the 4
mentioned libraries.

Let me add cypari2 [1]. This used to be code inside Sage and it is now a
project on github (and also a standard Sage package). cypari2 can be
installed in a standard Python installation. and is tested on a regular
basis within Python2, Python3 and Sage. Since it is an independent
project, its doctest coverage is actually better!

The code quality is up to the project developers and not the consequence
of the peer review process that we have in Sage. The only advantage I
see in this process is its friendlyness (it helps users to become
developers) and the helpful automated doctesting.


The examples that I developed above concerned somehow the "kernel" part
of Sage and not specialized code that a researcher is likely to develop
for her purpose. Let me use the same example as you did: the statistics
software R [2]. They have a database of ~10000 external packages that
are tested on a regular basis. I don't see why this is something we
could not afford in Sage. pip does support installation/uninstallation.
It would be simple to pick the packages one by one and use a patchbot to
test them.


For the last point, including code in Sage is dependent on reviewers.
Since this is on a volontary basis, some important tickets get ignored.
Because too technical, because of lack of time, ... On the other hand,
some code gets included too quickly. We now have a huge amount of code
that is a pain to maintain. There are some bugs that are pending since
many years and are note likely to be solved in the next future. Each
time Sage grows it makes it harder to have them fixed.


As a conclusion, I would be much happier with

1) A Sage kernel as small as possible with strictly specified API and
slow release cycle.

2) External packages registered in a database and tested on a regular
basis by patchbots. The release cycle is to be decided by developers.


[1] https://github.com/defeo/cypari2
[2] https://www.r-project.org/

On 25/07/2017 08:43, Travis Scrimshaw wrote:
> I disagree with the conclusion of Marc's slides; in particular that it is
> the "right way". In fact, I am going to be bold and propose that only
> adding new code like that is the wrong way for developing Sage. Following
> the logic, we should take all of the code in Sage, put in into (separate)
> 3rd party packages, and have Sage be the bindings between some of the base
> dependencies. However, these different parts of Sage can have very
> non-trivial dependencies and local doctests do not necessarily have.

> Moreover, there is absolutely *no* quality control measures put in place:

> anyone can put code in a 3rd party package with little-to-no documentation
> or doctests. So code is likely to break, not work, and/or be unusable. That
> really is the only advantage I can see to having a 3rd party package: you
> do not have to do Sage's review process.
>
> Now, for distributing code with the eventual goal to merge into Sage, this
> is a good methodology. However, it still does not replace a good trac
> ticket with code that should be there for people to review. Additionally,
> if you do all of the stuff to bring it to Sage's standards, how much harder
> is it to submit it as a trac ticket for integration into Sage? If it
> languishes in needs review, then users can use it (and hopefully one of
> them will eventually review it, but that seems unlikely). We also do not
> have the infrastructure to do something like R with package management. IDK
> what QC measures R has in place to fairly assess their process.
>
> In case it was never mentioned here, there is also a large list of
>> external sage-dependent packages here:
>> https://wiki.sagemath.org/SageMathExternalPackages

>> <https://www.google.com/url?q=https%3A%2F%2Fwiki.sagemath.org%2FSageMathExternalPackages&sa=D&sntz=1&usg=AFQjCNFVe1YmfRWENzfRockIeaZD5qCY7w>

[Eric Gourgoulhon]

Hi Travis,

Le mardi 25 juillet 2017 08:43:47 UTC+2, Travis Scrimshaw a écrit :

 SageManifolds has now been fully integrated into Sage proper, so it probably should not be on the list.

 I've just removed it ;-)

Regards,

Eric.

[Dominique Laurain]

@ Marc and William : thanks for the provided information ... I am not interested with darmonpoints either, and I agree : it will be more useful to improve the sagemath template project. For making sagedays projects, public, I think it is interesting matter too ...for "examples" about to use the sagemath + various environments (Cocalc,..). The way how to do it, i don't know...creating a Cocalc "sagedays-public" user to collect various worksheets (read access only except for admin)  ?

Dominique.

[Travis Scrimshaw]

You are making several assertions

1) Sage source code is of good quality
2) We do not test "external packages"
3) Adding and maintaining source code in Sage is simple

Let me discuss them in order.

In my opinion, the most powerful part of Sage comes from its third party
libraries (such as NTL, flint, pari or singular). The Python user
contributed code is often slow and of bad quality compared to the 4
mentioned libraries.

These are packages we have built upon, not that are built upon Sage. So this is completely unrelated to my point about the code quality of Sage.

Let me add cypari2 [1]. This used to be code inside Sage and it is now a
project on github (and also a standard Sage package). cypari2 can be
installed in a standard Python installation. and is tested on a regular
basis within Python2, Python3 and Sage. Since it is an independent
project, its doctest coverage is actually better!

Again, cypari2 was not really built upon Sage, but instead something that should exist as a standalone project that Sage could then build upon. So this is a non-example.

The code quality is up to the project developers and not the consequence
of the peer review process that we have in Sage. The only advantage I
see in this process is its friendlyness (it helps users to become
developers) and the helpful automated doctesting.

I am assuming by "this process" is the Sage peer review. It also does usually improve the quality of the code and the documentation. It also means that we only need to have one release manager who merges code into master.

The examples that I developed above concerned somehow the "kernel" part
of Sage and not specialized code that a researcher is likely to develop
for her purpose. Let me use the same example as you did: the statistics
software R [2]. They have a database of ~10000 external packages that
are tested on a regular basis. I don't see why this is something we
could not afford in Sage. pip does support installation/uninstallation.
It would be simple to pick the packages one by one and use a patchbot to
test them.

That is precisely what could be a problem. Doing things one-by-one means you may not see how one dependency affects another. Also, what are the rules? If B is based on A, do we require that A does not break anything in B or is B responsible for any downstream changes in A? If it is the former, then you have a useless extra layer because there is no point in have separate modules. It is functionally acting as one complete unit. If it is the latter, then there is no guarantee that stuff will not break.

However, what quality control measures does R have in place? Does it arbitrarily allow packages to break? What do they do with dependencies? How much duplication do they have? IIRC, they also have better tools in place for package management; something we would need to do develop if we decided to break Sage up into little bits and pieces.
 

For the last point, including code in Sage is dependent on reviewers.
Since this is on a volontary basis, some important tickets get ignored.
Because too technical, because of lack of time, ... On the other hand,
some code gets included too quickly. We now have a huge amount of code
that is a pain to maintain. There are some bugs that are pending since
many years and are note likely to be solved in the next future. Each
time Sage grows it makes it harder to have them fixed.

How much of that code is really a maintenance burden? Would any of those bugs actually be fixed if you go to external packages? How many of them would actually be found, much less reported (to the correct place)? If code being included too quickly is a problem, then why are you suggesting a method to include code even faster with less QC measures?

As a conclusion, I would be much happier with

  1) A Sage kernel as small as possible with strictly specified API and
slow release cycle.

  2) External packages registered in a database and tested on a regular
basis by patchbots. The release cycle is to be decided by developers.

By "developers," you mean the external package developer(s)? You need to specify what rules you want in place. Nobody has yet to actually specify anything like this.

Best,
Travis

[Simon King]

On 2017-07-25, Vincent Delecroix <20100.d...@gmail.com> wrote:
> The code quality is up to the project developers and not the consequence
> of the peer review process that we have in Sage.

Of course, if the project developers decide on using peer review and unit/doc
testing, then they'll get a good quality. And now assume the development
team of some package consists of a single person. Difficult to do peer
review within the team, then...

> As a conclusion, I would be much happier with
>
> 1) A Sage kernel as small as possible with strictly specified API and
> slow release cycle.
>
> 2) External packages registered in a database and tested on a regular

> basis by patchbots...

... and which the Sage development team supports by providing a peer review,
please, unless the developer team of the external package has enough resources.

Kind regards,
Simon

[Simon King]

On 2017-07-27, Travis Scrimshaw <tsc...@ucdavis.edu> wrote:
> I am assuming by "this process" is the Sage peer review. It also does
> usually improve the quality of the code and the documentation.

+1

> That is precisely what could be a problem. Doing things one-by-one means
> you may not see how one dependency affects another. Also, what are the
> rules? If B is based on A, do we require that A does not break anything in
> B or is B responsible for any downstream changes in A? If it is the former,
> then you have a useless extra layer because there is no point in have
> separate modules. It is functionally acting as one complete unit.

No. It just means that A has a stable and reliable API.

> How much of that code is really a maintenance burden? Would any of those
> bugs actually be fixed if you go to external packages? How many of them
> would actually be found, much less reported (to the correct place)? If code
> being included too quickly is a problem, then why are you suggesting a
> method to include code even faster with less QC measures?

+1

Cheers,
Simon