SageMath packaging status, 2018-09
Samuel Lelièvre
This pkgs.org page about SageMath
https://pkgs.org/download/sagemath
reveals a rather good state of SageMath packaging
in GNU/Linux distributions:
- Arch Linux: SageMath 8.3
- Debian Sid: SageMath 8.2
- Debian 9 (Stretch): SageMath 7.4
- Fedora Rawhide: SageMath 8.2
- Fedora 28: SageMath 8.0
- Fedora 27: SageMath 7.6
- Fedora 26: SageMath 7.6
- OpenMandriva Lx 3.0: SageMath 7.4
- Ubuntu 18.04: SageMath 8.1
Along with the new or continued efforts on Binder,
Conda, Docker, Gentoo, Homebrew, Nix, Sage Debian Live,
Virtual machines, Windows installer, this is very good!
Thanks to all involved, and keep up the good work.
Please don't hesitate to update the "Distribution" page
on the SageMath wiki (or tell me if you see something
that needs updating there).
Please don't hesitate to add links from the SageMath page
of each distribution's wiki to the following pages:
- the Distribution page on the SageMath wiki
https://wiki.sagemath.org/Distribution
- the pkgs.org page about SageMath
https://pkgs.org/download/sagemath
- the sage-packaging mailing list
https://groups.google.com/forum/#!forum/sage-packaging
- the "Release Monitoring" website which tracks releases
of hundreds of software components:
https://release-monitoring.org/
Please discuss anything that would make your life easier
as a packager here on sage-packaging, or on sage-devel.
Please open tickets as needed on Sage's trac.
If you spot Sage Days you would like to go to, register and
tell the organisers you are involved with packaging SageMath
for your distribution. A remote Sage days to train packagers
to the Sage development workflow could also be arranged.
See the list of upcoming Sage Days at:
https://wiki.sagemath.org/Workshops
In particular, I'm one of the organisers of
Software tools for mathematics
Koper, Slovenia
24 -- 28 September 2018
http://stm.famnit.upr.si/
https://wiki.sagemath.org/days97
and can allocate funding for SageMath packagers to attend
(and we could organise a special session).
Kind regards,
Samuel
Timo Kaufmann
Am Donnerstag, 6. September 2018 11:19:27 UTC+2 schrieb Samuel Lelievre:
> Dear SageMath packagers,
>
> This pkgs.org page about SageMath
>
> https://pkgs.org/download/sagemath
repology.org is a more exhaustive source:
https://repology.org/metapackage/sagemath/versions
I just noticed that the nixpkgs stable version isn't listed there because of
a naming issue (its named "sage-wrapper" on repology). I'll fix that sometime.
> Please discuss anything that would make your life easier
> as a packager here on sage-packaging, or on sage-devel.
My biggest gripes (already mentioned in other places too) are
- the brittleness of the doctests. Often times they check string output of python
libraries that is not intended to be stable. As a result they break at pretty
much every minor package version change.
- the number of package patches. I think sage-the-distribution makes it too easy
to patch dependencies without considering the consequences. Every patch to a
dependency is a burdon on all packagers and workarounds within sage should be
preferred. I already discussed this in sage-devel without much success.
- packaging concerns still seem second class
Minor issues:
- the commit policy is weird. Commits often aren't self contained changes that
pass the doctests but commits like "update package" "fix some doctests in file
x" "fix some more doctests in file y" instead. That makes cherry picking and
bisecting unnecessarily hard.
- the treatment of random failures. There are several open trac tickets with the
"random_fail" keyword [1]. In my opinion they should all be release blockers
and at least be marked as known failures before a release. Random failures in
package tests are a huge pain for continuous integration.
- the tight entanglement between "sage the library" and "sage the distribution".
I don't think implementing an ad-hoc package manager with make is a good idea.
> If you spot Sage Days you would like to go to, register and
> tell the organisers you are involved with packaging SageMath
> for your distribution. A remote Sage days to train packagers
> to the Sage development workflow could also be arranged.
> See the list of upcoming Sage Days at:
>
> https://wiki.sagemath.org/Workshops
>
> In particular, I'm one of the organisers of
>
> Software tools for mathematics
> Koper, Slovenia
> 24 -- 28 September 2018
> http://stm.famnit.upr.si/
> https://wiki.sagemath.org/days97
>
> and can allocate funding for SageMath packagers to attend
> (and we could organise a special session).
I'm actually a bit tempted since I will already be half way in that direction
on the 24th anyways. I'm not sure if I can take the additional time off and if
I would even add any value there.
> Kind regards,
> Samuel
By the way a contributor is nearly finished[2] adding darwin support to the sage
nix package. There still is some openblas/fflas_ffpack issue[3] however and it
has been a while since their last comment.
My current remaining patches (ignoring package update patches) are
- https://trac.sagemath.org/ticket/25358
- https://trac.sagemath.org/ticket/25316
Which is way less than I started out with. I hope to be able to package sage
unpatched in the future, but the prospects on getting those two issues resolved
seem bleak. Upstreaming the other patches worked pretty well.
Timo
[1] https://trac.sagemath.org/query?status=!closed&keywords=~random_fail
[2] https://github.com/NixOS/nixpkgs/pull/45364
[3] https://trac.sagemath.org/ticket/26130
Julian Rüth
thanks for sharing your thoughts here.
* Timo Kaufmann <eisf...@gmail.com> [2018-09-06 05:33:23 -0700]:
> > Please discuss anything that would make your life easier
> > as a packager here on sage-packaging, or on sage-devel.
>
> My biggest gripes (already mentioned in other places too) are
>
> - the brittleness of the doctests. Often times they check string output of python
> libraries that is not intended to be stable. As a result they break at pretty
> much every minor package version change.
several years and only recently started to get involved in packaging it.
As a developer I was not really aware of this problem and did not see
how that would matter. So I guess that most developers are like that.
It's probably something that reviewers should be a bit more picky about.
If you can make this precise with a few examples, we could include
something in Sage's developer guide.
> - the number of package patches. I think sage-the-distribution makes it too easy
> to patch dependencies without considering the consequences. Every patch to a
> dependency is a burdon on all packagers and workarounds within sage should be
> preferred. I already discussed this in sage-devel without much success.
the patching that goes beyond build issues. We should probably voice
that criticism more when SPKGs gut upgraded within Sage.
> - the commit policy is weird. Commits often aren't self contained changes that
> pass the doctests but commits like "update package" "fix some doctests in file
> x" "fix some more doctests in file y" instead. That makes cherry picking and
> bisecting unnecessarily hard.
release manager. I am not sure if `git bisect` can be made to do that.
> - the treatment of random failures. There are several open trac tickets with the
> "random_fail" keyword [1]. In my opinion they should all be release blockers
> and at least be marked as known failures before a release. Random failures in
> package tests are a huge pain for continuous integration.
> - the tight entanglement between "sage the library" and "sage the distribution".
> I don't think implementing an ad-hoc package manager with make is a good idea.
developing Sage are annoyed by all the issues that come with this
approach. Unfortunately Timo Kluck's attempt to use emerge instead of
make a few years ago never made it…
I tried to replace SPKGs with conda at some point but never made it past
a hacky prototype. I know that Erik Bray has also thought about this
topic and is doing some work that should make it easier to complement
SPKGs with alternative package sources.
> > [...]
> > Koper, Slovenia
> > 24 -- 28 September 2018
> on the 24th anyways. I'm not sure if I can take the additional time off and if
> I would even add any value there.
of the classes but otherwise I think that I'd have plenty of time to
work on packaging.
> My current remaining patches (ignoring package update patches) are
> - https://trac.sagemath.org/ticket/25358
> - https://trac.sagemath.org/ticket/25316
patching Python in Sage-the-distribution. I don't know if there was
ever a vote on this on sage-devel but it appears to me that if there's a
security issue in Python and the rest of the world is shipping it, then
we should not be the ones to fix it in Sage-the-distribution.
In any case, Sage should absolutely not check for this in its doctests.
(Generally, we should imho not doctest the world as
Sage-the-distribution is providing it but as our users are going to find
it on their machines…) If you can bring your ticket into a state that
the patchbots are happy with I am very happy to try my best to wave it
through.
The only good thing about this is that it is going away with the Python
3 upgrade at least…
Thanks for all your contributions btw and yeah, some day we'll be able
to ship an unpatched Sage :)
julian
Erik Bray
>
> Hi Timo,
>
> thanks for sharing your thoughts here.
>
> * Timo Kaufmann <eisf...@gmail.com> [2018-09-06 05:33:23 -0700]:
> > to patch dependencies without considering the consequences. Every patch to a
> > dependency is a burdon on all packagers and workarounds within sage should be
> > preferred. I already discussed this in sage-devel without much success.
> I think that there's some movement lately that SageMath should reduce
> the patching that goes beyond build issues. We should probably voice
> that criticism more when SPKGs gut upgraded within Sage.
and have tried to keep an eye on things like that. Fortunately, I
think there have been fewer if any cases of this lately (though I have
not reviewed all the existing patches). I'm also okay with patches
that fix an upstream bug, as upstream packagers might want to consider
including such patches in their builds as well. Though such patches
should also be submitted upstream, and new features in Sage should not
depend on patches--even bug patches--merely to function.
Best,
E
Erik Bray
>
> thanks for sharing your thoughts here.
>
> * Timo Kaufmann <eisf...@gmail.com> [2018-09-06 05:33:23 -0700]:
> > I don't think implementing an ad-hoc package manager with make is a good idea.
> It probably was reasonable back in the day. I know that many people
> developing Sage are annoyed by all the issues that come with this
> approach. Unfortunately Timo Kluck's attempt to use emerge instead of
> make a few years ago never made it…
>
> I tried to replace SPKGs with conda at some point but never made it past
> a hacky prototype. I know that Erik Bray has also thought about this
> topic and is doing some work that should make it easier to complement
> SPKGs with alternative package sources.
review (or at least constructive commentary) on this ticket would be a
very helpful start: https://trac.sagemath.org/ticket/24919
(apparently I need to rebase it yet again; I will do so tomorrow).
This ticket was opened 6 months ago and has gotten some testing by
others already, but no solid +1 or -1. It's part of a plan I
announced back in May 2017 (!) to overhaul sage-the-distribution to
better handle using system dependencies where available, rather than
building so many of its own dependencies needlessly. I did the
majority of that work over a couple months in one big chunk in the
Summer of 2017, but it has taken since then more than a year to split
that work into piecemeal updates and get them all reviewed and
integrated.
The existing work (like the ticket referenced in the above ticket)
won't *immediately* help downstream packaging, but it should
long-term. If/when #24919 or something like it is integrated then I
should be able to implement configure-time checks for the majority of
Sage's major dependencies pretty quickly. If it would help, I also
have plans to make it easy for downstream packaging systems to
override these checks (e.g. by using dpkg to check if a needed
dependency is already installed). But this hasn't been implemented
yet.
Longer-term, what I have wanted to do practically since I started on
the project is to completely separate sagelib and
sage-the-distribution into separate projects with separate
repositories. Step 1 is to actually take the sage-the-distribution
pseudo-packaging-system and disentangle it from Sage itself. Then
sagelib would become just-another-SPKG, and sage-the-distribution
would be the Sage packaging system as an independent entity, plus all
the metadata/scripts for the specific set of packages that make up
sage-the-distribution (including all the runtime and build time
dependencies for sagelib, plus all the 'optional' packages, etc.).
This would constitute a big change in how Sage is developed, but I
think largely for the better. It's rare, when switching between
branches, that changes in sagelib are significantly affected by
different versions of packages in the underlying distribution.
Sometimes it does matter, so it's important to have an easy way when
switching branches, to ensure that you also build against the right
dependency set. But I think this can be managed in a way that doesn't
force as frequent rebuilds. My work in #24919 moves more toward not
checking for specific SPKG versions for various dependencies, but just
ensuring that some dependency is sufficiently satisfied, and most of
the time that's good enough.
> > > [...]
> > > Software tools for mathematics
> > > Koper, Slovenia
> > > 24 -- 28 September 2018
>
> > I'm actually a bit tempted since I will already be half way in that direction
> > on the 24th anyways. I'm not sure if I can take the additional time off and if
> > I would even add any value there.
> I will be there as well. It would be great to meet up. I am giving one
> of the classes but otherwise I think that I'd have plenty of time to
> work on packaging.
users, rather than development. Maybe I could come for a couple days
too, if it would be useful.
> > My current remaining patches (ignoring package update patches) are
> > - https://trac.sagemath.org/ticket/25358
> > - https://trac.sagemath.org/ticket/25316
> These are about our infamous patch to Python 2.7. I am opposed to
> patching Python in Sage-the-distribution. I don't know if there was
> ever a vote on this on sage-devel but it appears to me that if there's a
> security issue in Python and the rest of the world is shipping it, then
> we should not be the ones to fix it in Sage-the-distribution.
> In any case, Sage should absolutely not check for this in its doctests.
> (Generally, we should imho not doctest the world as
> Sage-the-distribution is providing it but as our users are going to find
> it on their machines…) If you can bring your ticket into a state that
> the patchbots are happy with I am very happy to try my best to wave it
> through.
Also, most agree that the fix there should be to distutils (which
could even be monkey-patched!), not to sys.path handling, and Sage's
test suite definitely should not be covering this unless we're the
ones doing the monkey-patching.
> The only good thing about this is that it is going away with the Python
> 3 upgrade at least…
>
> Thanks for all your contributions btw and yeah, some day we'll be able
> to ship an unpatched Sage :)
necessary bugfix patch accepted for PARI. Last I tried they were
needlessly resistant...
Jeroen Demeyer
> - the number of package patches. I think sage-the-distribution makes it too easy
> to patch dependencies without considering the consequences. Every patch to a
> dependency is a burdon on all packagers and workarounds within sage should be
> preferred. I already discussed this in sage-devel without much success.
to get patches accepted upstream in CPython, even for obvious bug fixes.
I don't understand why Sage should take the blame for adding a patch but
upstream does *not* get any blame for ignoring very reasonable patches.
Timo Kaufmann
> Hi Timo,
>
> thanks for sharing your thoughts here.
>
> > > Please discuss anything that would make your life easier
> > > as a packager here on sage-packaging, or on sage-devel.
> >
> > My biggest gripes (already mentioned in other places too) are
> >
> > - the brittleness of the doctests. Often times they check string output of python
> > libraries that is not intended to be stable. As a result they break at pretty
> > much every minor package version change.
> I agree that this is very annoying. I have been a Sage contributor for a
> several years and only recently started to get involved in packaging it.
> As a developer I was not really aware of this problem and did not see
> how that would matter. So I guess that most developers are like that.
Developers usually use the sage-the-distribution and thus have the luxury of
not having to care about different versions. If sage were like other packages
(not having full control over its environment), more people would be aware of
the problems.
Especially because the reviwe process is pretty "anarchistic" (which does have
its advantages), it currently mostly depends on the mood and the priorities of
the reviewer.
> It's probably something that reviewers should be a bit more picky about.
> If you can make this precise with a few examples, we could include
> something in Sage's developer guide.
be eclib where much of the output of the mwrank binary is tested. Because of
that a lot of minor changes were needed in the recen upgrade[1]. Instead we
shouldn't test the output directly but instead parse it and only test the
correct parsing. If we only test the numerical results we can then also make
use of tolerances instead of `...`. We could even add a default relative
tolerance to the doctests. I don't think it is very useful to test any number
exactly. If there is a real error, it will usually be significantly different
from the correct value. Its not really sage's place to test downstream packages
results up to the 50's place after the decimal point.
> > - the number of package patches. I think sage-the-distribution makes it too easy
> > to patch dependencies without considering the consequences. Every patch to a
> > dependency is a burdon on all packagers and workarounds within sage should be
> > preferred. I already discussed this in sage-devel without much success.
> I think that there's some movement lately that SageMath should reduce
> the patching that goes beyond build issues. We should probably voice
> that criticism more when SPKGs gut upgraded within Sage.
didn't end in consensus. I think it is true though that the amount of patches
is reducing, although I haven't been around long enough to really judge that.
> > - the commit policy is weird. Commits often aren't self contained changes that
> > pass the doctests but commits like "update package" "fix some doctests in file
> > x" "fix some more doctests in file y" instead. That makes cherry picking and
> > bisecting unnecessarily hard.
> I usually found it good enough to only move between merges made by the
> release manager. I am not sure if `git bisect` can be made to do that.
have to `git bisect skip` everything that you don't want to test. Making an
effort to make nice self-contained patches would be better though.
I think part of the issue is that most of sages contributors (I'm assuming) are
mathematicians and not software engineers. That is necessarily so because
software engineers often lack the necessary domain language (I know I often do).
But it does mean that many contributors are not aware of some best practices.
> > - the treatment of random failures. There are several open trac tickets with the
> > "random_fail" keyword [1]. In my opinion they should all be release blockers
> > and at least be marked as known failures before a release. Random failures in
> > package tests are a huge pain for continuous integration.
> This seems like a reasonable policy. Have you tried to propose this on sage-devel?
doesn't agree with me about that. The discussion is continued in #25968[4]. I
didn't have the energy to start a bigger discussion about that.
> > - the tight entanglement between "sage the library" and "sage the distribution".
> > I don't think implementing an ad-hoc package manager with make is a good idea.
> It probably was reasonable back in the day. I know that many people
> developing Sage are annoyed by all the issues that come with this
> approach. Unfortunately Timo Kluck's attempt to use emerge instead of
> make a few years ago never made it…
>
> I tried to replace SPKGs with conda at some point but never made it past
> a hacky prototype. I know that Erik Bray has also thought about this
> topic and is doing some work that should make it easier to complement
> SPKGs with alternative package sources.
managing is pure, making sure that everything that needs to be rebuilt always
is rebuilt. All package receipts are stored in a single git repo, which can
be pinned at a specific commit. Dependencies can be declared in a simple file
and then `nix-shell` can be used to enter an environment where all the
dependencies are available. It is distribution agnostic. It supports transparent
substitution, meaning that packages can be fetched by the build server or build
locally if not available. Cross-compilation and distributed builds are also
supported. Builds happen in a sandbox. Binary reproducibility is also an option.
Maybe I'll suggest that some day, but I'm not very hopeful that it would gain
traction. If someone wants to discuss that we should probably fork into another
thread otherwise this would get too off-topic.
> > My current remaining patches (ignoring package update patches) are
> > - https://trac.sagemath.org/ticket/25358
> > - https://trac.sagemath.org/ticket/25316
> These are about our infamous patch to Python 2.7. I am opposed to
> patching Python in Sage-the-distribution. I don't know if there was
> ever a vote on this on sage-devel but it appears to me that if there's a
> security issue in Python and the rest of the world is shipping it, then
> we should not be the ones to fix it in Sage-the-distribution.
> In any case, Sage should absolutely not check for this in its doctests.
> (Generally, we should imho not doctest the world as
> Sage-the-distribution is providing it but as our users are going to find
> it on their machines…)
but Jeroen didn't seem to agree.
> If you can bring your ticket into a state that
> the patchbots are happy with I am very happy to try my best to wave it
> through.
for #255358. As far as I can see the reported failures are some pyflakes
issue and `--optional - magma` doctest failures, both should be unrelated.
Could you leave a comment at #255358 explaining the issue?
#25316 was marked as solved by Jeroen but it is only solved in python3. He
didn't reply when I clarified that. Should I re-set that to needs_review?
> The only good thing about this is that it is going away with the Python
> 3 upgrade at least…
recent progress in getting sage python3-ready.
> Thanks for all your contributions btw and yeah, some day we'll be able
> to ship an unpatched Sage :)
[1] https://trac.sagemath.org/ticket/25862#ticket
[2] https://groups.google.com/forum/#!topic/sage-devel/-lBph6WWg5E
[3] https://trac.sagemath.org/ticket/25790
[4] https://trac.sagemath.org/ticket/25968
Timo Kaufmann
> [...]
> > I think that there's some movement lately that SageMath should reduce
> > the patching that goes beyond build issues. We should probably voice
> > that criticism more when SPKGs gut upgraded within Sage.
>
> and have tried to keep an eye on things like that. Fortunately, I
> think there have been fewer if any cases of this lately (though I have
> not reviewed all the existing patches). I'm also okay with patches
> that fix an upstream bug, as upstream packagers might want to consider
> including such patches in their builds as well. Though such patches
> should also be submitted upstream, and new features in Sage should not
> depend on patches--even bug patches--merely to function.
downstream's bugs. Even though they may not be as pretty, monkey patches and
workarounds should be preferred. If patches are absolutely necessary, they
should be documented well. That includes a post to this mailing list, a practice
that I've been trying to start recently.
> On Thu, Sep 6, 2018 at 4:47 PM Julian Rüth wrote:
> >
> > Hi Timo,
> >
> > thanks for sharing your thoughts here.
> >
> > > I don't think implementing an ad-hoc package manager with make is a good idea.
> > It probably was reasonable back in the day. I know that many people
> Big +1. If anyone wants to help improve this, giving a positive
> review (or at least constructive commentary) on this ticket would be a
> very helpful start: https://trac.sagemath.org/ticket/24919
> (apparently I need to rebase it yet again; I will do so tomorrow).
sure it will actually be much of a benefit for packaging. It simply improves
sages own make-base packaging system, which distributions generally do not use.
Although thinking of it the two long-term advantages I can think off are
- it may be an incentive to be a little bit more flexible in the doctests once
enough dependencies are taken from the system
- if the configure checks are very well written (e.g. test features, not
versions) they could be used to verify that packages used by a distribution
are sufficient
> Longer-term, what I have wanted to do practically since I started on
> the project is to completely separate sagelib and
> sage-the-distribution into separate projects with separate
That would be great :) Thanks for your efforts.
> This would constitute a big change in how Sage is developed, but I
> think largely for the better. It's rare, when switching between
> branches, that changes in sagelib are significantly affected by
> different versions of packages in the underlying distribution.
> Sometimes it does matter, so it's important to have an easy way when
> switching branches, to ensure that you also build against the right
> dependency set. But I think this can be managed in a way that doesn't
> force as frequent rebuilds. My work in #24919 moves more toward not
that.
> > > My current remaining patches (ignoring package update patches) are
> > > - https://trac.sagemath.org/ticket/25358
> > > - https://trac.sagemath.org/ticket/25316
> +1
>
> Also, most agree that the fix there should be to distutils (which
> could even be monkey-patched!), not to sys.path handling, and Sage's
> test suite definitely should not be covering this unless we're the
> ones doing the monkey-patching.
> Yep. I think right now one of my biggest challenges is getting a
> necessary bugfix patch accepted for PARI. Last I tried they were
> needlessly resistant...
Timo Kaufmann
Its really not about blame. Fixes should be done as far upstream as possible.
However if that is not possible, working around it is better than simply fixing
it ourselves.
Jeroen Demeyer
> I don't think it is very useful to test any number
> exactly.
floating-point computations. However, Sage and some of its packages *do*
guarantee a certain precision in some cases. It would be wrong to *not*
test that.
> Obviously being biased, but I think nix would be perfect for the job.
>
> traction.
suggested, emerge has been suggested, nix has been suggested. However,
actually replacing the whole Sage build system is not a simple task. But
if you're serious about actually doing it, I'm pretty sure that it would
gain traction.
Jeroen Demeyer
> Its really not about blame.
But you are certainly complaining here on this mailing list that Sage
patches upstream. Instead, I think that you should really be complaining
to upstream that they are not accepting those patches.
Erik Bray
a build issue, sure. But if it's doing something that Sage relies to
function *at all* in some area (even if we believe we're fixing a bug)
it is problematic. If I were reviewing an issue like this I'd have to
be convinced that every possible avenue wasn't taken to work around
the bug before relying on a patch. Again, it depends on the patch
though. If upstream maintainers are being unreasonable about a patch
to an obvious bug, it's more likely you'll convince downstream
packagers to include that patch in their package, and maybe that's
okay.
Erik Bray
to vent a bit :)
mwrank for which Sage's parsing fails, then I believe it should be
Sage's responsibility to support parsing the outputs of different
versions of the program (not necessarily *every* version, but at least
some more flexible range). If we want to pin to an *exact* version we
should vendor the library directly into the Sage sources (which can
still be problematic, depending on the policies of a given
distribution; e.g. I think Debian will try to un-vendor things like
this but you can clarify; it also depends on how much it's modified,
etc.).
> > > - the number of package patches. I think sage-the-distribution makes it too easy
> > > to patch dependencies without considering the consequences. Every patch to a
> > > dependency is a burdon on all packagers and workarounds within sage should be
> > > preferred. I already discussed this in sage-devel without much success.
> > I think that there's some movement lately that SageMath should reduce
> > the patching that goes beyond build issues. We should probably voice
> > that criticism more when SPKGs gut upgraded within Sage.
>
> The only movement I'm aware of is the sage-devel post I stared myself[2], which
> didn't end in consensus. I think it is true though that the amount of patches
> is reducing, although I haven't been around long enough to really judge that.
>
> > > - the commit policy is weird. Commits often aren't self contained changes that
> > > pass the doctests but commits like "update package" "fix some doctests in file
> > > x" "fix some more doctests in file y" instead. That makes cherry picking and
> > > bisecting unnecessarily hard.
> > I usually found it good enough to only move between merges made by the
> > release manager. I am not sure if `git bisect` can be made to do that.
>
> Yes that is a workaround. I don't think git-bisect can be made to do that, you'd
> have to `git bisect skip` everything that you don't want to test. Making an
> effort to make nice self-contained patches would be better though.
>
> I think part of the issue is that most of sages contributors (I'm assuming) are
> mathematicians and not software engineers. That is necessarily so because
> software engineers often lack the necessary domain language (I know I often do).
> But it does mean that many contributors are not aware of some best practices.
describing sounds like some of the Python 3 updates which have
admittedly been a bit haphazard. I also don't necessarily believe
every commit must be 100% self-contained. For example, if I'm
refactoring some area of code (as part of a larger set of logically
connected changes) I will do that refactoring as a single commit that
can be explained in the commit message. In other words, I mostly make
commits based on how clear it is that my documentation of the commit
applies to the actual code diff. So this doesn't necessarily equate
to "1 feature = 1 commit" or something like that. I'm not sure if
that's really what you're referring to though.
I do personally try to squash commits I make during development if
they are not very interesting or self-contained. I've observed that a
lot of Sage contributors are not well-versed in using tools like "git
rebase", and often also occasionally up on merges. While Sage's "git
trac" command is very convenient for many cases, for beginners and
advanced developers alike, we could probably do more as a community to
train Sage developers on effective git usage.
> > > - the treatment of random failures. There are several open trac tickets with the
> > > "random_fail" keyword [1]. In my opinion they should all be release blockers
> > > and at least be marked as known failures before a release. Random failures in
> > > package tests are a huge pain for continuous integration.
> > This seems like a reasonable policy. Have you tried to propose this on sage-devel?
>
> No I haven't. I noticed this issue after #25790[3] where it seemed that Volker
> doesn't agree with me about that. The discussion is continued in #25968[4]. I
> didn't have the energy to start a bigger discussion about that.
blocker here is just available developer resources to investigate
random failures, especially if they are difficult to reproduce. So
they are probably deemed "more acceptable".
Personally, I have long wished for Sage's test runner to have a way to
mark tests as "known failures" so we can at least have the tests
"pass" even if some test that is known to fail in some cases fails.
This is not a silver bullet, because in my experience failing tests
marked as "known failures" do end up going unfixed for a longer time.
At the very least, every test run should have a summary of "known
fail" tests that did fail, so that we don't just forget about them
completely (this is opposed to, for example, some test runners I've
seen such as nose where a "known fail" just shows up as a "K" in the
test output and is easily ignored/forgotten).
> > > - the tight entanglement between "sage the library" and "sage the distribution".
> > > I don't think implementing an ad-hoc package manager with make is a good idea.
> > It probably was reasonable back in the day. I know that many people
> > developing Sage are annoyed by all the issues that come with this
> > approach. Unfortunately Timo Kluck's attempt to use emerge instead of
> > make a few years ago never made it…
> >
> > I tried to replace SPKGs with conda at some point but never made it past
> > a hacky prototype. I know that Erik Bray has also thought about this
> > topic and is doing some work that should make it easier to complement
> > SPKGs with alternative package sources.
>
> Obviously being biased, but I think nix would be perfect for the job. Package
> managing is pure, making sure that everything that needs to be rebuilt always
> is rebuilt. All package receipts are stored in a single git repo, which can
> be pinned at a specific commit. Dependencies can be declared in a simple file
> and then `nix-shell` can be used to enter an environment where all the
> dependencies are available. It is distribution agnostic. It supports transparent
> substitution, meaning that packages can be fetched by the build server or build
> locally if not available. Cross-compilation and distributed builds are also
> supported. Builds happen in a sandbox. Binary reproducibility is also an option.
>
> Maybe I'll suggest that some day, but I'm not very hopeful that it would gain
> traction. If someone wants to discuss that we should probably fork into another
> thread otherwise this would get too off-topic.
package managers for Sage, and looked at a number of possible ways to
replace Sage's current hand-rolled "packaging system" such that it is.
I believe nix was looked at, as well as conda, Guix, and hashdist. I
personally rather liked Guix, but I think it fell somewhat on deaf
ears. I have a feeling that it's Guile/Scheme-based configuration
language turned a lot of people off. Indeed it threw me for a loop as
well at first, but I pressed on and decided I liked it. I also have
found that I like hashdist, but it doesn't seem to be getting much
traction.
All that said, after that workshop, there was no real conclusion or
strong consensus; I think the strongest consensus was maybe for conda,
but that will be a problem for Sage on Cygwin, as there is not
officially a conda for Cygwin, or conda binaries for Cygwin. I have
thought of starting my own conda package channel for Cygwin, which I
think would actually be very useful since the packaging system for
Cygwin, such that there is one at all, is not very useful or friendly.
But that is a big project that I think is a little outside the scope I
have time for. But who knows; maybe I've overestimated the effort. I
haven't really tried.
So my ultimate conclusion and working premise since then is to press
on with improving the Sage-the-distribution system with an eye toward
keeping the existing Sage development workflow somewhat working, but
to also make it easier to maybe create binary packages, and to also
(as I explained in my last post) move toward distentangling the
sagelib sources from the sage-the-distribution sources.
> > > My current remaining patches (ignoring package update patches) are
> > > - https://trac.sagemath.org/ticket/25358
> > > - https://trac.sagemath.org/ticket/25316
> > These are about our infamous patch to Python 2.7. I am opposed to
> > patching Python in Sage-the-distribution. I don't know if there was
> > ever a vote on this on sage-devel but it appears to me that if there's a
> > security issue in Python and the rest of the world is shipping it, then
> > we should not be the ones to fix it in Sage-the-distribution.
> > In any case, Sage should absolutely not check for this in its doctests.
> > (Generally, we should imho not doctest the world as
> > Sage-the-distribution is providing it but as our users are going to find
> > it on their machines…)
>
> I totally agree and made pretty much those exact same points in the tickets,
> but Jeroen didn't seem to agree.
>
> > If you can bring your ticket into a state that
> > the patchbots are happy with I am very happy to try my best to wave it
> > through.
>
> That would be great :) I'm having trouble interpreting the patchbot output
> for #255358. As far as I can see the reported failures are some pyflakes
> issue and `--optional - magma` doctest failures, both should be unrelated.
> Could you leave a comment at #255358 explaining the issue?
patchbots, or only enable them after they've first made an effort to
fix ones that fail regularly :(
Erik Bray
> While I think you're doing great work and #24919 is a big improvement, I'm not
> sure it will actually be much of a benefit for packaging. It simply improves
> sages own make-base packaging system, which distributions generally do not use.
packaging indirectly by encouraging more use of developers' system
packages instead of using Sage's packages, so that more of Sage will
be tested against a wider range of system packages more regularly.
In the past of course supporting the vast array of package
combinations has made Sage difficult for many users to build, which is
why it's useful to have Sage's collection of precisely pinned
dependencies to fall back on. So I have kept that as an option--for
any of Sage's dependencies, sage-the-distribution makes it relatively
easy to slip in a supported version. There is also of course the case
of high-performance libraries like mpir which many users may want to
have local builds of.
> Although thinking of it the two long-term advantages I can think off are
>
> - it may be an incentive to be a little bit more flexible in the doctests once
> enough dependencies are taken from the system
> - if the configure checks are very well written (e.g. test features, not
> versions) they could be used to verify that packages used by a distribution
> are sufficient
checks for, and probably most of them will be quite simple at first,
and then adjusted / tightened with appropriate experience and feedback
from users and developers on different systems. If some dependency
breaks, they will still be able to fall back on the version shipped
with sage-the-distribution.
> > Longer-term, what I have wanted to do practically since I started on
> > the project is to completely separate sagelib and
> > sage-the-distribution into separate projects with separate
> > [...]
>
> That would be great :) Thanks for your efforts.
>
> > This would constitute a big change in how Sage is developed, but I
> > think largely for the better. It's rare, when switching between
> > branches, that changes in sagelib are significantly affected by
> > different versions of packages in the underlying distribution.
> > Sometimes it does matter, so it's important to have an easy way when
> > switching branches, to ensure that you also build against the right
> > dependency set. But I think this can be managed in a way that doesn't
> > force as frequent rebuilds. My work in #24919 moves more toward not
>
> Again regarding rebuilds, replacing the make-based system with nix would solve
> that.
but I don't remember much about it. I still think it might be a
bigger effort than it's worth to completely replace Sage's "packaging
system" with something else, but I haven't *tried* either. Certainly
the conda-forge folks have made impressive progress so it's not
impossible.
> > Yep. I think right now one of my biggest challenges is getting a
> > necessary bugfix patch accepted for PARI. Last I tried they were
> > needlessly resistant...
>
> We're not currently patching PARI -- should we?
Sage just did not work at all, but with more recent versions of PARI I
believe that's no longer necessary.
There is a rather critical patch for PARI on Cygwin, but obviously
that's of no relevance on Debian.
Jeroen Demeyer
> Personally, I have long wished for Sage's test runner to have a way to
> mark tests as "known failures"
Tests with that marker are just ignored completely.
Erik Bray
>
> On 2018-09-07 12:04, Erik Bray wrote:
> > Personally, I have long wished for Sage's test runner to have a way to
> > mark tests as "known failures"
>
> There is # known bug
this somehow platform-dependent, but that's another issue...
> Tests with that marker are just ignored completely.
should be a summary of results from "known bug" tests at the end of
the tests...
Timo Kaufmann
Well there already is the sage package for nix. So at least for linux, it
wouldn't be much work. Some of the patches sage applies for other platforms
would need to be added, but I can't test that myself.
I'm advocating to change the policy for sage (I'd like to think I'm not
complaining, but thats mostly a matter of definition). I can't do the same
for every upstream package. What am I supposed to do, nag people on upstream
PRs until they grow agitated? I'm saying that yes upstream should fix stuff
but *also* sage should not make it harder than necessary for *its* downstream
and users.
Timo Kaufmann
> [...]
> mwrank for which Sage's parsing fails, then I believe it should be
> Sage's responsibility to support parsing the outputs of different
> versions of the program (not necessarily *every* version, but at least
> some more flexible range).
Exactly :)
> If we want to pin to an *exact* version we
> should vendor the library directly into the Sage sources (which can
> still be problematic, depending on the policies of a given
> distribution; e.g. I think Debian will try to un-vendor things like
> this but you can clarify; it also depends on how much it's modified,
> etc.).
Just in case you meant me with "you", I am not involved with debian. I think if
there truly are breaking changes or important new features and accomodating both
versions is unreasonable, it is okay to not support an older version. It just
shouldn't be nearly as common as it is with sage. I'm not sure if vendoring
would be an improvement.
> > [...]
> > Yes that is a workaround. I don't think git-bisect can be made to do that, you'd
> > have to `git bisect skip` everything that you don't want to test. Making an
> > effort to make nice self-contained patches would be better though.
> >
> > I think part of the issue is that most of sages contributors (I'm assuming) are
> > mathematicians and not software engineers. That is necessarily so because
> > software engineers often lack the necessary domain language (I know I often do).
> > But it does mean that many contributors are not aware of some best practices.
>
> I'm not exactly sure what you mean here. A lot of what you're
> describing sounds like some of the Python 3 updates which have
> admittedly been a bit haphazard.
I'm mostly talking about package updates, since that is what I'm most involved
with. A random example I dug up right no is #25277. I *don't* want to point
fingers right now and since we don't currently have a policy the contributor
didn't do anything wrong. It is just not the way I consider ideal. Commits include:
_test_divides in Rings()
divides for integer mod ring
smarter iteration over pairs of elements
# there are lots of doctest failures after this
doctests fix
skip _test_divides in the symbolic
remove useless tests
> I also don't necessarily believe
> every commit must be 100% self-contained. For example, if I'm
> refactoring some area of code (as part of a larger set of logically
> connected changes) I will do that refactoring as a single commit that
> can be explained in the commit message.
self-contained does not mean big. It just means that if sage built and the
doctests succeeded before the commit, they should also do so after. There are
a lot of git subcommands that make the assumption that commit are like that:
git bisect, git cherry-pick and git revert come to mind. So a refactoring can
very well (and should) be a seperate commit. In the above example the necessity
of a "doctests fix" commit show that the commits before that were clearly not
self contained.
Also there is the somewhat less important matter of the commit message. I prefer
the style used in the git book[1] (short, imperative, first letter capitalized).
But just some standardization would be nice.
> they are not very interesting or self-contained. I've observed that a
> lot of Sage contributors are not well-versed in using tools like "git
> rebase", and often also occasionally up on merges. While Sage's "git
> trac" command is very convenient for many cases, for beginners and
> advanced developers alike, we could probably do more as a community to
> train Sage developers on effective git usage.
Thats what I meant by sages unique position of having lots of contributions by
non-CS people. For many contributors sage is probably the first open soruce
project they ever contribute to.
> > > > - the treatment of random failures. There are several open trac tickets with the
> > [...]
> I think I generally agree with you about this. Perhaps the bigger
> blocker here is just available developer resources to investigate
> random failures, especially if they are difficult to reproduce. So
> they are probably deemed "more acceptable".
Yes I understand that we can't fix them all, but we should at least mark them as
known before release. As Jeroen noted that is already possible and is what I
tried to do in #25968 for one such failure for the last release. I agree that
still reporting those known failures would be better, but at least there are the
open trac tickets as reminders.
> > Obviously being biased, but I think nix would be perfect for the job. Package
> > [...]
>
> Two years ago in Cernay (or was it three??) we had a workshop about
> package managers for Sage, and looked at a number of possible ways to
> replace Sage's current hand-rolled "packaging system" such that it is.
> I believe nix was looked at, as well as conda, Guix, and hashdist. I
> personally rather liked Guix, but I think it fell somewhat on deaf
> ears. I have a feeling that it's Guile/Scheme-based configuration
> language turned a lot of people off. Indeed it threw me for a loop as
> well at first, but I pressed on and decided I liked it. I also have
> found that I like hashdist, but it doesn't seem to be getting much
> traction.
Guix is pretty much the same as Nix, just a different CLI and different language.
Its a good tool and more polished in some areas, I'm just sad that they felt the
need to split the community. Nix has a bigger community and does not flat-out
reject all unfree packages (while still allowing users to disable them), which
is why I chose Nix over Guix.
> All that said, after that workshop, there was no real conclusion or
> strong consensus; I think the strongest consensus was maybe for conda,
> but that will be a problem for Sage on Cygwin, as there is not
> officially a conda for Cygwin, or conda binaries for Cygwin.
I'm not sure what the state of nix for windows is. I think I read somewhere
that it runs on bash-on-windows. If you're interested in looking into that
and maybe even make sage build there, I'd be happy to help. I would assume
that the effort wouldn't be too great (as long as n ix already runs on windows
and the necessary fixes are already known since the spkgs work there), but
maybe I'm naive.
> > > If you can bring your ticket into a state that
> > > the patchbots are happy with I am very happy to try my best to wave it
> > > through.
> >
> > That would be great :) I'm having trouble interpreting the patchbot output
> > for #255358. As far as I can see the reported failures are some pyflakes
> > issue and `--optional - magma` doctest failures, both should be unrelated.
> > Could you leave a comment at #255358 explaining the issue?
>
> I wish people would either stop enabling optional tests on their
> patchbots, or only enable them after they've first made an effort to
> fix ones that fail regularly :(
So there is nothing to be done then?
Timo Kaufmann
> [...]
> of high-performance libraries like mpir which many users may want to
> have local builds of.
Another nice thing about the transparent binary substiutions in nix is that
users can modify a single package (like configure mpir to build with optimizations)
and build that locally while everything else is still fetched from a build server.
> > Again regarding rebuilds, replacing the make-based system with nix would solve
> > that.
>
> I'll have to take a closer look at nix. I know I looked at it before
> but I don't remember much about it. I still think it might be a
> bigger effort than it's worth to completely replace Sage's "packaging
> system" with something else, but I haven't *tried* either. Certainly
> the conda-forge folks have made impressive progress so it's not
> impossible.
As I said before, sage builds on nix so at least for linux its really not that
big an effort. Fixes for other platforms would just need to be ported. I don't
know which platforms are oficially supported by sage.
Maybe I'll write a post to sage-devel where I explain the advantages of nix in
more detail.
Erik Bray
rebase, and I usually try to ask people to do that when I review
tickets, but I don't review most tickets.
I'm guilty of this myself, however. In the case of Sage it's also
more common to find doctest failures later that were not caught during
development. This is because the size of Sage's test suite is too
large and it's almost impossible to do a normal build/test/commit
cycle. At best I might test one or two modules that are closely
related to the changes I'm making. But sometimes there can be
failures in tests in distantly unrelated modules (and some of these
tests are really bad IMO, especially some of the tests that use the
actual Sage sources as input to the test, so that arbitrary changes to
the sources can randomly break those tests).
Two related things I'm working on to improve this situation:
1) I have a prototype for line coverage built into Sage's doctest
runner, so I can get *actual* test coverage results. Unfortunately,
between enabling line profiling in Cython, and the overhead of the
line profiler itself, it makes the tests painfully slow. It's not
very practical, I've found, to collect coverage over the entire test
suite. Granted, there might still be areas where that can be
improved--I haven't investigated carefully. That said, the coverage
results are very useful on the scale of individual modules, so
2) One thing I'm trying to do is ensure that each new module I work on
is fully covered by the doctests in that module, rather than having
its test coverage spread around various other modules as is currently
the case with many parts of Sage. I don't have my coverage
integration merged in yet, but once it is it will be easier to ensure
that modules are better covered by their own tests.
Of course, Sage is inescapably a big system with many
interdependencies, so even 100% test coverage for some module does not
rule out the possibility of distant test failures. But it should
reduce the likelihood.
> Also there is the somewhat less important matter of the commit message. I prefer
> the style used in the git book[1] (short, imperative, first letter capitalized).
> But just some standardization would be nice.
community to agree to a commit message style (huge bikeshed magnet but
a good idea in principle) it would be nice.
> > they are not very interesting or self-contained. I've observed that a
> > lot of Sage contributors are not well-versed in using tools like "git
> > rebase", and often also occasionally up on merges. While Sage's "git
> > trac" command is very convenient for many cases, for beginners and
> > advanced developers alike, we could probably do more as a community to
> > train Sage developers on effective git usage.
>
> > blocker here is just available developer resources to investigate
> > random failures, especially if they are difficult to reproduce. So
> > they are probably deemed "more acceptable".
>
> Yes I understand that we can't fix them all, but we should at least mark them as
> known before release. As Jeroen noted that is already possible and is what I
> tried to do in #25968 for one such failure for the last release. I agree that
> still reporting those known failures would be better, but at least there are the
> open trac tickets as reminders.
not very interesting to me. It *could* become interesting, especially
if Microsoft would expose and document more of its internals. It's a
convenience for doing development on Windows, sure, but I actually
hardly use it. Having Docker has mostly obviated it for me since if I
want to bring up a Linux machine with any distro I can just pull down
the appropriate Docker image and run in that.
However, if more of its internals were exposed it could be very useful
for Cygwin, especially the fork and socket implementations.
The problem with using the WSL for Sage on Windows is that, while it
may be useful for development, it's not intended to be used for
software distribution; in other words, distributing pre-built Windows
software that relies on the WSL to work is not supported. This is why
Sage for Windows uses Cygwin instead--Cygwin is just a handful of DLLs
you can include in your software distribution.
> > I'll have to take a closer look at nix. I know I looked at it before
> > but I don't remember much about it. I still think it might be a
> > bigger effort than it's worth to completely replace Sage's "packaging
> > system" with something else, but I haven't *tried* either. Certainly
> > the conda-forge folks have made impressive progress so it's not
> > impossible.
>
> big an effort. Fixes for other platforms would just need to be ported. I don't
> know which platforms are oficially supported by sage.
>
> Maybe I'll write a post to sage-devel where I explain the advantages of nix in
> more detail.
specifics of Nix. I've heard of it of course, but I've never tried to
use it or read the docs. Anything that should work as the "primary
supported" packaging system for Sage would have to work on
Linux/FreeBSD, macOS, and preferably Cygwin at a minimum. For macOS I
have no idea. For Cygwin it could work *in principle* but there are
usually some hairy bits when it comes to things like
packaging/building on Cygwin. This person claims to have gotten Nix
sort of working on Cygwin:
https://invalidmagic.wordpress.com/2011/01/19/running-the-nix-package-manager-on-windows/
But that was in 2011, so I doubt it works all the same today. Still,
it might be interesting to try sometime.
But this is why my approach, after not being quite satisfied with any
solution I saw at the Cernay workshop, has been to focus on improving
what we already have, and instead make it so that Sage does not rely
on building so many of its dependencies, so that it can instead use
more existing system packages regardless what the underlying package
manager is. This could include, where appropriate, hooks for package
managers as well. But I haven't made it to that stage yet.
> > > > If you can bring your ticket into a state that
> > > > the patchbots are happy with I am very happy to try my best to wave it
> > > > through.
> > >
> > > That would be great :) I'm having trouble interpreting the patchbot output
> > > for #255358. As far as I can see the reported failures are some pyflakes
> > > issue and `--optional - magma` doctest failures, both should be unrelated.
> > > Could you leave a comment at #255358 explaining the issue?
> >
> > I wish people would either stop enabling optional tests on their
> > patchbots, or only enable them after they've first made an effort to
> > fix ones that fail regularly :(
>
> So there is nothing to be done then?
Andreas Enge
> Am Freitag, 7. September 2018 12:04:45 UTC+2 schrieb Erik M. Bray:
> > Two years ago in Cernay (or was it three??) we had a workshop about
> > package managers for Sage, and looked at a number of possible ways to
> > replace Sage's current hand-rolled "packaging system" such that it is.
> > I believe nix was looked at, as well as conda, Guix, and hashdist. I
> > personally rather liked Guix, but I think it fell somewhat on deaf
> > ears. I have a feeling that it's Guile/Scheme-based configuration
> > language turned a lot of people off. Indeed it threw me for a loop as
> > well at first, but I pressed on and decided I liked it.
I gave the talk on Guix there and was interested in packaging Sage for Guix,
but unfortunately did not have the time since then.
> Guix is pretty much the same as Nix, just a different CLI and different language.
> Its a good tool and more polished in some areas, I'm just sad that they felt the
> need to split the community. Nix has a bigger community and does not flat-out
> reject all unfree packages (while still allowing users to disable them), which
> is why I chose Nix over Guix.
software. So it is also not running on either Mac or Windows, which as I
understood it was the main argument for not switching to Guix as a build
tool for Sage.
Another thing we are interested in is to build everything from source, so
we try to unbundle as much as possible (at the time, that was different
in Nix: when I packaged texlive for Guix and compiled it from source, the
Nix packages consisted of unwrapped Debian binaries, if I remember well;
this has changed in the meantime, at least for texlive).
We are also striving for reproducible builds, but this may also be true
for Nix.
Concerning Docker images, they can easily be created with Guix, see
https://www.gnu.org/software/guix/blog/2017/creating-bundles-with-guix-pack/
(Docker is mentioned in the "Bells and whistles" section). So if we had
Sage in Guix, it would be easy to create Docker images for it.
Andreas
Timo Kaufmann
> [...]
> > doctests succeeded before the commit, they should also do so after. There are
> > a lot of git subcommands that make the assumption that commit are like that:
> > git bisect, git cherry-pick and git revert come to mind. So a refactoring can
> > very well (and should) be a seperate commit. In the above example the necessity
> > of a "doctests fix" commit show that the commits before that were clearly not
> > self contained.
>
> I see, yes, those are cases where I would normally encourage people to
> rebase, and I usually try to ask people to do that when I review
> tickets, but I don't review most tickets.
Yes I haven't even required this on the few tickets I reviewed myself, since
there are no existing guidelines to point people to. There is little one can do
as a single contributor/reviewer, especially if the majority of contributor /
reviewers don't agree / are not aware of a problem. I don't have the energy to
work on changing that either.
> I'm guilty of this myself, however. In the case of Sage it's also
> more common to find doctest failures later that were not caught during
> development. This is because the size of Sage's test suite is too
> large and it's almost impossible to do a normal build/test/commit
> cycle.
Ah yes, that is also a problem. Most changes I did were small enough to be only
one commit, so build/compile/test worked for me. But I understand that that is
harder when making bigger changes in multiple commits. Maybe the `--short`
doctest will at least somewhat help here. It doesn't have to be perfect either.
Many times commits are just split into "change this file" and "change that
file". In that case, it would be obvious that they should be merged / seperated
diefferently.
> At best I might test one or two modules that are closely
> related to the changes I'm making. But sometimes there can be
> failures in tests in distantly unrelated modules (and some of these
> tests are really bad IMO, especially some of the tests that use the
> actual Sage sources as input to the test, so that arbitrary changes to
> the sources can randomly break those tests).
I personally think sage may have pushed the "test all the things" mentallity
a little too far. While tests are essential and good, I feel like some tests
are just there for the sake of having tests and aren't very useful.
> Two related things I'm working on to improve this situation:
>
> 1) I have a prototype for line coverage built into Sage's doctest
> runner, so I can get *actual* test coverage results.
> [...]
Nice effort, thanks again for your good work :) Although I think there is no
getting around testing the integration of modules outside of the modules.
> > Also there is the somewhat less important matter of the commit message. I prefer
> > the style used in the git book[1] (short, imperative, first letter capitalized).
> > But just some standardization would be nice.
>
> I'm very inconsistent about things like this. If we could get the
> community to agree to a commit message style (huge bikeshed magnet but
> a good idea in principle) it would be nice.
Yes, consistency is far more important than any particular style. Yet again,
I don't want to "fight that battle" and start the bike-shedding.
> > [...]
> > I'm not sure what the state of nix for windows is. I think I read somewhere
> > that it runs on bash-on-windows. If you're interested in looking into that
> > and maybe even make sage build there, I'd be happy to help. I would assume
> > that the effort wouldn't be too great (as long as n ix already runs on windows
> > and the necessary fixes are already known since the spkgs work there), but
> > maybe I'm naive.
>
> The Windows Subsystem for Linux (aka "bash-on-windows") is currently
> not very interesting to me. It *could* become interesting, especially
> if Microsoft would expose and document more of its internals. It's a
> convenience for doing development on Windows, sure, but I actually
> hardly use it. Having Docker has mostly obviated it for me since if I
> want to bring up a Linux machine with any distro I can just pull down
> the appropriate Docker image and run in that.
>
> However, if more of its internals were exposed it could be very useful
> for Cygwin, especially the fork and socket implementations.
>
> The problem with using the WSL for Sage on Windows is that, while it
> may be useful for development, it's not intended to be used for
> software distribution; in other words, distributing pre-built Windows
> software that relies on the WSL to work is not supported. This is why
> Sage for Windows uses Cygwin instead--Cygwin is just a handful of DLLs
> you can include in your software distribution.
What do you mean it is not supported? You can certainly fetch pre-build binaries
with nix on bash-on-windows.
For what its worth I checked, and there are multiple Cygwin-specific fixes
in nixpkgs. So apparently it does run there. CI tests are only run on
x86-64-linux, aarch64-linux and darwin however. And I'm note sure what the
situation about shipping binary packages is. I assume bash-on-linux can just
use linux binaries, but Cygwin probably cannot. So since I don't think we
currently officially ship Cygwin binaries, you'd have to build everything
from source. It is possible to create your own server to serve binaries
however.
> > [...]
> > Maybe I'll write a post to sage-devel where I explain the advantages of nix in
> > more detail.
>
> That would be interesting. Like I said, I've not very aware of the
> specifics of Nix. I've heard of it of course, but I've never tried to
> use it or read the docs. Anything that should work as the "primary
> supported" packaging system for Sage would have to work on
> Linux/FreeBSD, macOS, and preferably Cygwin at a minimum. For macOS I
> have no idea. For Cygwin it could work *in principle* but there are
> usually some hairy bits when it comes to things like
> packaging/building on Cygwin. This person claims to have gotten Nix
> sort of working on Cygwin:
> https://invalidmagic.wordpress.com/2011/01/19/running-the-nix-package-manager-on-windows/
> But that was in 2011, so I doubt it works all the same today. Still,
> it might be interesting to try sometime.
Well as I said, darwin support is already close. Actually the only blocker i'm
aware of is a test failure in a test sage-the-distribution just doesn't run, so
the same failure is probably also present there (#26130). I think the BSDs are
somewhat supported, although I know little about them and the differences from
linux. As long as nix runs on a platform, porting sage's dependencies should be
manageable since all the fixes are already present in the spkgs.
> But this is why my approach, after not being quite satisfied with any
> solution I saw at the Cernay workshop, has been to focus on improving
> what we already have, and instead make it so that Sage does not rely
> on building so many of its dependencies, so that it can instead use
> more existing system packages regardless what the underlying package
> manager is. This could include, where appropriate, hooks for package
> managers as well. But I haven't made it to that stage yet.
Thank you for that. I appreciate that you take the concerns seriously.
> > [...]
> > So there is nothing to be done then?
>
> Well, not nothing: https://gitlab.com/sagemath/dev/trac/pipelines
How do I interpret that?
Timo Kaufmann
> > Guix is pretty much the same as Nix, just a different CLI and different language.
> > Its a good tool and more polished in some areas, I'm just sad that they felt the
> > need to split the community. Nix has a bigger community and does not flat-out
> > reject all unfree packages (while still allowing users to disable them), which
> > is why I chose Nix over Guix.
>
> As a GNU project, it is an important point for Guix to only support free
> software. So it is also not running on either Mac or Windows, which as I
> understood it was the main argument for not switching to Guix as a build
> tool for Sage.
I'm still not convinced the hard-fork was necessary -- the two communities
could have accomplished more if we had sticked to a common package base.
All unfree packages are clearly marked as such and "Guix" could have been
Nix with the only difference being a hardcoded inability to parse those
packages (essentially hardcoding `allowUnfree = false`). That discussion
is off-topic of course and would probably be a better fit at
discourse.nixos.org :)
> Another thing we are interested in is to build everything from source, so
> we try to unbundle as much as possible (at the time, that was different
> in Nix: when I packaged texlive for Guix and compiled it from source, the
> Nix packages consisted of unwrapped Debian binaries, if I remember well;
> this has changed in the meantime, at least for texlive).
As is Nix, although there are some compromises when there is a lack of manpower.
Which is why it would have been nice to work together.
> We are also striving for reproducible builds, but this may also be true
> for Nix.
Yes that is also true for Nix, although I think Guix's effort may be further
along. Again, lack of resources to make it a main priority. That is a personal
"pet peeve" of mine that I'd love to see improved. I hope I'll have time to
improve it myself in the future.
> Concerning Docker images, they can easily be created with Guix, see
> https://www.gnu.org/software/guix/blog/2017/creating-bundles-with-guix-pack/
> (Docker is mentioned in the "Bells and whistles" section). So if we had
> Sage in Guix, it would be easy to create Docker images for it.
I believe that is possible in Nix too, although I am not certain (no experience
with docker). The only thing I could find with a quick search is this[1], but
that is unmaintained.
Erik Bray
>
> Am Montag, 10. September 2018 18:16:12 UTC+2 schrieb Erik M. Bray:
> > At best I might test one or two modules that are closely
> > related to the changes I'm making. But sometimes there can be
> > failures in tests in distantly unrelated modules (and some of these
> > tests are really bad IMO, especially some of the tests that use the
> > actual Sage sources as input to the test, so that arbitrary changes to
> > the sources can randomly break those tests).
>
> I personally think sage may have pushed the "test all the things" mentallity
> a little too far. While tests are essential and good, I feel like some tests
> are just there for the sake of having tests and aren't very useful.
testing the default behavior of Python's open(...) built-in on a
missing file. There was no larger context to it. That was literally
just it.
I think the Sage community would do well to read this essay:
http://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf Not
all of sage's doctests are unit tests, but many are. I don't agree
100% with that paper either, nor do I agree with all of the exact
prescriptions. But it's worth taking in and reflecting on.
> > The problem with using the WSL for Sage on Windows is that, while it
> > may be useful for development, it's not intended to be used for
> > software distribution; in other words, distributing pre-built Windows
> > software that relies on the WSL to work is not supported. This is why
> > Sage for Windows uses Cygwin instead--Cygwin is just a handful of DLLs
> > you can include in your software distribution.
>
> What do you mean it is not supported? You can certainly fetch pre-build binaries
> with nix on bash-on-windows.
software. Perhaps you're no familiar with the efforts we've made in
those areas but have a look at:
https://opendreamkit.org/2017/10/11/SageWindows/
Requiring that users have Windows 10, with developer mode enabled, and
learn bash-for-windows just to use Sage is fine for
intermediate/advanced developer-users, but not for casual users who
want to use Sage for some computations.
> For what its worth I checked, and there are multiple Cygwin-specific fixes
> in nixpkgs. So apparently it does run there. CI tests are only run on
> x86-64-linux, aarch64-linux and darwin however. And I'm note sure what the
Perhaps it worked at one point but it is not maintained.
> situation about shipping binary packages is. I assume bash-on-linux can just
> use linux binaries, but Cygwin probably cannot. So since I don't think we
> currently officially ship Cygwin binaries, you'd have to build everything
> from source. It is possible to create your own server to serve binaries
> however.
macOS support, looks like the best bet right now and Sage already
mostly works there. But I think it will be necessary to create a
conda channel specifically for distributing Cygwin binaries. I'm not
sure how straightfoward that would be, and there are at least a couple
hundred packages that would have to be distributed.
> > > So there is nothing to be done then?
> >
> > Well, not nothing: https://gitlab.com/sagemath/dev/trac/pipelines
>
> How do I interpret that?
Timo Kaufmann
> > > The problem with using the WSL for Sage on Windows is that, while it
> > > may be useful for development, it's not intended to be used for
> > > software distribution; in other words, distributing pre-built Windows
> > > software that relies on the WSL to work is not supported. This is why
> > > Sage for Windows uses Cygwin instead--Cygwin is just a handful of DLLs
> > > you can include in your software distribution.
> >
> > What do you mean it is not supported? You can certainly fetch pre-build binaries
> > with nix on bash-on-windows.
>
> I think you misread me. It's not for distribution of Windows
> software. Perhaps you're no familiar with the efforts we've made in
> those areas but have a look at:
> https://opendreamkit.org/2017/10/11/SageWindows/
Ah so you mean you can't package it up in a windows installer? I don't really
care much about Windows so I'm not aware, though I realize that windows support
is important for many.
> Requiring that users have Windows 10, with developer mode enabled
I wasn't aware you need to enable developer mode or something. I (somewhat
naively maybe) assumed you could just create a desktop shortcut executing
something like `bash -c sage`.
> > For what its worth I checked, and there are multiple Cygwin-specific fixes
> > in nixpkgs. So apparently it does run there. CI tests are only run on
> > x86-64-linux, aarch64-linux and darwin however. And I'm note sure what the
>
> The latest posts I saw on the subject suggest that it does not work.
> Perhaps it worked at one point but it is not maintained.
I just asked on IRC and got
"I think Cygwin, in theory, should be able to build Nix and whatnot fine. It's "just" a C++ app. Nixpkgs is a totally different story, though there's some code scattered in there still. Support for Cygwin has sort of been on/off over the years, IIRC, so -- "it works, if it works"?"
so if you're really interested, its probably best you just try. Since the Nix
binary *should* work, it *should* only be a matter of porting platform-specific
patches to packages where necessary and not already applied.
> > situation about shipping binary packages is. I assume bash-on-linux can just
> > use linux binaries, but Cygwin probably cannot. So since I don't think we
> > currently officially ship Cygwin binaries, you'd have to build everything
> > from source. It is possible to create your own server to serve binaries
> > however.
>
> Of course. And the same is true for Conda. I think Conda, for its
> macOS support, looks like the best bet right now and Sage already
> mostly works there.
As I said nix also supports macos (even first-level support). Conda may or
may not be more appropriate. I've barely heard of it and never used it, so I
can't judge.
> > > > So there is nothing to be done then?
> > >
> > > Well, not nothing: https://gitlab.com/sagemath/dev/trac/pipelines
> >
> > How do I interpret that?
>
> As an alternative to the patchbot.
Didn't you imply that there were some specific changes #25358 and #25316 need?
Erik Bray
better or worse (but mostly for better!) the most popular packaging
system for scientific Python. The whole point behind it was that
while pip works fine (generally) for installing Python packages, even
those with C extensions, it does not enable packaging non-Python
dependencies which are often needed especially for scientific
packages. So conda builds on existing technology similar to Python
virtualenvs but adds support for packaging non-Python packages. And
of course, since it allows packaging arbitrary binaries it is
no-longer necessarily Python-specific either, though it was conceived
originally primarily for Python.
It is already heavily used in much of the scientific
community--especially scientific Python--and has the most resources
and support within that community, AFAICT (I'm just saying this based
on personal experience within the community; I don't have any dta
about that).
Timo Kaufmann
> > > [...]
> >
> > As I said nix also supports macos (even first-level support). Conda may or
> > may not be more appropriate. I've barely heard of it and never used it, so I
> > can't judge.
>
> You should probably look into it then because it's currently, for
> better or worse (but mostly for better!) the most popular packaging
> system for scientific Python. The whole point behind it was that
> while pip works fine (generally) for installing Python packages, even
> those with C extensions, it does not enable packaging non-Python
> dependencies which are often needed especially for scientific
> packages. So conda builds on existing technology similar to Python
> virtualenvs but adds support for packaging non-Python packages. And
> of course, since it allows packaging arbitrary binaries it is
> no-longer necessarily Python-specific either, though it was conceived
> originally primarily for Python.
>
> It is already heavily used in much of the scientific
> community--especially scientific Python--and has the most resources
> and support within that community, AFAICT (I'm just saying this based
> on personal experience within the community; I don't have any dta
> about that).
Reading thorugh the first few pages of the user-guide it does seem interesting
in comparsion to PIP, but it doesn't offer anything my current package manager
doesn't. So since repos generally try to avoid 3rd party package managers
anyways, I don't think it would be very useful for me.
Erik Bray
Timo Kaufmann
> > Reading thorugh the first few pages of the user-guide it does seem interesting
> > in comparsion to PIP, but it doesn't offer anything my current package manager
> > doesn't. So since repos generally try to avoid 3rd party package managers
> > anyways, I don't think it would be very useful for me.
>
> Yes, but with no disrespect intended, you are not the primary audience here :)
Of course :)