Fwd: [sage-trac] #25814: Upgrade to cysignals 1.7.2

[Erik Bray]

Volker, be reasonable. You should know that there have been unique
challenges involved in getting a buildbot set up for Windows, not the
least of which being simply the availability of dedicated hardware.
In the meantime, I *am* the buildbot for Windows; I build and test
each tagged release myself and report back results quite reliably.
It's not ideal or sustainable long term but ''we are working on
that''. It's kind of an insult to the hard work I've done ensuring
that Windows *is* a fully supported platform to claim that it isn't.
Meanwhile it is available on the download page and has many active
users. If anything you could say it's better supported than most
platforms.

As for this issue, the problem in Cysignals was fixed back in April,
but the fix never included in Sage before now because it simply
required making a new release, which we figured we could get around to
before the next Sage release. I wrote an e-mail to both you and
Jeroen personally asking that we could ensure that a new Cysignals
release is made in time for Sage 8.3. Jeroen took care of his end of
that without complain. Now please take care of yours.

Best,
E

---------- Forwarded message ---------
From: sage-trac <tr...@sagemath.org>
Date: Tue, Jul 17, 2018 at 12:03 AM
Subject: Re: [sage-trac] #25814: Upgrade to cysignals 1.7.2
To:


#25814: Upgrade to cysignals 1.7.2
-------------------------------------+-------------------------------------
Reporter: jdemeyer | Owner:
Type: enhancement | Status: positive_review
Priority: blocker | Milestone: sage-8.3
Component: packages: | Resolution:
standard |
Keywords: upgrade, | Merged in:
cysignals |
Authors: Jeroen Demeyer | Reviewers: Erik Bray
Report Upstream: N/A | Work issues:
Branch: | Commit:
u/embray/pkgs/cysignals/update-1.7.2|
a41caf4adf015103499f96e6fa69a94bfb530b82
Dependencies: | Stopgaps:
-------------------------------------+-------------------------------------

Comment (by vbraun):

No buildbot => not a fully supported platform

--
Ticket URL: <https://trac.sagemath.org/ticket/25814#comment:11>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple,
Mathematica, and MATLAB

[Volker Braun]

I do appreciate your hard work in making Cygwin work. But IMHO thats a rather high-risk upgrade with little reward. Nobody stops you from merging that ticket when building cygwin binaries. And realistically we don't have anyone developing on Cygwin only at this point.

[ ]   I want to delay all other tickets by 1-2 weeks so that this one Cygwin bug can be fixed.

[Travis Scrimshaw]

On Wednesday, July 18, 2018 at 8:13:12 AM UTC+10, Volker Braun wrote:

I do appreciate your hard work in making Cygwin work. But IMHO thats a rather high-risk upgrade with little reward. Nobody stops you from merging that ticket when building cygwin binaries. And realistically we don't have anyone developing on Cygwin only at this point.

I suspect we will have a few people at the upcoming SageDays next week who will be doing that (or at least, we will be trying to get to do that). I also had to develop on Cygwin for a few weeks while trying to debug issues with the Linux side of my machine.

[X]   I want to delay all other tickets by 1-2 weeks so that this one Cygwin bug can be fixed.

unless you (Volker) can guarantee that the next full release will be fully ready by Monday morning Eastern US time. However, that is only because it would be more prudent for the SageDays.

Best,

Travis

[Volker Braun]

On Wednesday, July 18, 2018 at 5:46:35 AM UTC+2, Travis Scrimshaw wrote:

unless you (Volker) can guarantee that the next full release will be fully ready by Monday morning Eastern US time. However, that is only because it would be more prudent for the SageDays.

Thats likely if we don't start merging high-risk tickets. 

[Erik Bray]

On Wed, Jul 18, 2018 at 12:13 AM Volker Braun <vbrau...@gmail.com> wrote:

There's at least a few things I should say here.

> I do appreciate your hard work in making Cygwin work. But IMHO thats a rather high-risk upgrade with little reward. Nobody stops you from merging that ticket when building cygwin binaries.
> And realistically we don't have anyone developing on Cygwin only at this point.

This isn't a development-related bug it's a runtime bug; so I don't
understand your comment "realistically we don't have anyone developing
on Cygwin only at this point". And it's a pretty serious bug IMO
because it can cause the Sage process to simply *exit* with no error
message and an error code of zero; granted this only happens in some
extreme stack corruption scenarios, such as a stack overflow or
possibly some other situations. In most software, especially plain
Python software, that is not something we would have to worry about.
However, for something like Sage and much of the mathematics software
it wraps this is not as far-fetched a scenario, it's a very painful
result to debug.

All that said, your claim that this is a "high-risk" upgrade is also
highly specious. This is upgrading Cysignals from 1.7.1 to 1.7.2 [1]
which contains a couple bug fixes, the most significant of which (in
terms of patch size) impacts Cygwin only. So I'm not sure what
information you're going on when you call something "high risk". I
also think it's worth emphasizing that this bug is a regression: It
did not happen in the previous Sage release.

Also going back to "developing on Cygwin" it depends on what you mean
by "developing". Everyone who uses Sage is "developing" on some
level. Are you saying there are no Sage users on Windows?

> [ ] I want to delay all other tickets by 1-2 weeks so that this one Cygwin bug can be fixed.

This is again specious, and needlessly disparaging and antagonistic.
I don't believe if takes 2 weeks to run CI tests--we are in the
process of disproving that in #24655 [2] and related work (and if it
does take that long that's a process problem, not something you can
use as a cudgel).

I hate to sound like a broken record (having to write this phrase yet
again is itself repetitive), but there are things we could do
differently that would save yourself, me, and others a lot of sense of
antagonism every time you're trying to come out with a release. How
can you claim that a release would be "delayed by 1-2 weeks" when
there is no agreed upon and published release schedule?

The purpose of having a release schedule is, among other things, the
way to avoid these kinds of struggles. This would not be so important
on a smaller-project developed by only one or two people. But for a
project consisting of a large community of developers--most of whom
also have other priorities and are not always actively working on Sage
(even me!)--this kind of communication is absolutely necessary in
order to give people a chance to plan and react. It's quite
normal--common even--to have someone working an issue that they
absolutely plan to have ready for an upcoming release, but need to set
aside temporarily over other priorities, with intention to finish
before the release. But if they have no idea when that release is
going to happen, how can they plan accordingly? Sure, it can occur
that a major new feature, or a controversial bug fix simply won't be
ready in time for a release, and that's tough luck. But you vastly
increase the likelihood of that happening when people don't know when
the release is going to be and can't plan ahead for that. Take that
scenario times N and it gets that much worse.*

Currently, the only obvious indication that a release is imminent is
that you unilaterally decide to tag something a "release candidate".
When is that going to happen? After there's been a "beta9"? Is there
ever a "beta10"? "beta11"? The tag history seems to indicate there
could be. This time the "release candidate" came after "beta8".
Sometimes it comes after "beta6". I know there is usually some logic
behind this, such as an upcoming Sage days, but you can't expect
everyone to know that, or to read your mind--I'll come back to that
though.

After the "release candidate 0" point, there is also no communication
as to exactly how soon after you tag that "release candidate" you plan
to make the final release. You would think there would be some
communication of that at least on sage-devel, the primary channel for
Sage developers to communicate with each other and make decisions
about the direction of the project. Instead it just sort of "happens"
as though it were a force of nature.

By contrast: Looking at the past releases (informally; I haven't
precisely quantized this) it seems there's a new release roughly every
three to five months. So let's say we plan to come out with a release
once every four months, which is easy to keep track of. It would be
good even to pin exact dates well in advance: This does not mean those
exact dates must be kept--there are not specific market forces holding
us to them. Various factors can affect them such as holidays,
availability of the principal actors, serious blocker issues, etc.
It's okay to slip a couple days to a couple weeks if need be,
especially if this is communicated as much in advance if possible.
But having some dates planned out well in advance still helps
at-a-glance planning and decision making. There is also nothing
preventing the occasional faster, intermediate release when
appropriate (such as if there is a SageDays coming up for which some
new features are desirable). Again, this still needs to be
communicated in advance.

With a scheduled (but flexible) target release date in place, one then
has a basis for setting policies such as cut-off points for disruptive
changes in the release branch. Somewhere between 2 weeks to a month a
head of time, depending on how often release blockers are found during
development. Maybe three weeks is good enough. Knowing that this
cutoff point is coming also gives the development community time to
cooperatively triage issues (by actually using milestones!) in order
to better come up with a list of issues that can reasonably be
accomplished by the next release. If done, even a little bit, this
helps the release manager by reducing the number of issues we need to
consider for planning a release. A triaging process also allows for
debate/discussion over the relative importance of an issue further in
advance, so it doesn't have to be as much of a panicked "last minute"
fight. If there's a question about availability of continuous
integration workers, this can also be mitigated (if it isn't already)
by giving build priority to changes targeted for a soon-upcoming
release. This can be done based on various issue metadata.

This may sound idealized, and it is--this will never be perfect--but
at least having a process that is documented and agreed upon makes a
lot more possible with significantly less stress and antagonism
involved. It gives everyone time to have their needs addressed (and
does not guarantee that their needs will be met, but at least properly
addressed), and leaves less ambiguity and unilateralism about the
decision-making process.

If setting a release "schedule" is somehow too impractical, there
should at the very least be a process for preparing a release, which
includes the aforementioned "grace period", triaging that precedes or
coincides with the grace period, and above all else *communication*
about the release plan.

As someone with politically anarchist leanings I appreciate the
anarchic structure of the Sage development community (and open source
in general). But that doesn't work well if there aren't
democratically agreed-upon and documented processes. It also doesn't
work well if the development cycle is entirely decided (if even with
good reasons) by one person who doesn't communicate their plans or
reasoning in a public venue. That just feels more like chaos than
anarchy to me.

Thanks,
E


* Case in point: As I wrote previously, I made this fix back in April,
but tabled getting the fix directly into Sage since I thought it would
be no big deal to make a Cysignals release and update Sage's
dependency on it; at the time we determined it was a low-risk,
low-effort issue that could be handled before the next
release--whenever that might be... Given that "release candidate" is
the only direct evidence that a release is imminent, I took it upon
myself to e-mail you directly to ensure that the release could account
for some otherwise ready work that needed to be included. That should
at least indicate that it was "important enough". Instead that was
summarily ignored, which is mysterious at best.

[1] https://github.com/sagemath/cysignals/compare/1.7.1...1.7.2
[2] https://trac.sagemath.org/ticket/24655

[Jeroen Demeyer]

On 2018-07-18 13:34, Erik Bray wrote:
> This is again specious, and needlessly disparaging and antagonistic.
> I don't believe if takes 2 weeks to run CI tests

Just replying on this point: the problem is that many things are not
caught by CI, especially things related to the build system and portability.

[Volker Braun]

On Wednesday, July 18, 2018 at 1:34:56 PM UTC+2, Erik Bray wrote:

All that said, your claim that this is a "high-risk" upgrade is also
highly specious.  This is upgrading Cysignals from 1.7.1 to 1.7.2 [1]
which contains a couple bug fixes, the most significant of which (in
terms of patch size) impacts Cygwin only.

I haven't looked at the diff set, but we did have regressions from minor cysignals changes before. Its just that signal handlers, due to their asynchronous nature and interactions with the OS, are notoriously hard to reason about. 

[Timo Kaufmann]

As one small data point I can say that I already upgraded cysignals in nixpkgs (tested Linux, although it could technically be used on other platforms too). I didn't run into any obvious issues.

I also think we should merge that upgrade for 8.3. Erik put in a lot of work to make that platform supported and I think the least we should do is support him in supporting it :)

Sage has a lot of tests. It might even test a little too much (making the test suite brittle). I'm sure they would catch most regressions faster than 1-2 weeks.

And worst case, if a regression slips through, why are we not doing bug fix releases? E.g. sage 8.3.1.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Samuel Lelievre]

Le mercredi 18 juillet 2018 23:44:33 UTC+2, Volker Braun:

>
> I haven't looked at the diff set, but we did have regressions
> from minor cysignals changes before. Its just that signal handlers,
> due to their asynchronous nature and interactions with the OS,
> are notoriously hard to reason about.

I would really love #25814 to be merged for Sage 8.3.

Can you please give it a try?

[Volker Braun]

On Wednesday, July 18, 2018 at 1:34:56 PM UTC+2, Erik Bray wrote:

As I wrote previously, I made this fix back in April,
but tabled getting the fix directly into Sage since I thought it would
be no big deal to make a Cysignals release and update Sage's
dependency on it

By definition, if fixing a bug can easily wait for a month or two then its not a blocker.

[Erik Bray]

That's certainly true! But that's why we need a window in the first
place in which we can discover and find those issues. If what Volker
is saying in terms of adding additional 1-2 weeks of just "breathing
room" then I agree with that and I apologize if I misunderstood.
Though I do disagree with the assertion that the Cysignals update is
"high-risk" (I would say that is much more the case with
https://trac.sagemath.org/ticket/25857 which is why I mostly agree
with Volker's analysis of that one (though I don't like the unilateral
"blocker -> major" changes either).

What I do believe was disparaging and antagonistic was the wording,
not the substance.

[Erik Bray]

I agree of course, and if you don't believe me you are welcome to look
at the diff. However, you could take my word for it that virtually
nothing changed w.r.t. actual signal handling. There was one
`#define` added as an alias for some platforms, and the rest is
completely Cygwin-specific in `#ifdef` blocks, and is well-tested.

[Erik Bray]

On Wed, Jul 18, 2018 at 11:53 PM Timo Kaufmann <eisf...@gmail.com> wrote:
>
> As one small data point I can say that I already upgraded cysignals in nixpkgs (tested Linux, although it could technically be used on other platforms too). I didn't run into any obvious issues.
>
> I also think we should merge that upgrade for 8.3. Erik put in a lot of work to make that platform supported and I think the least we should do is support him in supporting it :)
>
> Sage has a lot of tests. It might even test a little too much (making the test suite brittle). I'm sure they would catch most regressions faster than 1-2 weeks.
>
> And worst case, if a regression slips through, why are we not doing bug fix releases? E.g. sage 8.3.1.

I have suggested that many times as well, but have been told it's too
much trouble for some reason. I didn't bring it up this time just to
keep the topic narrowed.

One thing I should be clear about is that I greatly respect the amount
of time and effort Volker puts into being release manager as it is.
And I know from experience that it's just that much more extra work to
maintain bug fix releases (but I also know that it's not *that much*
either, I'd say so even for Sage).

If we could configure the buildbots to also run tests against a bug
fix release branch (separate from develop) and I can figure out how
the binary releases are done I would by happy to make bug fix releases
on an as-needed basis.

> On July 18, 2018 11:44:33 PM GMT+02:00, Volker Braun <vbrau...@gmail.com> wrote:
>>
>> On Wednesday, July 18, 2018 at 1:34:56 PM UTC+2, Erik Bray wrote:
>>>
>>> All that said, your claim that this is a "high-risk" upgrade is also
>>> highly specious. This is upgrading Cysignals from 1.7.1 to 1.7.2 [1]
>>> which contains a couple bug fixes, the most significant of which (in
>>> terms of patch size) impacts Cygwin only.
>>
>>
>> I haven't looked at the diff set, but we did have regressions from minor cysignals changes before. Its just that signal handlers, due to their asynchronous nature and interactions with the OS, are notoriously hard to reason about.
>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[Erik Bray]

We must be using different definitions of "blocker" then. Just
because a bug doesn't prevent development from continuing does not
mean it isn't a *release* blocker. Any bug that introduces
regressions into the test suite should be a blocker if it means not
being able to release a product with the test suite fully passing.

Indeed it would have been convenient to fix it sooner since test runs
would have stopped failing earlier. But "known issues" during
development are less important that known issues (that are fixed!) at
release time.

Certainly, if there is a known bug that cannot be reasonably fixed in
time for a release then reasonable exceptions have to be made such as
marking the test as a "known failure" or disabling it. Sage's test
runner would actually probably benefit a lot from a "known failure"
flag for tests (though it may have to be platform-specific, which is
tricky).

In any case, this still only proves my point that if I actually *knew*
when a release was coming up I'd have pressed to get that done sooner.
If there's a fix for a known issue there's no reason to exclude it
from a release.

[Volker Braun]

On Thursday, July 19, 2018 at 10:16:46 AM UTC+2, Erik Bray wrote:

If there's a fix for a known issue there's no reason to exclude it
from a release.

Look at it this way, we are delaying hundreds of tickets by a week to fix your pet bug 3 months earlier. Of course you can forever argue about the relative importance of bugs, but if you compare the accumulated bug-time-in-product then excluding it and releasing earlier is the right thing to do.

[Sébastien Labbé]

We must be using different definitions of "blocker" then.  Just
because a bug doesn't prevent development from continuing does not
mean it isn't a *release* blocker.  Any bug that introduces
regressions into the test suite should be a blocker if it means not
being able to release a product with the test suite fully passing.

The definition of "blocker" ticket must be adapted to the release calendar we choose to have. If the definition of blocker ticket is too inclusive, than it is just impossible to respect the rule of 1 release each 3 months approximatively. Then, the impact of many late blocker tickets becomes that we do 2 or 3 releases per year instead of 4. But this does not solves the problem. There will still exist some blocker tickets which will ask again for a lower frequency of releases at the end of each cycle. It may even make things even worse as releasing less often means we stabilize the development branch (rc) less often, which means it is harder to stablize it...

[Jeroen Demeyer]

On 2018-07-19 11:04, Sébastien Labbé wrote:
> The definition of "blocker" ticket must be adapted to the release
> calendar we choose to have.

I really disagree with this. I think it's important that the calender is
adjusted depending on blocker tickets, not the other way around.

[Erik Bray]

I've said this already but that's a process problem. If we made a
release branch there would be nothing stopping anyone from continuing
to merge into the mainline while a release is tidied up. This is an
artificial "crisis" created through process. It reminds me of the
US's asinine debt ceiling and the fiscal cliff [1].

[1] https://en.wikipedia.org/wiki/United_States_fiscal_cliff

* Also, "your pet bug" implies that it's of no importance, and
dismisses the reasons it matters at all with a turn of phrase. I'm
not a perfect communicator either but please, spare me.

[Erik Bray]

+1