How to release SymPy more often
50 views
Skip to first unread message
Aaron Meurer
Nov 12, 2019, 6:28:26 PM11/12/19
to sympy
I've been thinking about how we can release SymPy more often. It's
apparent that there are two main things that have prevented it from
happening:
- My limited time to do the release
- The release blocking issues (the issues on the milestone on GitHub)
The first issue I hope should be solved by getting more people than
just myself to do the releases. That's why I'm excited that Oscar is
doing the 1.5 release.
The second issue is harder. I think we need a better policy on what
issues are allowed to block a release. I've started a discussion at
https://github.com/sympy/sympy/issues/17885. I encourage everyone to
give input on it.
Aside from making it clearer which kinds of issues are allowed to
block a release, does anyone have any suggestions on how we can manage
release blocking issues more effectively, or generally how we can
release more often?
Aaron Meurer
apparent that there are two main things that have prevented it from
happening:
- My limited time to do the release
- The release blocking issues (the issues on the milestone on GitHub)
The first issue I hope should be solved by getting more people than
just myself to do the releases. That's why I'm excited that Oscar is
doing the 1.5 release.
The second issue is harder. I think we need a better policy on what
issues are allowed to block a release. I've started a discussion at
https://github.com/sympy/sympy/issues/17885. I encourage everyone to
give input on it.
Aside from making it clearer which kinds of issues are allowed to
block a release, does anyone have any suggestions on how we can manage
release blocking issues more effectively, or generally how we can
release more often?
Aaron Meurer
Jason Moore
Nov 12, 2019, 6:46:19 PM11/12/19
to sympy
There are two things that I think are important:
- don't include backwards incompatible changes in releases without a deprecation cycle (cycle should be measured in real time, not # cycles)
- don't introduce new features that we aren't confident we want to support as public API
If we have strict policy about those two and releasing is automated enough that it isn't a time burden, could't we theoretically release after ever PR?
Jason
--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6JEutrC1mzOpS6dHihk6NzTz4uWkhiuHKkEBOUuDqqzdA%40mail.gmail.com.
Aaron Meurer
Nov 12, 2019, 6:56:38 PM11/12/19
to sympy
To me the backwards compatibility thing is part of the problem. A lot
of the time the release blockers are about cleaning up some API that
is new since the previous release. We need to do it so that we aren't
forced to break it again in a later release. Or sometimes they are
about fixing a major regression. Regressions aren't always noticed
right away, though obviously they are noticed a lot faster if we
actually release a version with a change.
But I do agree that would could and should automate everything to the
point where we could theoretically do it every PR. I don't know that
we would actually want to release *that* often, but that's the level
of automation we need. We are pretty close as far as automation goes.
I think there are still a couple of minor things that are missing. And
we would also have to move our release script to run on a CI (which I
think is a good idea anyway).
Aaron Meurer
> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1Ai91j_8%2B1thSTtO8%2BOUfqQMi5dgqY8-v0G-G_cD_J%2B5iQ%40mail.gmail.com.
of the time the release blockers are about cleaning up some API that
is new since the previous release. We need to do it so that we aren't
forced to break it again in a later release. Or sometimes they are
about fixing a major regression. Regressions aren't always noticed
right away, though obviously they are noticed a lot faster if we
actually release a version with a change.
But I do agree that would could and should automate everything to the
point where we could theoretically do it every PR. I don't know that
we would actually want to release *that* often, but that's the level
of automation we need. We are pretty close as far as automation goes.
I think there are still a couple of minor things that are missing. And
we would also have to move our release script to run on a CI (which I
think is a good idea anyway).
Aaron Meurer
Oscar Benjamin
Nov 12, 2019, 7:23:22 PM11/12/19
to sympy
On Tue, 12 Nov 2019 at 23:46, Jason Moore <moore...@gmail.com> wrote:
>
> There are two things that I think are important:
>
> - don't include backwards incompatible changes in releases without a deprecation cycle (cycle should be measured in real time, not # cycles)
It isn't always possible to have a deprecation cycle e.g. for the
>
> There are two things that I think are important:
>
> - don't include backwards incompatible changes in releases without a deprecation cycle (cycle should be measured in real time, not # cycles)
changes to finiteness assumptions that will be released soon:
https://github.com/sympy/sympy/wiki/Release-Notes-for-1.5
Those changes broke a lot of things (or required a lot of fixes) in
SymPy itself but are clearly not generally backward compatible. There
is no way to introduce those changes with a deprecation cycle though.
> - don't introduce new features that we aren't confident we want to support as public API
I would add another requirement because there is a lot of unmaintained
code in SymPy right now. If a new module is added e.g.
https://github.com/sympy/sympy/pull/17866 then maybe there should be a
policy of saying that someone needs to commit to maintaining it.
Oscar
Aaron Meurer
Nov 17, 2019, 12:01:47 AM11/17/19
to sympy
Thinking more about the automation, I think we need to have the
release be 100% automated. Right now it is 90% automated (not counting
the problem of fixing blocker issues).
Oscar, it will be a good idea to make a note of anything in the
release process that isn't automated that should be. One thing I
forgot about is the release notes. It looks like you already added the
deprecations. We should figure out how to automate that. Maybe the bot
can do it. The highlights can't be automated because they are human
curated, so I think that means those should just be removed. Unless
there is some way we can do it with the bot as well (maybe by letting
people tag a PRs notes as a "highlighted note"?).
The AUTHORS/.mailmap isn't automated either. I think they only way
around that would be to run the authors/mailmap scripts as part of the
Travis tests, so that we require any PR with a new new author or new
git metadata to update those files. The main problem with that is that
it puts an extra burden on new contributors, by definition. So I don't
know if there's a better way to do it, or if it's worth doing at all.
It isn't difficult to do and it also can be done at any time, not just
during a release. Maybe we could add it to the CI but as an allowed
failure, that way it isn't a requirement to merge, but it will alert
people when it gets out of date.
The actual updating of files isn't automated right now, which is on
purpose. But I think we need to make it so that the release script
runs on a CI. It already runs in a Docker container, so this should
not be hard to do with one of the various CIs that support custom
containers. It would then run as a dry run on every commit. Then when
we want to do a release proper, either beta, release candidate, or
final, we tag a commit and the CI actually uploads stuff when it is
built on a tag. We've used the current release script since 2017 with
1.1, and with the exception of that 1.1 release, we haven't needed any
bugfix releases. So I think it's safe enough now that we can automate
the uploading.
The only other thing I can remember that isn't automated is various
updates that you have to do to the release script, like updating the
tarball whitelist. But if we run the script on CI that will not be a
problem because it will be updated when it needs to be, instead of
when the release happens.
There's might be other stuff that isn't quite automated, but I'm not
remembering it.
Once we have
1. Full release automation, and
2. A consistent set of rules for release blockers,
then at any point of time we can just look at the release milestone.
If it is empty, we can tag a release. If it isn't, we fix or remove
the blocking issues then tag. If it becomes streamlined enough we
won't even have to do beta or rc releases (but that would be down the
road, once we know everything is working well).
Aaron Meurer
release be 100% automated. Right now it is 90% automated (not counting
the problem of fixing blocker issues).
Oscar, it will be a good idea to make a note of anything in the
release process that isn't automated that should be. One thing I
forgot about is the release notes. It looks like you already added the
deprecations. We should figure out how to automate that. Maybe the bot
can do it. The highlights can't be automated because they are human
curated, so I think that means those should just be removed. Unless
there is some way we can do it with the bot as well (maybe by letting
people tag a PRs notes as a "highlighted note"?).
The AUTHORS/.mailmap isn't automated either. I think they only way
around that would be to run the authors/mailmap scripts as part of the
Travis tests, so that we require any PR with a new new author or new
git metadata to update those files. The main problem with that is that
it puts an extra burden on new contributors, by definition. So I don't
know if there's a better way to do it, or if it's worth doing at all.
It isn't difficult to do and it also can be done at any time, not just
during a release. Maybe we could add it to the CI but as an allowed
failure, that way it isn't a requirement to merge, but it will alert
people when it gets out of date.
The actual updating of files isn't automated right now, which is on
purpose. But I think we need to make it so that the release script
runs on a CI. It already runs in a Docker container, so this should
not be hard to do with one of the various CIs that support custom
containers. It would then run as a dry run on every commit. Then when
we want to do a release proper, either beta, release candidate, or
final, we tag a commit and the CI actually uploads stuff when it is
built on a tag. We've used the current release script since 2017 with
1.1, and with the exception of that 1.1 release, we haven't needed any
bugfix releases. So I think it's safe enough now that we can automate
the uploading.
The only other thing I can remember that isn't automated is various
updates that you have to do to the release script, like updating the
tarball whitelist. But if we run the script on CI that will not be a
problem because it will be updated when it needs to be, instead of
when the release happens.
There's might be other stuff that isn't quite automated, but I'm not
remembering it.
Once we have
1. Full release automation, and
2. A consistent set of rules for release blockers,
then at any point of time we can just look at the release milestone.
If it is empty, we can tag a release. If it isn't, we fix or remove
the blocking issues then tag. If it becomes streamlined enough we
won't even have to do beta or rc releases (but that would be down the
road, once we know everything is working well).
Aaron Meurer
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAHVvXxTAbKHtNxvKY%3DiA3QbBUu4OCWrLWzRvvCwVkL1nmhDaVw%40mail.gmail.com.
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
Oscar Benjamin
Nov 17, 2019, 6:07:39 AM11/17/19
to sympy
On Sun, 17 Nov 2019 at 05:01, Aaron Meurer <asme...@gmail.com> wrote:
>
> Thinking more about the automation, I think we need to have the
> release be 100% automated. Right now it is 90% automated (not counting
> the problem of fixing blocker issues).
>
> Oscar, it will be a good idea to make a note of anything in the
> release process that isn't automated that should be.
I will make notes and rewrite the release/README afterwards. To be
>
> Thinking more about the automation, I think we need to have the
> release be 100% automated. Right now it is 90% automated (not counting
> the problem of fixing blocker issues).
>
> Oscar, it will be a good idea to make a note of anything in the
> release process that isn't automated that should be.
honest though the reason it has taken 5 days to get to the point of
having a beta release is actually not because of the *lack* of
automation. It has taken that long because I needed to figure out what
was going on in the release script which is very slow to run and the
only way to try and fix anything is to go through a PR with full
Travis tests etc.
I am in the process of getting the right permissions to be able to use
docker at work which will speed things up but for now I am running
this on a laptop at home with a relatively slow internet connection.
The first time the release script runs it needs to get the docker
containers. Each time the release script runs (and multiple times
within each run) it needs to download a lot of files from conda. It
also runs the whole sympy test suite in between doing the other tasks.
For me right now it takes around 2 hours to run the release script.
Once the script fails it takes about 5 minutes to write the code that
fixes it but that needs to be sent back to github and it takes 1.5
hours to run the tests before they can be merged and the release
script can start again. That means that any issue in the release
script has a 4 hour turn around time to detect the problem and attempt
something to fix it.
What would make all of this faster is if the elements of the release
script were split out as separate scripts that could be tested locally
without going through a PR and docker etc. This is the case for the
mailmap script but not for others like testing the tarball whitelist.
> One thing I forgot about is the release notes. It looks like you already added the
> deprecations. We should figure out how to automate that. Maybe the bot
> can do it.
used git grep to find any deprecation warnings with the 1.5 version.
That can be automated but that only catches things that are
deprecated. The backward compatibility breaks that don't/can't have a
deprecation warning are the more important ones to document here and I
don't see any way to pick up on those automatically.
Perhaps changing the PR template so that there is an explicit backward
compatibility release notes section would help.
> The highlights can't be automated because they are human
> curated, so I think that means those should just be removed. Unless
> there is some way we can do it with the bot as well (maybe by letting
> people tag a PRs notes as a "highlighted note"?).
>
> The AUTHORS/.mailmap isn't automated either. I think they only way
> around that would be to run the authors/mailmap scripts as part of the
> Travis tests, so that we require any PR with a new new author or new
> git metadata to update those files. The main problem with that is that
> it puts an extra burden on new contributors, by definition. So I don't
> know if there's a better way to do it, or if it's worth doing at all.
> It isn't difficult to do and it also can be done at any time, not just
> during a release. Maybe we could add it to the CI but as an allowed
> failure, that way it isn't a requirement to merge, but it will alert
> people when it gets out of date.
releases there wouldn't be so much to update there. It wold be good to
split out the part of the release script that shows the authors though
because that only happens at the end whereas the update_mailmap.py
script runs at the beginning.
> The only other thing I can remember that isn't automated is various
> updates that you have to do to the release script, like updating the
> tarball whitelist. But if we run the script on CI that will not be a
> problem because it will be updated when it needs to be, instead of
> when the release happens.
easier to do if it could be checked before running the full release
script.
Full automation is a possibility here but I also think that it can be
made a lot more efficient as a mostly automated process.
Oscar
Aaron Meurer
Nov 17, 2019, 4:39:53 PM11/17/19
to sympy
slow is that it runs the full test suite. We could remove that once
it's on CI because the other CI already runs the tests (although that
would make it not be able to catch bugs like
https://github.com/sympy/sympy/issues/17884 so maybe it isn't a good
idea). The docs also take a little while to build but not an hour.
rever locally also is able to cache results so it doesn't have to
rerun things that haven't changed (the docker script reruns from
scratch for reproducibility). So we just need to clarify the process
of how to test the release vs. how to do the final release once
everything is working.
>
> What would make all of this faster is if the elements of the release
> script were split out as separate scripts that could be tested locally
> without going through a PR and docker etc. This is the case for the
> mailmap script but not for others like testing the tarball whitelist.
official release in a docker container so that it is isolated and
reproducible, but for testing you can call rever directly.
something like that. The only problem is the dependencies make it so
that it requires you to run the tests first. So we should modify the
dependency tree so the tests aren't required for anything except for
the full release process.
But ultimately we should have the bot add the authors to the release
notes. I just haven't worked out all the details and implemented yet
it. See https://github.com/sympy/sympy-bot/issues/44. The bot also
needs to be made so it can create the new release notes file
https://github.com/sympy/sympy-bot/issues/26. Both things are doable.
We just need to implement them.
>
> > The only other thing I can remember that isn't automated is various
> > updates that you have to do to the release script, like updating the
> > tarball whitelist. But if we run the script on CI that will not be a
> > problem because it will be updated when it needs to be, instead of
> > when the release happens.
>
> This is a straight-forward thing to fix but it would be quicker and
> easier to do if it could be checked before running the full release
> script.
>
> Full automation is a possibility here but I also think that it can be
> made a lot more efficient as a mostly automated process.
>
> Oscar
>
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAHVvXxS9XUS_unw844q4C-jC%2B-4EL6Hd9Gqnj%2BU8rBNUWkRK5A%40mail.gmail.com.
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages