sage -br rebuilds things that were not changed

[Marco Cognetta]

Hello,

I have been writing some code for sage and have encountered the following problem. When running ./sage -br, it goes through a long cythonizing step every time, even on files that have not been changed. I have played around with it a little and if I run ./sage -br, wait for it to finish (about 1-1.5 hours), and run ./sage -br again it will build almost instantly without the cythonizing step. However, if I change to a new branch that has no changes which would necessitate recythonizing code, it will go through the cythonizing step again. Upon switching back to the previous branch it goes through the step again.

It also seems to be choosing different areas of the code to cythonize (it has a counter for the number of files that will be cythonized which sometimes changes values between calls of ./sage -br).

Is there a nice fix for this or do I need to make dist-clean && make to just start over?

Thank you,

Marco

[Jeroen Demeyer]

On 2016-09-09 08:43, Marco Cognetta wrote:
> However, if I change to a
> new branch that has no changes which would necessitate recythonizing
> code, it will go through the cythonizing step again.

What makes you think that there are no changes which would necessitate
recythonizing?

Cython does dependency checking, so if it rebuilds something, there must
be a reason. In fact, the reason is explicitly stated when cythonizing
(with messages like building foo because it depends on bar).

[leif]

And in case you don't trust cython, you can compare files (or whole
subtrees) yourself:

git diff old_branch new_branch -- src/sage/...


-leif

[Marco Cognetta]

It has happened to me where I build sage, turn off my computer, turn it back on, and build it again. There were no changes made in the meantime and it still does the cythonizing step. Also, I have had branches that are identical (except for a new file that does not have any dependencies to be rebuilt) which will also go through this cythonizing step (first when I switch to the branch, and then again when I switch back to master and try to build it again).

[Jeroen Demeyer]

On 2016-09-09 12:56, Marco Cognetta wrote:
> It has happened to me where I build sage, turn off my computer, turn it
> back on, and build it again. There were no changes made in the meantime
> and it still does the cythonizing step.

Are you really sure that you remember this correctly? Turning off and on
your computer should have no effect (unless there is a problem with your
OS or hardware).

> Also, I have had branches that
> are identical (except for a new file that does not have any dependencies
> to be rebuilt) which will also go through this cythonizing step (first
> when I switch to the branch, and then again when I switch back to master
> and try to build it again).

Are these branches somewhere public? Then I could have a look. But right
now, I have too little information to know whether there is a real bug
or just something that you are doing wrongly.

[leif]

Jeroen Demeyer wrote:
> On 2016-09-09 12:56, Marco Cognetta wrote:
>> It has happened to me where I build sage, turn off my computer, turn it
>> back on, and build it again. There were no changes made in the meantime
>> and it still does the cythonizing step.
>
> Are you really sure that you remember this correctly? Turning off and on
> your computer should have no effect (unless there is a problem with your
> OS or hardware).

And "turn off" hopefully means properly shutting down the system (maybe
by pushing the soft-off button / closing the laptop, depending on your
OS and settings).

>> Also, I have had branches that
>> are identical (except for a new file that does not have any dependencies
>> to be rebuilt) which will also go through this cythonizing step (first
>> when I switch to the branch, and then again when I switch back to master
>> and try to build it again).
>
> Are these branches somewhere public? Then I could have a look. But right
> now, I have too little information to know whether there is a real bug
> or just something that you are doing wrongly.

And are you sure these "identical" branches are based on the same commit?

It for example makes a difference whether you fetch a branch, pull it
into some betaX, or pull it into some betaY / other branch than (the
same version of) develop, say. You may compare what 'git log' shows for
both, i.e., the last commit that's not from you, such as "Updated
SageMath version to 7.4.beta3".


As mentioned, you can compare branches directly, or just specific files
of your both branches in question. If you don't want to see the actual
changes (but rather an overview of which files differ and by how much),
you can use

git diff --stat old_branch new_branch -- src/sage/...

For the two branches which allegedly only differ by a newly created
file, the above should give just a single line for exactly that file
(plus a one-line summary on the last line).


-leif

[Marco Cognetta]

One public branch is the trac ticket #21423. I just tested the turning on and off again and it happened and then failed. It may be something weird with that branch, but it said it was up to date with the sage repo and I didn't change anything outside of the graph/generators/families.py file.

https://trac.sagemath.org/ticket/21423

[  2/299] Cythonizing sage/libs/ppl.pyx

************************************************************************

Traceback (most recent call last):

  File "setup.py", line 597, in <module>

    run_cythonize()

  File "setup.py", line 589, in run_cythonize

    'profile': profile,

  File "/home/boss/github/sage/local/lib/python2.7/site-packages/Cython-0.24.1-py2.7-linux-x86_64.egg/Cython/Build/Dependencies.py", line 906, in cythonize

    result.get(99999)  # seconds

  File "/home/boss/github/sage/local/lib/python/multiprocessing/pool.py", line 567, in get

    raise self._value

CompileError: sage/libs/pari/gen.pyx

************************************************************************

Error building the Sage library

************************************************************************

Makefile:6: recipe for target 'sage' failed

[Marco Cognetta]

It seems there are some differences that were not showing up when I was checking status.

[Paul Masson]

I'm a bit confused by your answer, because I've been seeing the same sort of behavior. For the last week I've been rebuilding a branch based on 7.4.beta2. With minor changes to one file, sage -b has been running very quickly. Today I made one minor change and the recythonizing kicked in.

In between this build and the last, I did create a new branch based on 7.4.beta4 to correct a single typo, but I didn't build that branch.

Are you saying that even rebuilding the same branch while making the same sort of changes can lead to recythonizing? Or is it because I made a new branch in the meantime?

Thanks.

[Jeroen Demeyer]

On 2016-09-09 23:34, Paul Masson wrote:
> I'm a bit confused by your answer, because I've been seeing the same
> sort of behavior. For the last week I've been rebuilding a branch based
> on 7.4.beta2. With minor changes to one file, sage -b has been running
> very quickly. Today I made one minor change and the recythonizing kicked in.
>
> In between this build and the last, I did create a new branch based on
> 7.4.beta4 to correct a single typo, but I didn't build that branch.
>
> Are you saying that even rebuilding the same branch while making the
> same sort of changes can lead to recythonizing? Or is it because I made
> a new branch in the meantime?

Sorry, I don't get what you mean. Can you specify *exactly* the steps
that you do and where you think that too much is being recythonized?

[Paul Masson]

A little over a week ago I made a new branch from 7.4.beta2 for Trac 21370. I initially ran make on this branch.

For the last week, I have made changes to one file, graphs/graph_plot.py, for that ticket.

For the last week, every time I run sage -b after modifying that one file, the process completes quickly.

Yesterday I made a new branch from 7.4.beta4 for Trac 21458. Since it was one letter in a .rst file, I didn't run build or make on that branch.

Today I switched back to the branch for 21370, made a change to that same one file, and sage -b began recythonizing.

Why would recythonizing be necessary when only changing the same file on the same branch?

Thanks.

[Jeroen Demeyer]

On 2016-09-10 00:03, Paul Masson wrote:
> Why would recythonizing be necessary when only changing the same file on
> the same branch?

You are not changing just one file. When you checkout a new branch, a
lot of files get changed. When you checkout the old branch again, a lot
of files get changed again. The fact that they are changed to have the
same content that they originally had, is irrelevant. As far as the
filesystem is concerned, those files are changed.

[Volker Braun]

Whenever you switch to a branch with a different working set you change timestamps of modified files. Git does not track timestamps. Updated timestamps cause recompilation.

[William Stein]

On Fri, Sep 9, 2016 at 3:08 PM, Volker Braun <vbrau...@gmail.com> wrote:
> Whenever you switch to a branch with a different working set you change
> timestamps of modified files. Git does not track timestamps. Updated
> timestamps cause recompilation.

So... instead of using timestamps, maybe Cython should use file hashes
to determine whether or not to recompile code? My distant memory of
when we first wrote the cython dep code for Sage long long ago was
that I used timestamps, then Craig Citro or Robert Bradshaw wrote
something better using hashes, but I'm not sure. I'm cc'ing them in
case they have any thoughts.


--
William (http://wstein.org)

[Paul Masson]

+1 That would save on time and confusion.

[Volker Braun]

There is cycache, but its currently disabled in Sage as we ran into some bugs.

[Johan S. H. Rosenkilde]

Yes, and even fixing Cythonizing to using file hashes is only a partial
solution, right: Make would still cause huge recompilations all over the
place, AFAIK.

I've completely stopped jumping directly between tickets, and always use
the following schema for checking out a ticket:

$ git checkout develop
$ git trac pull develop
$ Run "make" if develop was not up to date
$ git checkout -b <ticketno>_<ticket-description>
$ git trac pull <ticketno>

If I already had a branch checking out #ticketno, then I'll delete it
first. This way, you'll never inadvertently check out an old releases of
Sage. I'm sure many developers are using variations of this schema.

IMHO, this or a similar way of checking out should be the recommended
one in the Sage Developer's manual. It would save a lot of people a lot
of compilation time.

Best,
Johan

[Volker Braun]

Thats ok for reviewing tickets, and implemented as "git trac try <ticketno>". 

But if you want to actually make changes then this creates a new merge commit which furthermore is against the conventional order (where the feature branch is the first parent). So it makes the commit history harder to understand.

[Johan S. H. Rosenkilde]

Volker Braun writes:

> Thats ok for reviewing tickets, and implemented as "git trac try
> <ticketno>".

OK. I had a chat with Thierry Monteil and we agreed there were some
subtle differences I don't remember - but I'll take a look at "git trac
try".

> But if you want to actually make changes then this creates a new merge
> commit which furthermore is against the conventional order (where the
> feature branch is the first parent). So it makes the commit history harder
> to understand.

I've heard arguments against "gratuitously" merging the ticket to
newest release when e.g. just fixing a typo. But that seems to me to be
a rather theoretical complaint, since the ticket will be merged later on
anyway.

I agree with the problem with the merge order which can be annoying. I
don't know if there is an obscure Git command for changing that. But if
the alternative is to spend tons of time recompiling every time I'm
checking out a ticket, I'm not sure I'll do the trade ;-)

Best,
Johan

[leif]

Volker Braun wrote:
> There is cycache, but its currently disabled in Sage as we ran into some
> bugs.

Well, since quite a while though, isn't it?

Did anyone recently check whether the issue still exists, or could you
give a pointer?


-leif

[leif]

Johan S. H. Rosenkilde wrote:
> Volker Braun writes:
>
>> Thats ok for reviewing tickets, and implemented as "git trac try
>> <ticketno>".
>
> OK. I had a chat with Thierry Monteil and we agreed there were some
> subtle differences I don't remember - but I'll take a look at "git trac
> try".
>
>> But if you want to actually make changes then this creates a new merge
>> commit which furthermore is against the conventional order (where the
>> feature branch is the first parent). So it makes the commit history harder
>> to understand.

I'd rather say "impossible", or more precisely, to later review exactly
the commits that belong to the ticket in question.

> I've heard arguments against "gratuitously" merging the ticket to
> newest release when e.g. just fixing a typo. But that seems to me to be
> a rather theoretical complaint, since the ticket will be merged later on
> anyway.

Absolutely not, unless you're /creating/ a new / "the" branch on the
ticket. If you're taking an already existing branch from a ticket and
make changes to be pushed back later, you should *always* fetch, not
pull into whatever head you currently have locally.

But maybe I just misunderstood you.

For new branches, I think it's best to base them on the last *stable*
release (i.e., check out master before creating the new branch), unless
you really need fixes from later betas, or it's clear (or likely) that
you'd get merge conflicts with later betas if you base your branch on
master. (But you'd notice that sooner or later when looking at the
colour of the branch on the trac ticket -- unfortunately nobody ever
gets notified when the colour changes after trac's develop has been
updated to the next beta, something either trac or a patchbot could
implement.)

If you always base your branches on the latest *beta*, others will just
get forced to pull and build the latter (with typically loads of
completely unrelated changes), no matter whether they fetch or pull your
branch. So in other words, you're (at least potentially) moving rebuild
load to others.


-leif

[Volker Braun]

The upstream bugs are still open:

https://github.com/cython/cython/issues?q=is%3Aissue%20is%3Aopen%20cycache

[Johan S. H. Rosenkilde]

leif writes:
>> But if you want to actually make changes then this creates a new merge
>> commit which furthermore is against the conventional order (where the
>> feature branch is the first parent). So it makes the commit history harder
>> to understand.
>
> I'd rather say "impossible", or more precisely, to later review exactly
> the commits that belong to the ticket in question.

I'd say that's a Git tools question. Without flags, git log will show
the individual commits in chronological order, but you can ask to do
topsort or something. Personally, it seldom comes up for me, and when it
has, I've been able to get by using the commit numbers given by Trac or
searching through the log. Sure, I probably have not been involved in
the most complicated tickets Sage has ever seen, though.

>> I've heard arguments against "gratuitously" merging the ticket to
>> newest release when e.g. just fixing a typo. But that seems to me to be
>> a rather theoretical complaint, since the ticket will be merged later on
>> anyway.
>
> Absolutely not, unless you're /creating/ a new / "the" branch on the
> ticket. If you're taking an already existing branch from a ticket and
> make changes to be pushed back later, you should *always* fetch, not
> pull into whatever head you currently have locally.
>
> But maybe I just misunderstood you.

"whatever head" is always develop, so I'm always pulling into the newest
beta release.

> For new branches, I think it's best to base them on the last *stable*
> release (i.e., check out master before creating the new branch), unless
> you really need fixes from later betas, or it's clear (or likely) that
> you'd get merge conflicts with later betas if you base your branch on
> master. (But you'd notice that sooner or later when looking at the
> colour of the branch on the trac ticket -- unfortunately nobody ever
> gets notified when the colour changes after trac's develop has been
> updated to the next beta, something either trac or a patchbot could
> implement.)
>
> If you always base your branches on the latest *beta*, others will just
> get forced to pull and build the latter (with typically loads of
> completely unrelated changes), no matter whether they fetch or pull your
> branch. So in other words, you're (at least potentially) moving rebuild
> load to others.

I don't see why all developers shouldn't always be on the latest beta.

It's also the only version that would make sense to base *all* future
tickets on, though - as you say - many tickets might not care about the
diff between master and develop. So in the interest of minimising
compilation time for developers overall, I think we should base all
tickets on the latest beta. Otherwise, when you jump between
small-ticket-A and small-ticket-B, you might suddenly trigger a huge
recompile since B happened to need one of the 100 tickets between master
and develop, while A didn't.

Without this or a similar schema, working on multiple tickets becomes
absolutely unbearable for me due to compilation time. How do you guys
cope with that?

Best,
Johan

[Nils Bruin]

On Saturday, September 10, 2016 at 9:14:08 AM UTC-7, leif wrote:

For new branches, I think it's best to base them on the last *stable*
release (i.e., check out master before creating the new branch), unless
you really need fixes from later betas, or it's clear (or likely) that
you'd get merge conflicts with later betas if you base your branch on
master.

I would think so too. What's more, I'd say "git trac checkout <ticket number>" on a ticket that has no branch connected (yet) should do whatever is best. I don't think it bases a branch on master, but on develop. If it's clear what the best base is for new ticket branches, I'd say "git trac checkout" should do that.

[leif]

Johan S. H. Rosenkilde wrote:

> leif writes:
>>> But if you want to actually make changes then this creates a new merge
>>> commit which furthermore is against the conventional order (where the
>>> feature branch is the first parent). So it makes the commit history harder
>>> to understand.
>>
>> I'd rather say "impossible", or more precisely, to later review exactly
>> the commits that belong to the ticket in question.
>
> I'd say that's a Git tools question. Without flags, git log will show
> the individual commits in chronological order, but you can ask to do
> topsort or something. Personally, it seldom comes up for me, and when it
> has, I've been able to get by using the commit numbers given by Trac or
> searching through the log. Sure, I probably have not been involved in
> the most complicated tickets Sage has ever seen, though.

Well, probably it's just me, but I always (also) review tickets on trac
(i.e., via what git-o-lite gives); the whole branch, but also each
individual commit. You just need a browser to do so.

Having a Sage installation at hand, it's even more tedious to do so on
the command line when the relevant commits are "out-of-order" / mixed
with unrelated commits.

>>> I've heard arguments against "gratuitously" merging the ticket to
>>> newest release when e.g. just fixing a typo. But that seems to me to be
>>> a rather theoretical complaint, since the ticket will be merged later on
>>> anyway.
>>
>> Absolutely not, unless you're /creating/ a new / "the" branch on the
>> ticket. If you're taking an already existing branch from a ticket and
>> make changes to be pushed back later, you should *always* fetch, not
>> pull into whatever head you currently have locally.
>>
>> But maybe I just misunderstood you.
>
> "whatever head" is always develop, so I'm always pulling into the newest
> beta release.

Well, that doesn't make much of a difference, unless the ticket is
incidentally based on the same (which is rarely the case, since most
tickets live longer than a single beta-release cycle). And as said,
whether you pull or fetch is only relevant if you push back changes. I
guess you review and test many more tickets than you actually push
reviewer changes to.

For trivial changes, you can also put inline patches onto trac the owner
of the ticket / branch can easily apply, and for more complex changes,
you can always create a new branch on trac *without* changing the
ticket's branch to that. Then the owner of the original branch can
cherry-pick from your branch, if it's based on something else.

(With different Sage installations, you could do similar locally of
course, i.e., "backport" the changes you made on top of your develop to
master+fetched original branch, say, and push the resulting branch. I
personally often first make changes in some random Sage installation,
then moving them around, later pushing to trac from a "suited" one.)

>> For new branches, I think it's best to base them on the last *stable*
>> release (i.e., check out master before creating the new branch), unless
>> you really need fixes from later betas, or it's clear (or likely) that
>> you'd get merge conflicts with later betas if you base your branch on
>> master. (But you'd notice that sooner or later when looking at the
>> colour of the branch on the trac ticket -- unfortunately nobody ever
>> gets notified when the colour changes after trac's develop has been
>> updated to the next beta, something either trac or a patchbot could
>> implement.)
>>
>> If you always base your branches on the latest *beta*, others will just
>> get forced to pull and build the latter (with typically loads of
>> completely unrelated changes), no matter whether they fetch or pull your
>> branch. So in other words, you're (at least potentially) moving rebuild
>> load to others.
>
> I don't see why all developers shouldn't always be on the latest beta.

Time? Resources?

Note that many, I guess, have Sage installations here and there, and
most likely don't or cannot keep each of them always what you'd call
"up-to-date", but rather at one of the previous "stable" releases, or
something inbetween.

> It's also the only version that would make sense to base *all* future
> tickets on, though - as you say - many tickets might not care about the
> diff between master and develop. So in the interest of minimising
> compilation time for developers overall, I think we should base all
> tickets on the latest beta. Otherwise, when you jump between
> small-ticket-A and small-ticket-B, you might suddenly trigger a huge
> recompile since B happened to need one of the 100 tickets between master
> and develop, while A didn't.

[Random insertion: Wasn't it you who [had to] upgrade[d] to Ubuntu
16.04 just to get the next version of Emacs? ;-) ]

Obviously, there are more "recent" beta releases than stable ones. And
as mentioned, tickets and their branches age.

So you'll never have all tickets (not even a large percentage) based on
the latest beta.

It's easier if you keep one Sage installation at master, and another at
some later beta. The problem (if tickets are based on the latest beta)
IMHO is that one is too often forced to "upgrade" for no reason, at
least if one forgets to first check what release a branch was based on.
If people keep pushing back "rebased" branches, the situation just worsens.

> Without this or a similar schema, working on multiple tickets becomes
> absolutely unbearable for me due to compilation time. How do you guys
> cope with that?

moar coars? ;-)

More seriously, different Sage installations, trying to pick the most
suited when going to actually build a fetched branch. But sometimes at
least, you also want to test on a specific machine (hardware, OS/distro,
compiler, whatever).


-leif

[Daniel Krenn]

On 2016-09-10 18:36, Johan S. H. Rosenkilde wrote:
> Without this or a similar schema, working on multiple tickets becomes
> absolutely unbearable for me due to compilation time. How do you guys
> cope with that?

Having a version of each stable/rc/beta SageMath in a separate
directory. Every now and then I delete an old rc/beta. For working on a
ticket I use exactly the version of the branch of this ticket.
Recompiliation time is very low in this way (usually).

I have one additional directory "merge", which is only the git
repository without anything created for
- looking at tickets if I am in doubt about the current version (git
describe) and
- for merging in a newer stable/rc/beta into a branch.

Best wishes

Daniel

[Jeroen Demeyer]

On 2016-09-10 18:36, Johan S. H. Rosenkilde wrote:
> How do you guys cope with that?

https://en.wikipedia.org/wiki/Patience

https://ccache.samba.org/

[Jeroen Demeyer]

And I forgot to mention that I have 2 complete Sage installation trees
so I can let one compile while I use the other.

[Johan S. H. Rosenkilde]

Jeroen Demeyer writes:
>> https://ccache.samba.org/
>
> And I forgot to mention that I have 2 complete Sage installation trees
> so I can let one compile while I use the other.

Thanks, that's useful. Do you need any special setup or care for making
it work in the complicated Sage compilation system, or do you just
override gcc as suggested in the manual? How much does it speed up
compilation?

It would seem to me you could often come in the case that *both*
copies are/needs compiling: You want to work on ticket A and so checkout
that on Sage1. While it's doing a 1h compilation due to jumping between
releases, you checkout ticket B on Sage2. But that's also a release
jump, so compilation ensues. Perhaps ccache makes this moot?

Does it ever get clumsy wrt. tracking which tree is in which version and
has which ticket checked out?

I also have two Sage installations, but it's a stable one I use for
work, and a devel one. That way, if devel screws up for whatever reason,
I can still always continue working. I also use it for checking Sage
behaviour without my ticket or if devel is compiling.

Daniel Krenn writes:
> Having a version of each stable/rc/beta SageMath in a separate
> directory. Every now and then I delete an old rc/beta. For working on a
> ticket I use exactly the version of the branch of this ticket.
> Recompiliation time is very low in this way (usually).
>
> I have one additional directory "merge", which is only the git
> repository without anything created for
> - looking at tickets if I am in doubt about the current version (git
> describe) and
> - for merging in a newer stable/rc/beta into a branch.

Thanks.
Do you feel this can get clumsy?
Like, how do you find which release have which ticket checked out?
When you merge in a new release into a branch you will have high
compilation time - but perhaps you don't do this too often?



I seriously think a lot of beginners are struggling with compilation
time when working with Sage, inadvertently jumping a lot between
releases. Apparently, my workflow is not popular, but could we still
add something to the Development manual to help people?

Best,
Johan

[Jeroen Demeyer]

On 2016-09-11 10:48, Johan S. H. Rosenkilde wrote:
>
> Jeroen Demeyer writes:
>>> https://ccache.samba.org/
>>
>> And I forgot to mention that I have 2 complete Sage installation trees
>> so I can let one compile while I use the other.
>
> Thanks, that's useful. Do you need any special setup or care for making
> it work in the complicated Sage compilation system, or do you just
> override gcc as suggested in the manual?

I am using system-wide ccache and my OS sets up the environment
variables correctly. But a Sage-installed ccache should also "just work".

> How much does it speed up
> compilation?

A *lot* for everything involving compiling C code. So in the case of the
Sage library, the Cythonizing still needs to be done, but the second
step can be very fast. For packages, the most annoying ones are those
which do a lot of tuning.

> Does it ever get clumsy wrt. tracking which tree is in which version and
> has which ticket checked out?

Well, there is "git log" or "git status" for that.

> I also have two Sage installations, but it's a stable one I use for
> work, and a devel one. That way, if devel screws up for whatever reason,
> I can still always continue working. I also use it for checking Sage
> behaviour without my ticket or if devel is compiling.

In the two Sage trees that I have, I also try to use one as "stable"
release for real work. However, it does not matter much which branch
that one is on (as long as it's not too outdated). If 7.3 works for real
work, then 7.4.beta4 + some unrelated ticket most likely works too.

[Johan S. H. Rosenkilde]

leif writes:
> Well, probably it's just me, but I always (also) review tickets on trac
> (i.e., via what git-o-lite gives); the whole branch, but also each
> individual commit. You just need a browser to do so.

I use git-o-lite a lot too, but often switch back to my Emacs and git
log using Magit :-P

> Having a Sage installation at hand, it's even more tedious to do so on
> the command line when the relevant commits are "out-of-order" / mixed
> with unrelated commits.

I mostly take the commit id from trac. But yes, without git-o-lite it
can get difficult to find the commits you need. I'm sure there are "git
log" flags for it, but I'm not an expert. It just hasn't come up enough
for me to really try and find a solution.

>> "whatever head" is always develop, so I'm always pulling into the newest
>> beta release.
>
> Well, that doesn't make much of a difference, unless the ticket is
> incidentally based on the same (which is rarely the case, since most
> tickets live longer than a single beta-release cycle).

I just wanted to stress that it's not a random choice. Indeed, barring
the "history becomes difficult to read", I don't see any argument
against all devs always working on latest beta, in which case it would
always make sense to update all tickets to that. We could even
automatically upgrade all tickets to latest beta on releases. Though
based on this discussion, I'd guess many of you would get angry due to
"history becomes more difficult to read".

> For trivial changes, you can also put inline patches onto trac the owner
> of the ticket / branch can easily apply, and for more complex changes,
> you can always create a new branch on trac *without* changing the
> ticket's branch to that. Then the owner of the original branch can
> cherry-pick from your branch, if it's based on something else.

This is a good idea that would allow me to continue using my current
workflow, without annoying other devs.

> (With different Sage installations, you could do similar locally of
> course, i.e., "backport" the changes you made on top of your develop to
> master+fetched original branch, say, and push the resulting branch. I
> personally often first make changes in some random Sage installation,
> then moving them around, later pushing to trac from a "suited" one.)

Aha, sort of like what Daniel wrote, except that fetching the branch
into "random Sage installation" means you don't get the low compilation
time benefits.

>> I don't see why all developers shouldn't always be on the latest beta.
>
> Time? Resources?
>
> Note that many, I guess, have Sage installations here and there, and
> most likely don't or cannot keep each of them always what you'd call
> "up-to-date", but rather at one of the previous "stable" releases, or
> something inbetween.

What time? What resources? My dev model needs only a single Sage
installation and no extra software to achieve overall very low
compilation time. The other suggestions I've heard need multiple Sages,
extra software and/or does not minimise compilation time. Checking out a
ticket takes a whopping extra 3 seconds (if it takes longer it's because
of merge errors which I would have to fix anyway).

Note I have one extra Sage tree to ensure that I have a working Sage at
any time. It's on latest stable and I never develop on it.

> [Random insertion: Wasn't it you who [had to] upgrade[d] to Ubuntu
> 16.04 just to get the next version of Emacs? ;-) ]

Hmm, no. I've used Arch linux for 3 years now because Ubuntu has too old
software. But I've twice had major compilation issues due to Arch
upgrading to a new release of some dependency (e.g. ranlib or gcc) long
before any other distro.

> Obviously, there are more "recent" beta releases than stable ones. And
> as mentioned, tickets and their branches age.
>
> So you'll never have all tickets (not even a large percentage) based on
> the latest beta.
>
> It's easier if you keep one Sage installation at master, and another at
> some later beta. The problem (if tickets are based on the latest beta)
> IMHO is that one is too often forced to "upgrade" for no reason, at
> least if one forgets to first check what release a branch was based on.
> If people keep pushing back "rebased" branches, the situation just worsens.

OK, but I just still don't really see the problem of being "forced to
upgrade". I don't see how it hurts me that I'm always working on the
latest beta - my patch has to work with that anyway. Except for the
"history is more difficult to understand" argument. But is that really
enough of an argument to jump through hoops and rings for achieving low
compilation time in another way?

> More seriously, different Sage installations, trying to pick the most
> suited when going to actually build a fetched branch. But sometimes at
> least, you also want to test on a specific machine (hardware, OS/distro,
> compiler, whatever).

OK, thanks for describing your workflow.

Best,
Johan

[leif]

Johan S. H. Rosenkilde wrote:
>

Well, making changes doesn't necessarily imply fetching branches, nor
[immediate] (re)compilation. Also, occasionally I just "manually" grab
changes, i.e., patches or even only hunks, and apply them temporarily,
later creating a "clean" branch, probably elsewhere, in order to push or
just properly do final testing.

What would have to get rebuilt of course also varies broadly, depending
on what you change (or what a fetched or pulled branch brings with it).

>>> I don't see why all developers shouldn't always be on the latest beta.
>>
>> Time? Resources?
>>
>> Note that many, I guess, have Sage installations here and there, and
>> most likely don't or cannot keep each of them always what you'd call
>> "up-to-date", but rather at one of the previous "stable" releases, or
>> something inbetween.
>
>
> What time? What resources? My dev model needs only a single Sage
> installation and no extra software to achieve overall very low
> compilation time. The other suggestions I've heard need multiple Sages,
> extra software and/or does not minimise compilation time. Checking out a
> ticket takes a whopping extra 3 seconds (if it takes longer it's because
> of merge errors which I would have to fix anyway).

One Sage installation means one machine (you "always" have access to /
can currently use), with a specific processor, operating system/distro
version, and a single tool chain.

> Note I have one extra Sage tree to ensure that I have a working Sage at
> any time. It's on latest stable and I never develop on it.
>
>> [Random insertion: Wasn't it you who [had to] upgrade[d] to Ubuntu
>> 16.04 just to get the next version of Emacs? ;-) ]
>
> Hmm, no. I've used Arch linux for 3 years now because Ubuntu has too old
> software. But I've twice had major compilation issues due to Arch
> upgrading to a new release of some dependency (e.g. ranlib or gcc) long
> before any other distro.

Then you know what "upgrades" /can/ bring.

>> Obviously, there are more "recent" beta releases than stable ones. And
>> as mentioned, tickets and their branches age.
>>
>> So you'll never have all tickets (not even a large percentage) based on
>> the latest beta.
>>
>> It's easier if you keep one Sage installation at master, and another at
>> some later beta. The problem (if tickets are based on the latest beta)
>> IMHO is that one is too often forced to "upgrade" for no reason, at
>> least if one forgets to first check what release a branch was based on.
>> If people keep pushing back "rebased" branches, the situation just worsens.
>
> OK, but I just still don't really see the problem of being "forced to
> upgrade". I don't see how it hurts me that I'm always working on the
> latest beta - my patch has to work with that anyway. Except for the
> "history is more difficult to understand" argument. But is that really
> enough of an argument to jump through hoops and rings for achieving low
> compilation time in another way?

(Re)build time and time for docbuilding and testing isn't the only issue
(besides history); betas are betas, and pretty regularly bring new
issues you don't want to deal with / be disturbed by when working on
something else. (Which doesn't mean stable releases would always be
better in that regard.)


-leif

[leif]

Johan S. H. Rosenkilde wrote:

> I seriously think a lot of beginners are struggling with compilation
> time when working with Sage, inadvertently jumping a lot between
> releases. Apparently, my workflow is not popular, but could we still
> add something to the Development manual to help people?

Well, jumping between different tickets / switching branches too often
is certainly not something for a beginner.

They'll presumably though learn quickly what kind of changes are
"dangerous" w.r.t. rebuild time...


Interestingly, so far nobody mentioned what the patch/buildbots do.

When Sage was "relocatable", it was much easier to maintain different
Sage installations. For temporary / scratch installations just to test
tickets and throw them away afterwards, without much rebuild effort,
it's still possible to create tarballs of built vanilla releases (kind
of your own bdists), but because of the hardcoded absolute paths, this
(and other things) have become a headache.


-leif

[Paul Masson]

> But a Sage-installed ccache should also "just work". 

Jeroen, could you elaborate on this? How does one use ccache most simply with Sage?

If this is currently the only way to speed up rebuilding when switching between branches, then I would like to add a concise description to the appropriate documentation file.

[leif]

Paul Masson wrote:
>> But a Sage-installed ccache should also "just work".
>
> Jeroen, could you elaborate on this? How does one use ccache most simply
> with Sage?

./sage -i ccache

That's all, unless you want to configure it differently.

(After installation, see local/etc/ccache.conf.)


-leif

[Clemens Heuberger]

Basically, I use the same scheme; I am using git worktree in order not to get
lost (for instance, this prohibits checking out the same branch in several
directories and I can use git branch --merged and friends over all these
directories).


Regards,

Clemens