sage -upgrade aborts - uncommitted changes

[Kevin Horton]

I'm trying to upgrade sage on two boxes from 4.7 to 4.7.1. "sage -upgrade" fails with "There are uncommitted changes in the Sage root repository. Aborting."

On one box, "sage -hg diff" found that I had modified SAGE_ROOT, so I undid that. But it still complains about uncommitted changes, even though "sage -hg diff" finds nothing. On the second box, "sage -hg diff" finds nothing, yet the upgrade attempt fails for uncommitted changes.

Any suggestions would be much appreciated. I am a relative sage and hg neophyte, so don't make any assumptions about my knowledge or what troubleshooting I may have already done.

Thanks,
--
Kevin Horton
Ottawa, Canada

[Maarten Derickx]

Dear Kevin,

In the sage source tree there are multiple repositories, so you probably fixed it for one repository but still have uncommitted changes in others.

Below is a list of the 5 repositories relatively to SAGE_ROOT (the . one is SAGE_ROOT itself)

., data/extcode, devel/sage/, devel/sagenb/, and local/bin/

To check if there are uncommitted changes in one of these repositories you should cd to the directory an then execute 

sage -hg diff

there

[RegB]

Thanks,
That appeared to work for me on all the repositories except local/bin
in that one I got the dreaded message;
"Not trusting file /usr/local/sage-4.7/local/bin/.hg/hgrc from
untrusted user xyz, group root"
I tried all the usual chown, chmod things on everything from sage_root
and down.


On Aug 22, 10:55 am, Maarten Derickx <m.derickx.stud...@gmail.com>
wrote:

[Kevin Horton]

sage -hg diff finds nothing in any of the 5 repositories suggested above.  sage -hg st does find one extra file in devel/sage:

% sage -hg st

? sage/rings/finite_field_givaro.pyx

An attempt to do sage -upgrade still fails with:

There are uncommitted changes in the Sage root repository. Aborting.

The situation is exactly the same on both computers.

--

Kevin Horton

[Maarten Derickx]

I don't know if you are a developer, but it might be possible that you still have have queue in one of your repositories. To test this do

    sage -hg qdiff

in the different repositories.

Also did you try removing (or beter moving outside the sage directory) the not tracked file somewhere?

Also can you give the output of the command:

    whoami

and 

    ls -l /usr/local/sage-4.7/local/bin/.hg/

   

    

[Maarten Derickx]

ps http://mercurial.selenic.com/wiki/Trust

might be of help considering the untrusted user

[Kevin Horton]

I am not a sage developer. I have never knowing used mercurial queues.

I tried moving that untracked file aside, but I still got the same result when I tried to upgrade sage.

% whoami
kwh

% ls -l /usr/local/sage-4.7/local/bin/.hg/
ls: /usr/local/sage-4.7/local/bin/.hg/: No such file or directory

Note: sage is installed in /Users/kwh/apps/sage

ls -l /Users/kwh/apps/sage/local/bin/.hg
total 128
-rw-r--r-- 1 kwh staff 57 5 Mar 08:21 00changelog.i
drwxr-xr-x 2 kwh staff 68 5 Mar 08:21 attic
-rw-r--r-- 1 kwh staff 8 19 Jul 17:12 branch
-rw-r--r-- 1 kwh staff 95 5 Mar 08:21 branch.cache
-rw-r--r-- 1 kwh staff 95 19 Jul 17:12 branchheads.cache
-rw-r--r-- 1 kwh staff 2808 23 Aug 08:15 dirstate
-rw-r--r-- 1 kwh staff 52 5 Mar 08:21 hgrc
-rw-r--r-- 1 kwh staff 5 5 Mar 08:21 last-message.txt
drwxr-xr-x 4 kwh staff 136 5 Mar 08:21 patches
-rw-r--r-- 1 kwh staff 23 5 Mar 08:21 requires
drwxr-xr-x 9 kwh staff 306 19 Jul 17:12 store
-rw-r--r-- 1 kwh staff 19981 5 Mar 08:21 tags.cache
-rw-r--r-- 1 kwh staff 7 19 Jul 17:12 undo.branch
-rw-r--r-- 1 kwh staff 66 19 Jul 17:12 undo.desc
-rw-r--r-- 1 kwh staff 2808 19 Jul 17:12 undo.dirstate

Is there a log file somewhere that might provide more info on why sage refuses to upgrade?

In the future, it could be useful if sage would provide specific details on which uncommitted changes caused the upgrade to abort.

[Maarten Derickx]

Dear Kevin,

The two commands I gave where intender for RegB who had trouble with the untrusted hg message sorry for the confusion.

Well according to your error message sage cannot upgrade since the hg repository in  /Users/kwh/apps/sage has uncommited changes. But if that doesn't have any changes there must be something strange going on and maybe your hg repository got corrupted for some strange reason. It will be very hard for me to find out what exactly is going on without having acces to the machine. 

Note that sage -upgrade is know not to be very stable, it's existence is mainly for saving compilation time since rebuilding only the changed files is much faster then rebuilding the entire sage installation from source.

If you are just interested in having a newer version of sage I suggest you just install a new sage from scratch (either download a binary or manually build source code release) since doing this will take a lot less human effort then debugging the specific problems you have with the upgrade process. 

You don't have to be worried about your sage worksheet files since they are stored outside of the place where you installed sage.

I hope it doesn't bother you to much to do a reinstall.

Thanks,

Maarten

[Kevin Horton]

Dear Maarten,

I have a hard time believing it could be repository corruption, as I have the same symptoms on two sage installations - one on OS X and one on Linux. I have been using sage -upgrade for quite some time, and it has always worked well until the upgrade from 4.7 to 4.7.1.

I gave up on the upgrades this morning, and downloaded the binary installer for OS X on that machine and have a source installation churning away on the Linux machine.

Thanks for your help.

Best regards,

[leif]

On 24 Aug., 21:25, Kevin Horton <khorto...@rogers.com> wrote:
> On 2011-08-24, at 13:41 , Maarten Derickx wrote:
> I have a hard time believing it could be repository corruption, as I have the same symptoms on two sage installations - one on OS X and one on Linux.  I have been using sage -upgrade for quite some time, and it has always worked well until the upgrade from 4.7 to 4.7.1.

This *might* be due to incompatible Mercurial versions. (Yes, they
managed to break compatibility inbetween.)


-leif

[Keshav Kini]

Mercurial versions in Sage haven't changed between 4.7 and 4.7.1. In any case, incompatible Mercurial repository formats won't cause the error messages we're seeing here, afaik. If I recall correctly, `sage -upgrade` does seem to complain about "uncommitted changes" even if there are files which aren't even added yet, such as Kevin's sage/rings/finite_field_givaro.pyx (which doesn't exist in a vanilla 4.7 or 4.7.1 sage distribution). Wonder where that came from.

Anyway I wouldn't recommend using `sage -upgrade` if you can avoid it.

-Keshav

----
Join us in #sagemath on irc.freenode.net !

[leif]

On 25 Aug., 13:20, Keshav Kini <keshav.k...@gmail.com> wrote:
> Mercurial versions in Sage haven't changed between 4.7 and 4.7.1. In any
> case, incompatible Mercurial repository formats won't cause the error
> messages we're seeing here, afaik.

> > "There are uncommitted changes in the Sage root repository. Aborting."

I cannot find this message (apparently originating from Sage, not
Mercurial) either, at least not in 4.7.2.alpha2's nor 4.7.1.rc2's root
repo spkg-install; uncommitted changes (if any) should just get
checked in, and if that failed for some reason, there would be a
different error message.

[Deleting the existing repo in $SAGE_ROOT/ prior to upgrading should
also work, though that isn't very nice. Note that file mode changes
won't be shown by 'hg diff' by default; this requires 'hg diff --
git'.]

> If I recall correctly, `sage -upgrade`
> does seem to complain about "uncommitted changes" even if there are files
> which aren't even added yet, such as Kevin's
> sage/rings/finite_field_givaro.pyx (which doesn't exist in a vanilla 4.7 or
> 4.7.1 sage distribution). Wonder where that came from.
>
> Anyway I wouldn't recommend using `sage -upgrade` if you can avoid it.

I wouldn't say that. We've improved upgrading such that it is now
really meant to work, and we regularly test this.(Unfortunately Jeroen
decided to no longer support upgrades *from* devel versions, which
previously also worked.)

I must admit I couldn't follow or finally review the ticket
introducing the root repo w.r.t. upgrading though.

Anyway, if upgrading (at least from final releases to new final
releases) is broken for someone, this should be reported and get
fixed.


-leif

[leif]

P.S.:

What I can definitely say is that Sage 4.7.1.rc2's hg cannot handle
Sage 4.7.2.alpha2's root repo, which *will* break upgrading...

[Jeroen Demeyer]

On 2011-08-25 14:40, leif wrote:
> What I can definitely say is that Sage 4.7.1.rc2's hg cannot handle
> Sage 4.7.2.alpha2's root repo, which *will* break upgrading...

If this is true, then #10594 needs work!

[leif]

According to deps, it /should/^TM work, since we have:

# the spkg-install file for the root repo uses many 'hg' commands, so
# build mercurial first.
$(INST)/$(SAGE_ROOT_REPO): $(BASE) $(INST)/$(MERCURIAL) $(INST)/$
(PATCH)
$(INSTALL) "$(SAGE_SPKG) $(SAGE_ROOT_REPO) 2>&1" "tee -a $
(SAGE_LOGS)/$(SAGE_ROOT_REPO).log"


I haven't checked yet whether the scripts should also depend on
Mercurial for that reason; extcode already does.

The rationale used to be that Mercurial is available in upgrades, and
especially if there are already repos to upgrade, assuming a[ny]
previous version will be able to handle repos shipped in upgraded
packages as well.


-leif

[John H Palmieri]

On Thursday, August 25, 2011 9:12:01 AM UTC-7, leif wrote:

On 25 Aug., 16:51, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> On 2011-08-25 14:40, leif wrote:
>
> > What I can definitely say is that Sage 4.7.1.rc2's hg cannot handle
> > Sage 4.7.2.alpha2's root repo, which *will* break upgrading...
>
> If this is true, then #10594 needs work!

According to deps, it /should/^TM work

I've upgraded 4.7.1 to 4.7.2.alpha2 on two different platforms, with no issues.

--
John
 

[leif]

Found the "culprit" (for the message; this is from 4.7.rc2's sage-
update / #9433):

# check sage root repo. if present, check status; if there are
# any unchecked in changes, abort.
os.chdir(SAGE_ROOT)
err = subprocess.call('./sage -hg verify', shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
root_repo_intact = (err == 0)
if root_repo_intact:
P = subprocess.Popen('./sage -hg status', shell=True,
stdout=subprocess.PIPE)
output = P.communicate()[0]
if len(output) > 0: # uncommitted changes
print ("There are uncommitted changes in the "
+ "Sage root repository. Aborting.")
sys.exit(2)


John, is this still current, i.e., do we still need this (especially
aborting in case there are uncommitted changes)?

Note that len(output)>0 doesn't necessarily mean there are changes
that should be committed; any file unknown to Mercurial will show up
there.


-leif

[Kevin Horton]

Is this a new check in sage 4.7? If so, this almost certainly explains why I had this abort on my OS X machine, as I have certainly snooped about in the sage stuff with the Finder, which would litter it with .DS_Store files. These files would not be known to Mercurial, and could trigger this abort. That wouldn't have been an issue on my Linux box though.

I suggest that the check for uncommitted changes should be revised to allow .DS_Store files that are not known to Mercurial on OS X. Perhaps it should allow any unknown files on any platform.

[John H Palmieri]

On Thursday, August 25, 2011 6:42:30 PM UTC-7, Kevin Horton wrote:

I suggest that the check for uncommitted changes should be revised to allow .DS_Store files that are not known to Mercurial on OS X.  Perhaps it should allow any unknown files on any platform.

Actually, we should ignore the presence of .DS_store files.  See

<http://trac.sagemath.org/sage_trac/ticket/11748>

--
John

[John H Palmieri]

On Thursday, August 25, 2011 6:08:54 PM UTC-7, leif wrote:

Found the "culprit" (for the message; this is from 4.7.rc2's sage-
update / #9433):

    # check sage root repo.  if present, check status; if there are
    # any unchecked in changes, abort.
    os.chdir(SAGE_ROOT)
    err = subprocess.call('./sage -hg verify', shell=True,
                          stdout=subprocess.PIPE,
                          stderr=subprocess.PIPE)
    root_repo_intact = (err == 0)
    if root_repo_intact:
        P = subprocess.Popen('./sage -hg status', shell=True,
                             stdout=subprocess.PIPE)
        output = P.communicate()[0]
        if len(output) > 0: # uncommitted changes
            print ("There are uncommitted changes in the "
                   + "Sage root repository. Aborting.")
            sys.exit(2)


John, is this still current, i.e., do we still need this (especially
aborting in case there are uncommitted changes)?

I don't know about "need", but for example uncommitted changes in at least one of the scripts repo or the main Sage repo (or maybe both) causes upgrading to fail in strange ways.  So the point of the check is to fail gracefully, not fail very badly.
 

Note that len(output)>0 doesn't necessarily mean there are changes
that should be committed; any file unknown to Mercurial will show up
there.

If only "hg status" had an exit status of zero if no changes, something else otherwise. 

I think that the upgrade process is fraught with peril, and could be cleaned up significantly.  Handling uncommitted changes (or committed changes which need to be merged with the new spkg) isn't handled well at all, I think in any repo.

--
John

[Kevin Horton]

It would certainly be useful if sage would tell the user exactly what the "uncommitted changes" were, and if there was a way to force sage to upgrade anyway, if the user was convinced that these changes were not relevant. As it is now, it just aborts, for no known reason, and there is no way to force it to continue.

[leif]

On 26 Aug., 04:35, John H Palmieri <jhpalmier...@gmail.com> wrote:
> On Thursday, August 25, 2011 6:08:54 PM UTC-7, leif wrote:
> > John, is this still current, i.e., do we still need this (especially
> > aborting in case there are uncommitted changes)?
>
> I don't know about "need", but for example uncommitted changes in at least
> one of the scripts repo or the main Sage repo (or maybe both) causes
> upgrading to fail in strange ways.  So the point of the check is to fail
> gracefully, not fail very badly.

Well, uncommitted changes are committed upon root repo
(re)installation anyway. (Same for scripts and the library IIRC;
extcode for sure.)

We should just ignore lines starting with "? " there, and IMHO only
issue a warning if other changes remain; or prompt the user whether to
commit or continue, or exit, unless a to-be-implemented "force" option
was given.

> > Note that len(output)>0 doesn't necessarily mean there are changes
> > that should be committed; any file unknown to Mercurial will show up
> > there.
>
> > If only "hg status" had an exit status of zero if no changes, something
>
> else otherwise.

I couldn't really read that...

> I think that the upgrade process is fraught with peril, and could be cleaned
> up significantly.  Handling uncommitted changes (or committed changes which
> need to be merged with the new spkg) isn't handled well at all, I think in
> any repo.

Hmmm. Suggestions? Should we require *all* repos to be clean, and test
all of them for uncommitted changes *there*, proceeding as suggested
above in case they aren't?


-leif

[leif]

On 26 Aug., 05:14, Kevin Horton <khorto...@rogers.com> wrote:
> It would certainly be useful if sage would tell the user exactly what the "uncommitted changes" were, and if there was a way to force sage to upgrade anyway, if the user was convinced that these changes were not relevant.  As it is now, it just aborts, for no known reason, and there is no way to force it to continue.

That's exactly what I was thinking, too; see above. :-)


-leif

[Keshav Kini]

Here's how I think upgrading should work:

Don't check for cleanness in repos.
Pull from the new SPKGs' repos into the current installation repos.
Do not merge.
`hg update -c tip` in each of the current installation repos; save resulting messages (in a file, say) to display at the end of the upgrade process because users need to see it.
Done.

This way after sage -upgrade you will have a totally clean new version, but any modifications, commits, pushed patches, etc. you had before will be lying around in other branches (other heads). If you want to merge these changes, you should be doing it yourself, and not have it done during some huge upgrade process where everything is flashing by. It is very disconcerting to do `sage -upgrade` and then suddenly 15 minutes later have kdiff3 pop up with some random files in it!

If you had uncommited but tracked changes in the directory when you tried to upgrade, you would see an error message from hg update -c tip at the end of the upgrade process, as described above, which would tell you to go fix it. And most importantly this would not cause errors due to untracked files sitting around, as we have seen above.

-Keshav

[leif]

On 26 Aug., 05:44, Keshav Kini <keshav.k...@gmail.com> wrote:
> Here's how I think upgrading should work:
>
> Don't check for cleanness in repos.
> Pull from the new SPKGs' repos into the current installation repos.

> Do *not* merge.

> `hg update -c tip` in each of the current installation repos; save resulting
> messages (in a file, say) to display at the end of the upgrade process
> because users need to see it.
> Done.

That's [IMHO] ok for true upgrades, but totally inconvenient for
rebuilds, especially those indirectly triggered by dependencies, so
we'd have to introduce some other "flag" (env var setting) for this.

> This way after sage -upgrade you will have a totally clean new version, but
> any modifications, commits, pushed patches, etc. you had before will be
> lying around in other branches (other heads). If you want to merge these
> changes, you should be doing it yourself, and not have it done during some
> huge upgrade process where everything is flashing by. It is very
> disconcerting to do `sage -upgrade` and then suddenly 15 minutes later have
> kdiff3 pop up with some random files in it!
>
> If you had uncommited but tracked changes in the directory when you tried to
> upgrade, you would see an error message from hg update -c tip at the end of
> the upgrade process, as described above, which would tell you to go fix it.

> And most importantly this would not cause errors due to *untracked* files

> sitting around, as we have seen above.

Where? In the repos' root directories, or all in, say, $SAGE_ROOT/
upgrade.log?

Ceterum censeo there should be a way to prevent switching back to the
"main" Sage library branch upon rebuilds (reinstallations), just for
development / testing.


-leif

[John H Palmieri]

On Thursday, August 25, 2011 8:28:09 PM UTC-7, leif wrote:

On 26 Aug., 04:35, John H Palmieri <jhpalm...@gmail.com> wrote:
> On Thursday, August 25, 2011 6:08:54 PM UTC-7, leif wrote:
> > John, is this still current, i.e., do we still need this (especially
> > aborting in case there are uncommitted changes)?
>
> I don't know about "need", but for example uncommitted changes in at least
> one of the scripts repo or the main Sage repo (or maybe both) causes
> upgrading to fail in strange ways.  So the point of the check is to fail
> gracefully, not fail very badly.

Well, uncommitted changes are committed upon root repo
(re)installation anyway. (Same for scripts and the library IIRC;
extcode for sure.)

It tries to commit the changes, but since there are some "hg commit" commands without the "-m" option specified, it tries to pop open an editor.  With emacs as the editor, this screws up the output to the screen, the log for the relevant spkg, and in the end, the update.  This is correctable, of course.

We should just ignore lines starting with "? " there, and IMHO only
issue a warning if other changes remain; or prompt the user whether to
commit or continue, or exit, unless a to-be-implemented "force" option
was given.

Why should we ignore lines starting with "? "?  Perhaps the user wanted to add the file.  I don't see any reason to ignore these files, just like I wouldn't see a reason to automatically add the to the repo.

--
John

[Keshav Kini]

If by rebuilds you mean the forced reinstallation of a package version that's already installed, I don't see what the problem is. The procedure I outlined will do nothing in that case. Nothing will be pulled, so no new heads will be created. `hg update -c` may fail but I don't see how that's inconvenient. If it doesn't fail it will simply update to the head, which is hardly an inconveniece. Or am I missing something?

I guess by "where" you mean where the file containing the message about what cleanup needs to be done manually after an upgrade should go. I just meant that each spkg should append something to, say, $SAGE_ROOT/final_messages , and then sage-upgrade itself should cat and delete that file at the end of the whole process.

[leif]

On 26 Aug., 07:05, John H Palmieri <jhpalmier...@gmail.com> wrote:
> On Thursday, August 25, 2011 8:28:09 PM UTC-7, leif wrote:
> > Well, uncommitted changes are committed upon root repo
> > (re)installation anyway. (Same for scripts and the library IIRC;
> > extcode for sure.)
>
> It tries to commit the changes, but since there are some "hg commit"
> commands without the "-m" option specified, it tries to pop open an editor.  
> With emacs as the editor, this screws up the output to the screen, the log
> for the relevant spkg, and in the end, the update.  This is correctable, of
> course.

Oh, missed that. Of course the build shouldn't be interactive.

> > We should just ignore lines starting with "? " there, and IMHO only
> > issue a warning if other changes remain; or prompt the user whether to
> > commit or continue, or exit, unless a to-be-implemented "force" option
> > was given.
>
> Why should we ignore lines starting with "? "?  Perhaps the user wanted to
> add the file.  I don't see any reason to ignore these files, just like I

> wouldn't see a reason to automatically add them to the repo.

Ignoring them is IMHO ok since they won't have any effect on commits
or pulls from the new repos, and also the converse, i.e., they'll just
be left untouched. (And I guess in 99% of the cases "unknown" files
are simply left-around ones, not intended to get added to a repo.)

As kini suggested, we could log them to some file to be shown at the
end of the build though, or (in case we prompt the user before really
doing the upgrade) list them there in a warning before asking, too,
but my point was that we shouldn't unconditionally abort upgrading if
*only* "unknown" files are shown by 'hg status'.


-leif

[leif]

On 26 Aug., 07:51, Keshav Kini <keshav.k...@gmail.com> wrote:
> If by rebuilds you mean the forced reinstallation of a package version
> that's already installed, I don't see what the problem is.

I meant e.g. installing new spkg foo by

* copying it to spkg/standard/
* applying necessary patches for the new foo spkg to the Sage library
(and committing them)
* running 'env SAGE_UPGRADING=yes make' to (re)build it and also all
dependent packages, which includes the Sage library

(Note that there are situations where reinstalling / rebuilding the
Sage library will fail badly -- eventually quite late -- without
having some necessary patches applied, therefore "inconvenient", also
preventing automatic testing. In addition, packages in turn depending
on the Sage library won't get rebuilt in the first place in that
case.)

> The procedure I
> outlined will do nothing in that case. Nothing will be pulled, so no new
> heads will be created.

In fact there nothing should get pulled from the unmodified Sage
library repo, which contradicts your statement that we'll always end
up with "clean" / well-defined repos, ignoring or discarding all
changes previously made by a user.

If pulling from the "vanilla" repo *did* create a new branch (making
it the current head / tip), this would break the above procedure. (In
a "real" upgrade, the new Sage library -- and *perhaps* the other
repos -- will [currently] almost always have a different tip than the
previous one, such that pulling from that new repo should indeed
create a new branch and make it the new tip. So the problem IMHO
mainly reduces to having to always touch all repos upon new releases,
which we planned to *not* longer do IIRC, cf. discussion on sage-
release a while ago. Left-around untracked files may be another
problem; see below.)

Applying the patches to the Sage library in advance, then spkg-disting
that (and putting it into spkg/standard/) is also inconvenient just
because of the size of the Sage library spkg (>50 MB). I'd have to
create and provide such modified Sage library spkgs for each and every
new devel release, for IMHO no good reason. (And at least the patchbot
will be unable to do this by it/him/herself for a while I guess, even
when it/he/she -- hopefully soon -- supports tickets with spkgs,
although this is perhaps another story.)

> `hg update -c` may fail but I don't see how that's
> inconvenient. If it doesn't fail it will simply update to the head, which is
> hardly an inconveniece. Or am I missing something?

'hg update -c' apparently ignores new files (not yet added, nor
explicitly ignored). It only fails if there are outstanding changes to
already tracked files.

> I guess by "where" you mean where the file containing the message about what
> cleanup needs to be done manually after an upgrade should go. I just meant
> that each spkg should append something to, say, $SAGE_ROOT/final_messages ,
> and then sage-upgrade itself should cat and delete that file at the end of
> the whole process.

Yep. Although I wouldn't delete such logs after printing them at the
end. (However, they *should* end up in install.log anyway.)

> Join us in #sagemath on irc.freenode.net

I prefer irc.* over chat.* since the former conforms to good old
conventions. irc://irc.freenode.net is more explicit, but irc://chat.freenode.net
would also be ok. :-)


-leif

[Keshav Kini]

On Saturday, August 27, 2011 2:22:11 PM UTC+8, leif wrote:

On 26 Aug., 07:51, Keshav Kini <kesha...@gmail.com> wrote:
> If by rebuilds you mean the forced reinstallation of a package version
> that's already installed, I don't see what the problem is.

I meant e.g. installing new spkg foo by

 * copying it to spkg/standard/
 * applying necessary patches for the new foo spkg to the Sage library
(and committing them)
 * running 'env SAGE_UPGRADING=yes make' to (re)build it and also all
dependent packages, which includes the Sage library

(Note that there are situations where reinstalling / rebuilding the
Sage library will fail badly -- eventually quite late -- without
having some necessary patches applied, therefore "inconvenient", also
preventing automatic testing. In addition, packages in turn depending
on the Sage library won't get rebuilt in the first place in that
case.)

I guess I didn't give too much thought to other situations in which sage is "upgraded", other than when you just type `sage -upgrade [url]` on the command line. Shouldn't rebuilds ideally be semantically separated from "upgrades"? Not that I have any idea what that entails in Sage...
 

> The procedure I
> outlined will do nothing in that case. Nothing will be pulled, so no new
> heads will be created.

In fact there nothing should get pulled from the unmodified Sage
library repo, which contradicts your statement that we'll always end
up with "clean" / well-defined repos, ignoring or discarding all
changes previously made by a user.

Er, yeah, I meant you would get a "clean" new version, assuming you were actually upgrading :) I doubt anyone would expect "-upgrade" to revert their current version to a (prior) clean state. I also meant that you would get a clean working directory, not a clean repo, though again only in the case you were actually upgrading.
 

If pulling from the "vanilla" repo *did* create a new branch (making
it the current head / tip), this would break the above procedure. (In
a "real" upgrade, the new Sage library -- and *perhaps* the other
repos -- will [currently] almost always have a different tip than the
previous one, such that pulling from that new repo should indeed
create a new branch and make it the new tip. So the problem IMHO
mainly reduces to having to always touch all repos upon new releases,
which we planned to *not* longer do IIRC, cf. discussion on sage-
release a while ago. Left-around untracked files may be another
problem; see below.)

Any relation to #10231?
 

Applying the patches to the Sage library in advance, then spkg-disting
that (and putting it into spkg/standard/) is also inconvenient just
because of the size of the Sage library spkg (>50 MB). I'd have to
create and provide such modified Sage library spkgs for each and every
new devel release, for IMHO no good reason. (And at least the patchbot
will be unable to do this by it/him/herself for a while I guess, even
when it/he/she -- hopefully soon -- supports tickets with spkgs,
although this is perhaps another story.)


> `hg update -c` may fail but I don't see how that's
> inconvenient. If it doesn't fail it will simply update to the head, which is
> hardly an inconveniece. Or am I missing something?

'hg update -c' apparently ignores new files (not yet added, nor
explicitly ignored). It only fails if there are outstanding changes to
already tracked files.

Yup, which I figured was a desirable behavior, per various people's suggestions to not balk at unknown files lying around.
 

> I guess by "where" you mean where the file containing the message about what
> cleanup needs to be done manually after an upgrade should go. I just meant
> that each spkg should append something to, say, $SAGE_ROOT/final_messages ,
> and then sage-upgrade itself should cat and delete that file at the end of
> the whole process.

Yep. Although I wouldn't delete such logs after printing them at the
end. (However, they *should* end up in install.log anyway.)

Sure, we could instead attach the file to install.log after printing it. The main point would be to make sure we don't print messages from previous upgrades during the current one.
 

> Join us in #sagemath on irc.freenode.net

I prefer irc.* over chat.* since the former conforms to good old
conventions. irc://irc.freenode.net is more explicit, but irc://chat.freenode.net
would also be ok. :-)

Me too, that's why I said irc.freenode.net :P BTW that wasn't really part of my message, just a manual signature I append to my messages when I remember, as an advertisement!
 

-leif

-Keshav

----
Join us in #sagemath on irc.freenode.net !

[leif]

On 27 Aug., 10:05, Keshav Kini <keshav.k...@gmail.com> wrote:
> On Saturday, August 27, 2011 2:22:11 PM UTC+8, leif wrote:
>
> > On 26 Aug., 07:51, Keshav Kini <kesha...@gmail.com> wrote:
> > > If by rebuilds you mean the forced reinstallation of a package version
> > > that's already installed, I don't see what the problem is.
>
> > I meant e.g. installing new spkg foo by
>
> >  * copying it to spkg/standard/
> >  * applying necessary patches for the new foo spkg to the Sage library
> > (and committing them)
> >  * running 'env SAGE_UPGRADING=yes make' to (re)build it and also all
> > dependent packages, which includes the Sage library
>
> > (Note that there are situations where reinstalling / rebuilding the
> > Sage library will fail badly -- eventually quite late -- without
> > having some necessary patches applied, therefore "inconvenient", also
> > preventing automatic testing. In addition, packages in turn depending
> > on the Sage library won't get rebuilt in the first place in that
> > case.)
>
> I guess I didn't give too much thought to other situations in which sage is
> "upgraded", other than when you just type `sage -upgrade [url]` on the
> command line. Shouldn't rebuilds ideally be semantically separated from
> "upgrades"? Not that I have any idea what that entails in Sage...

Well, basically 'sage -upgrade ...' just grabs two separate files and
all updated / newer spkgs from the server and then performs a rebuild
(of the new spkgs and of all those that directly or indirectly depend
on these).

The mentioned procedure to "manually" install new spkgs and rebuild
all dependent ones is an inofficial by-product of the effort to make
upgrading more reliable (without artificial patch-level bumping and
the need to do 'sage -ba-force') though, and [therefore] currently
intentionally undocumented. ;-)

I'm planning to add some 'make' targets for that, e.g. 'rebuild', such
that one doesn't have to set weird environment variables. Then we can
document it as an officially supported procedure.


> Any relation to #10231 <http://trac.sagemath.org/sage_trac/ticket/10231>?

Yep, although I wasn't aware of that ticket (or at least forgot about
it).

The discussion on sage-release is a bit more recent, with a bunch of
pros and cons.

> > 'hg update -c' apparently ignores new files (not yet added, nor
> > explicitly ignored). It only fails if there are outstanding changes to
> > already tracked files.
>
> Yup, which I figured was a desirable behavior, per various people's
> suggestions to not balk at unknown files lying around.

Also IMO; John seemed to not like this idea.


> ----
> Join us in #sagemath on irc.freenode.net <irc://irc.freenode.net/sagemath> !

"Be aware that you may need to wait a while for your answer (perhaps
hours) as our channel is still new and gaining users."

Statistics? :P


-leif

[Keshav Kini]

The IRC channel grew a bit when I made a fuss about it on sage-devel some months ago, and bugged a freenode operator to redirect #sage-devel (which we had lost administrative control over) to #sagemath. Since then, it hasn't changed much. Pretty much the same faces as a couple months ago. There were a few more users during sd31 when I was promoting it in person, but it seems during sd32 nobody was using IRC (even #sagemath-days, which I set up as a coordination point for sage dayses)... is it time for another bout of proselytizing? Feel up to it? :P

-Keshav

----
Join us in #sagemath on irc.freenode.net !

[leif]

Revived...