upgrade fails again

[John Cremona]

Am I the only person for whom "sage -upgrade" always fails? It
downloads lots of stuff, builds lots of stuff, and leaves me with an
un-runnable sage which doesn't know whether it is the old version or
the new.

In this case my working 3.3.rc2 was upgraded. Afterwards, the banner
still says 3.3.rc2 and it crashes on startup like this:
ImportError Traceback (most recent call
last)
...
ImportError: /home/john/sage-3.3.rc2/local/lib/python2.5/site-packages/
sage/matrix/matrix_mod2_dense.so: undefined symbol:
mzd_echelonize_m4ri
Error importing ipy_profile_sage - perhaps you should run %upgrade?
WARNING: Loading of ipy_profile_sage failed.

So once again I just delete the entire directory and download a
complete source, ending up taking longer than if had just done that
from the start.

[mabshoff]

On Feb 21, 6:10 am, John Cremona <John.Crem...@gmail.com> wrote:

Hi John,

> Am I the only person for whom "sage -upgrade" always fails?  It
> downloads lots of stuff, builds lots of stuff, and leaves me with an
> un-runnable sage which doesn't know whether it is the old version or
> the new.

-upgrade is designed to upgrade one clean tree to another clean tree,
i.e. if you do development in it or use queues for example you will
run into trouble with some probability.

> In this case my working 3.3.rc2 was upgraded.  Afterwards, the banner
> still says 3.3.rc2 and it crashes on startup like this:
> ImportError                               Traceback (most recent call
> last)
> ...
> ImportError: /home/john/sage-3.3.rc2/local/lib/python2.5/site-packages/
> sage/matrix/matrix_mod2_dense.so: undefined symbol:
> mzd_echelonize_m4ri
> Error importing ipy_profile_sage - perhaps you should run %upgrade?
> WARNING: Loading of ipy_profile_sage failed.

Ok, have you checked ig the Sage library has more than one head? This
is usually the cause for the above and indicates half-way done merges.

> So once again I just delete the entire directory and download a
> complete source, ending up taking longer than if had just done that
> from the start.

Well, *if* it is "just" the Sage library that is somehow botched run

rm -rf devel/sage
./sage -f spkt/standard/sage-x.y.z.spkg

and it should all get back to normal quicker than a complete build.

I strongly suspect that you upgrade trees where you develop in, so I
would recommend that you use a clone for that one and switch back to
the main branch before upgrading.

Cheers,

Michael

[mabshoff]

oops:

rm -rf devel/sage*

Note that this will nuke all your changes, but that should be fairly
obvious :)

>  ./sage -f spkt/standard/sage-x.y.z.spkg
>
> and it should all get back to normal quicker than a complete build.
>
> I strongly suspect that you upgrade trees where you develop in, so I
> would recommend that you use a clone for that one and switch back to
> the main branch before upgrading.
>
> Cheers,
>
> Michael

Cheers,

Michael

[Simon King]

On 21 Feb., 15:27, mabshoff <mabsh...@googlemail.com> wrote:
> >  rm -rf devel/sage
>
> oops:
>
>   rm -rf devel/sage*
>
> Note that this will nuke all your changes, but that should be fairly
> obvious :)
>
> >  ./sage -f spkt/standard/sage-x.y.z.spkg

And I guess in the previous line it is spkg/standard/sage-x.y.z.spkg,
not spkt/...

Cheers,
Simon

[mabshoff]

Yep, I saw that too late after I send the other correction and thought
it was so obvious that I did not need to send another email to correct
this. Well, it too two now ;)

> Cheers,
>  Simon

Let me drive home this point again: Aside from a very few cases where
a dependency problem cause build failures (i.e some spkg with
dependencies had unbumped dependencies or I did something else stupid)
I had never fail an upgrade for me from a clean tree to a clean tree.
If you have missing symbols like in this case that John describes the
root cause is nearly always a screwed up merge that can most often be
corrected my merging multiple heads and doing a Sage library build via
"sage -b".

To avoid failed merges don't develop in the tree you upgrade in, use
either a completely separate build tree to develop for example by
building your vanilla tree, then -bdist and use that tree in another
place to do work or use clones of the Sage library to do development.
Once you want to upgrade, switch back to main, upgrade, clone again
and import your changes from the old clone for development into the
newly cloned upgraded vanilla tree. Anything else will likely cause
pain and suffering unless you have excellent hg Merge-Fu.

Cheers,

Michael

[Robert Bradshaw]

On Feb 21, 2009, at 6:23 AM, mabshoff wrote:

> On Feb 21, 6:10 am, John Cremona <John.Crem...@gmail.com> wrote:
>
> Hi John,
>
>> Am I the only person for whom "sage -upgrade" always fails? It
>> downloads lots of stuff, builds lots of stuff, and leaves me with an
>> un-runnable sage which doesn't know whether it is the old version or
>> the new.
>
> -upgrade is designed to upgrade one clean tree to another clean tree,
> i.e. if you do development in it or use queues for example you will
> run into trouble with some probability.

Some relevant tickets are at

http://trac.sagemath.org/sage_trac/ticket/4759
http://trac.sagemath.org/sage_trac/ticket/4833

- Robert

[John Cremona]

Thanks for the diagnosis and suggestions. Evidently I did the upgrade
from a clone, one I made yesterday to review one of those patches....

Suggestion: since the upgrade already gives a prompt and will not
continue unless you reply "yes", why not add to that prompt a line
saying don't go any further unless you are not in a clone. Even
better, detect whether or not you are in a clone and quite if you are
(if that is possible).

I'm now doing an upgrade from my rc0 build, not from any clone, and
we'll see what happens!

John

[John Cremona]

On Feb 21, 3:45 pm, John Cremona <John.Crem...@gmail.com> wrote:
> Thanks for the diagnosis and suggestions.  Evidently I did the upgrade
> from a clone, one I made yesterday to review one of those patches....
>
> Suggestion:  since the upgrade already gives a prompt and will not
> continue unless you reply "yes",  why not add to that prompt a line
> saying don't go any further unless you are not in a clone.  Even
> better, detect whether or not you are in a clone and quite if you are
> (if that is possible).
>
> I'm now doing an upgrade from my rc0 build, not from any clone, and
> we'll see what happens!
>

The upgrade failed in exactly the same way as above. To recap: I had
a working successful build of sage-3.3.rc0 and was in the main branch
(so starting sage did not give me a message about bing in a different
mercurial branch). I typed "sage -upgrade" followed by "y" at the
prpmpt, and went to work on something else. When I came back it had
finished, so I typed "sage" and got the error already reported: the
banner still says rc0 and there's an error on startup, saying

ImportError: libpynac-0.1.so.2: cannot open shared object file: No
such file or directory

Error importing ipy_profile_sage - perhaps you should run %upgrade?
WARNING: Loading of ipy_profile_sage failed.

I particularly like the suggestion that my problem is to upgrade
(which does nothing, of course).

I am going to need a lot of persuading before trying an upgrade again,
which is a pity since I'll only have time to download and build sage
while at home for a while, and my internet connection is not so fast
in rural Warwickshire (especially when my daughter is watching TV on
her laptop!) so downloading the whole distribution takes a couple of
hours.

John

[mabshoff]

On Feb 21, 8:57 am, John Cremona <John.Crem...@gmail.com> wrote:
> On Feb 21, 3:45 pm, John Cremona <John.Crem...@gmail.com> wrote:

Hi John,

> > I'm now doing an upgrade from my rc0 build, not from any clone, and
> > we'll see what happens!
>
> The upgrade failed in exactly the same way as above.  To recap:  I had
> a working successful build of sage-3.3.rc0 and was in the main branch
> (so starting sage did not give me a message about bing in a different
> mercurial branch).  I typed "sage -upgrade" followed by "y" at the
> prpmpt, and went to work on something else.  When I came back it had
> finished, so I typed "sage" and got the error already reported:  the
> banner still says rc0 and there's an error on startup, saying
>
>  ImportError: libpynac-0.1.so.2: cannot open shared object file: No
> such file or directory
> Error importing ipy_profile_sage - perhaps you should run %upgrade?
> WARNING: Loading of ipy_profile_sage failed.
>
> I particularly like the suggestion that my problem is to upgrade
> (which does nothing, of course).

Ok, please send us a link to the compressed install.log from the build
as well as the upgrade (it should be all in one file) and I will take
a look.

Right now I would suspect that either the spkgs you download are
corrupted or that another sage install is interfering via some bits of
the Sage environment, so please also send the output from 'env' on
that box and even better from the shell of the failed upgrade if you
still have it.

Can you make 100% sure that "hg heads" in the current Sage library
branch that you use shows only one head and that hg status does not
show any oddities?

> I am going to need a lot of persuading before trying an upgrade again,
> which is a pity since I'll only have time to download and build sage
> while at home for a while, and my internet connection is not so fast
> in rural Warwickshire (especially when my daughter is watching TV on
> her laptop!) so downloading the whole distribution takes a couple of
> hours.

Yeah, the main US site has gotten quite slow, even though that might
not be the cause for your slow rural connection :)

> John

Cheers,

Michael

[John Cremona]

Thanks, Michael. Obviously do not spend any (more) time on this
before 3.3 is out. Or maybe 3.4. or 3.4.1. or...

see http://www.warwick.ac.uk/staff/J.E.Cremona/upgrade.tgz

for the install log and env output.

I could not do either "hg heads" or "hg status" in sage-3.3.rc0/devel/
sage :

john@ubuntu%pwd
/home/john/sage-3.3.rc0/devel/sage
john@ubuntu%hg heads
abort: requirement 'fncache' not supported!
john@ubuntu%hg status
abort: requirement 'fncache' not supported!

Googling suggests that there is a mismatch between the versions of hg
in use. Mine is 0.9.5 while Sage has 1.0.1 which looks as if it might
be a cause of some problems. There is no more recent version in the
ubuntu repository I use. Anyway, using "sage -hg" instead of hg gives
the same errors. (Normally I only ever use hg from the Sage command-
line using hg_sage.apply() and the like.)

John

[mabshoff]

On Feb 21, 9:58 am, John Cremona <John.Crem...@gmail.com> wrote:

Hi John,

> Thanks, Michael.  Obviously do not spend any (more) time on this
> before 3.3 is out.

The Sage 3.3 sources are already in the usual place with the other
3.3.alpha and rc releases. They should be on the main website within a
few hours.

>  Or maybe 3.4. or 3.4.1. or...
>
> seehttp://www.warwick.ac.uk/staff/J.E.Cremona/upgrade.tgz

Ok, I will take a look.

> for the install log and env output.
>
> I could not do either "hg heads" or "hg status" in sage-3.3.rc0/devel/
> sage :
>
> john@ubuntu%pwd
> /home/john/sage-3.3.rc0/devel/sage
> john@ubuntu%hg heads
> abort: requirement 'fncache' not supported!
> john@ubuntu%hg status
> abort: requirement 'fncache' not supported!

./sage -sh should fix that, but as you indicate below "sage -hg" is
broken for you. This could be caused by either a broken env or some
other problem with the upgraded Sage - at this point it could be
either (or both? :).

> Googling suggests that there is a mismatch between the versions of hg
> in use.  Mine is 0.9.5 while Sage has 1.0.1

3.3.rc0 ships with hg 1.1.2:

mabshoff@sage:/scratch/mabshoff/sage-3.3.rc0$ ./sage -sh

Starting subshell with Sage environment variables set.
Be sure to exit when you are done and do not do anything
with other copies of Sage!
mabshoff@sage:/scratch/mabshoff/sage-3.3.rc0$ hg -v
Mercurial Distributed SCM (version 1.1.2)

If the hg with your Sage is 1.0.1 there is even something broken with
your Sage 3.3.rc0 base build, so assuming this is also the problem
with the other failed upgrade it is no surprise that things blow up
for you :(

> which looks as if it might
> be a cause of some problems.  There is no more recent version in the
> ubuntu repository I use.  Anyway, using "sage -hg" instead of hg gives
> the same errors.  (Normally I only ever use hg from the Sage command-
> line using hg_sage.apply() and the like.)

Ok.

> John

If you have the patience: Create a completely new user account on that
machine and build some 3.3.alpha or rc release in that. Then try
upgrading to the final 3.3. If that works it is something in your
environment.

Cheers,

Michael

[William Stein]

On Sat, Feb 21, 2009 at 8:57 AM, John Cremona <John.C...@gmail.com> wrote:
>
>
>
> On Feb 21, 3:45 pm, John Cremona <John.Crem...@gmail.com> wrote:
>> Thanks for the diagnosis and suggestions. Evidently I did the upgrade
>> from a clone, one I made yesterday to review one of those patches....
>>
>> Suggestion: since the upgrade already gives a prompt and will not
>> continue unless you reply "yes", why not add to that prompt a line
>> saying don't go any further unless you are not in a clone. Even
>> better, detect whether or not you are in a clone and quite if you are
>> (if that is possible).
>>
>> I'm now doing an upgrade from my rc0 build, not from any clone, and
>> we'll see what happens!
>>
>
> The upgrade failed in exactly the same way as above. To recap: I had
> a working successful build of sage-3.3.rc0 and was in the main branch
> (so starting sage did not give me a message about bing in a different
> mercurial branch). I typed "sage -upgrade" followed by "y" at the
> prpmpt, and went to work on something else. When I came back it had
> finished, so I typed "sage" and got the error already reported: the
> banner still says rc0 and there's an error on startup, saying
>
> ImportError: libpynac-0.1.so.2: cannot open shared object file: No
> such file or directory
> Error importing ipy_profile_sage - perhaps you should run %upgrade?
> WARNING: Loading of ipy_profile_sage failed.
>
> I particularly like the suggestion that my problem is to upgrade
> (which does nothing, of course).

That is an error message from Ipython suggesting to upgrade the
Ipython rc files. It's extremely annoying and has nothing to do with
the real problem at hand. It's just an instance of Sage not being
sufficiently polished, and surely something that would be very
confusing to new users. I think we should patch Ipython so it doesn't
do that, or at least gives a meaningful error message.

-- William

[mabshoff]

On Feb 21, 10:41 am, William Stein <wst...@gmail.com> wrote:
> On Sat, Feb 21, 2009 at 8:57 AM, John Cremona <John.Crem...@gmail.com> wrote

Hi,

> > I particularly like the suggestion that my problem is to upgrade
> > (which does nothing, of course).
>
> That is an error message from Ipython suggesting to upgrade the
> Ipython rc files.

Well, if that were it deleting $HOME/.ipython* should fix the problem,
but I would be surprised if that is the case and the whole story. The
missing symbols indicate some kind of problem, but at least rc0->rc3
did not upgrade pynac, so the missing symbol thing is indeed rather
odd.

>  It's extremely annoying and has nothing to do with
> the real problem at hand.  It's just an instance of Sage not being
> sufficiently polished, and surely something that would be very
> confusing to new users.  I think we should patch Ipython so it doesn't
> do that, or at least gives a meaningful error message.

+1

>  -- William

Cheers,

Michael

[mabshoff]

On Feb 21, 8:57 am, John Cremona <John.Crem...@gmail.com> wrote:
> On Feb 21, 3:45 pm, John Cremona <John.Crem...@gmail.com> wrote:

Hi John,

<SNIP>

> The upgrade failed in exactly the same way as above.  To recap:  I had
> a working successful build of sage-3.3.rc0 and was in the main branch
> (so starting sage did not give me a message about bing in a different
> mercurial branch).  I typed "sage -upgrade" followed by "y" at the
> prpmpt, and went to work on something else.  When I came back it had
> finished, so I typed "sage" and got the error already reported:  the
> banner still says rc0 and there's an error on startup, saying
>
>  ImportError: libpynac-0.1.so.2: cannot open shared object file: No
> such file or directory

William run into similar issues and tracked this down to a stale left
over pynac library. The fix is to delete them in spkg-install. This is
now a 3.4 blocker at

#5333 (pynac.spkg: Delete old pynac libray during spkg-install

Cheers,

Michael

[William Stein]

I just want to point out that I didn't actually hit that problem and
that wasn't the fix for the problem I saw, though it is a very good
idea :-).

My problem was a problem merging two heads in the main sage branch,
since I had made a lot of changes to it. The fix was to delete
sage-main and install it again from scratch.

William

[mabshoff]

On Feb 21, 4:09 pm, William Stein <wst...@gmail.com> wrote:
> On Sat, Feb 21, 2009 at 2:56 PM, mabshoff <mabsh...@googlemail.com> wrote:

<SNIP>

> > #5333 (pynac.spkg: Delete old pynac libray during spkg-install
>
> I just want to point out that I didn't actually hit that problem and
> that wasn't the fix for the problem I saw, though it is a very good
> idea :-).

Well, then I somewhat misunderstood you in IRC since I thought you had
hit this the problem on several boxen. Oh well, cleaning up the spkg
won't hurt and while I am at it I will also post a patch to make the
extensions linking against pynac depend on the pynac header so that on
upgrade they will be automatically rebuild.

> My problem was a problem merging two heads in the main sage branch,
> since I had made a lot of changes to it.  The fix was to delete
> sage-main and install it again from scratch.

:)

> William

Cheers,

Michael