Is it possible to do a binary upgrade similar to sage -upgrade?

[Carlos Córdoba]

Hi all,

Given the fast development pace of sage, I would like to know if it's possible to do a binary upgrade between releases, similar to what sage -upgrade does now. I think it's a bit cumbersome to download 400-500 gigs and uncompress this big tarball every month - two months, especially for people with old computers, small hard drives and/or who live in developing countries where internet connections are not too good.

If I understand correctly every sage package is released as a spkg tarball, so couldn't it be possible that sage just downloads the packages that have changed between releases and uncompress them in the main sage root?

Sorry if my question has been answered before.

Thanks,

Carlos

[William Stein]

2009/11/23 Carlos Córdoba <ccord...@gmail.com>:

If you have a compiler and know what you are doing you can type

sage -upgrade

and that will download and *build from source* only the spkg's that
have changed. This is far from rock solid, and is not recommended for
non-developers.

There is no option to do a binary-only upgrade, i.e., one that does
not involve compiling code.

-- william

[Harald Schilly]

On Nov 23, 4:51 pm, Carlos Córdoba <ccordob...@gmail.com> wrote:
> I think it's a bit cumbersome to download 400-500 gigs
> and uncompress this big tarball every month - two months, especially for
> people with old computers, small hard drives and/or who live in developing
> countries where internet connections are not too good.

There are several problem, for example, there are different builds for
all kinds of linux distributions and systems. It's hard enough to do
the binaries, even harder to rip this apart into smaller parts for
differential updates. One "easy" possibility would be to untar the
older and the newer binary distribution, do a binary diff across all
files - which has to be some intelligent diff program - and provide
the compressed differentials. I don't know if this would be of any
help or how good it will work in practice.

But: First, it's megabytes, not gigabites :) and secondly, i want to
emphasize that i provide metalinks for all binaries. They enable you
to download only parts day by day by resuming the download, select the
best mirror for you, and check the downloaded data for errors and
corrects them if there are some. You just need some client like aria2
for linux, give it a try!

H

[Jason Grout]

Harald Schilly wrote:

> On Nov 23, 4:51 pm, Carlos C�rdoba <ccordob...@gmail.com> wrote:
>> I think it's a bit cumbersome to download 400-500 gigs
>> and uncompress this big tarball every month - two months, especially for
>> people with old computers, small hard drives and/or who live in developing
>> countries where internet connections are not too good.
>
> There are several problem, for example, there are different builds for
> all kinds of linux distributions and systems. It's hard enough to do
> the binaries, even harder to rip this apart into smaller parts for
> differential updates. One "easy" possibility would be to untar the
> older and the newer binary distribution, do a binary diff across all
> files - which has to be some intelligent diff program - and provide
> the compressed differentials. I don't know if this would be of any
> help or how good it will work in practice.
>

What if we just keep build directories on sage.math? Then people can
use rsync to update their binary installations, which is an "intelligent
binary diff program providing compressed differentials".

So we can just have a directory sagemath.org/SUSE-binary-installation/
that is a direct copy of the build directory on the SUSE virtual
machine. To update, people would just use rsync to update their local
copy with that directory.

Thanks,

Jason

[William Stein]

On Mon, Nov 23, 2009 at 9:24 AM, Jason Grout
<jason...@creativetrax.com> wrote:
> Harald Schilly wrote:

Brainstorming for problems:

(1) The first "warning" is that no other distributions do upgrades
this way. There may be a reason.

(2) There are 25 binaries (at least) for each release. Doing what
you suggest requires storing them uncompressed, which would take about
50GB of disk space and could be some work to maintain (though not that
bad).

(3) Any changes or customizations the user makes anywhere to their
sage install will be *deleted*.

I think (3) is perhaps the biggest issue.

William

[Jason Grout]

William Stein wrote:
> (3) Any changes or customizations the user makes anywhere to their
> sage install will be *deleted*.
>
> I think (3) is perhaps the biggest issue.

I think emphasizing that this is a binary upgrade and *only* works to
overwrite your current sage directory to an exact copy of a fresh binary
build would be sufficient.

Note that doing the rsync (to a copy of your current sage directory) is
no different than downloading the binary and untarring it, but
presumably it is quite a bit faster and requires less bandwidth.

If someone borks their install and can't figure out how to fix it, this
also might be an easy way to "start fresh".

Jason

[Mike Hansen]

On Tue, Nov 24, 2009 at 12:39 AM, Jason Grout
<jason...@creativetrax.com> wrote:
> Note that doing the rsync (to a copy of your current sage directory) is
> no different than downloading the binary and untarring it, but
> presumably it is quite a bit faster and requires less bandwidth.

However, it is much more resource intensive on the server.

--Mike

[William Stein]

On Mon, Nov 23, 2009 at 9:39 AM, Jason Grout
<jason...@creativetrax.com> wrote:
> William Stein wrote:
>>  (3) Any changes or customizations the user makes anywhere to their
>> sage install will be *deleted*.
>>
>> I think (3) is perhaps the biggest issue.
>
>
> I think emphasizing that this is a binary upgrade and *only* works to
> overwrite your current sage directory to an exact copy of a fresh binary
> build would be sufficient.

Note, moreover, that if the users installs *any* optional packages,
these are all deleted too.

> Note that doing the rsync (to a copy of your current sage directory) is
> no different than downloading the binary and untarring it, but
> presumably it is quite a bit faster and requires less bandwidth.

It is no different assuming you didn't make any changes at all. And
of course, Mike's right -- it costs a lot more from a bandwidth
perspective....

> If someone borks their install and can't figure out how to fix it, this
> also might be an easy way to "start fresh".
>
> Jason
>

> --
> To post to this group, send email to sage-s...@googlegroups.com
> To unsubscribe from this group, send email to sage-support...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-support
> URL: http://www.sagemath.org



--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

[Jason Grout]

William Stein wrote:
> On Mon, Nov 23, 2009 at 9:39 AM, Jason Grout
> <jason...@creativetrax.com> wrote:
>> William Stein wrote:
>>> (3) Any changes or customizations the user makes anywhere to their
>>> sage install will be *deleted*.
>>>
>>> I think (3) is perhaps the biggest issue.
>>
>> I think emphasizing that this is a binary upgrade and *only* works to
>> overwrite your current sage directory to an exact copy of a fresh binary
>> build would be sufficient.
>
> Note, moreover, that if the users installs *any* optional packages,
> these are all deleted too.
>
>> Note that doing the rsync (to a copy of your current sage directory) is
>> no different than downloading the binary and untarring it, but
>> presumably it is quite a bit faster and requires less bandwidth.
>
> It is no different assuming you didn't make any changes at all. And
> of course, Mike's right -- it costs a lot more from a bandwidth
> perspective....

Yes, the processing cost on the server is something I hadn't considered.
That would probably make this not an option on a wide scale.

Well, it was an idea, anyway...

Jason

[Harald Schilly]

On Nov 23, 6:24 pm, Jason Grout <jason-s...@creativetrax.com> wrote:
> rsync

I think you mean rdiff-backup and I was thinking about bsdiff/bspatch.
About rsync, If you have problems downloading a big file, you will
even have more troubles doing rsync since it scans all 150,000 files
of sage and transfers all those that are different. My idea would be
to provide one single differential file that changes a binary to
another one.
As I told earlier, for now, with a flaky and weak connection,
metafiles are the best solution imho.

H

[Carlos Córdoba]

An xdelta or something like that, which could be joined with a previous tarball to form the latest release would be great.

Besides, wouldn't it be possible to create binary spkg's that instead of containing the source code would contain the binaries and data which result after compilation? You could do things like that in source-based linux distros like gentoo.

2009/11/23 Harald Schilly <harald....@gmail.com>

H

[Dan Drake]

On Mon, 23 Nov 2009 at 11:24AM -0600, Jason Grout wrote:
> What if we just keep build directories on sage.math? Then people can
> use rsync to update their binary installations, which is an "intelligent
> binary diff program providing compressed differentials".
>
> So we can just have a directory sagemath.org/SUSE-binary-installation/
> that is a direct copy of the build directory on the SUSE virtual
> machine. To update, people would just use rsync to update their local
> copy with that directory.

A compromise between "tarball on a webserver" and "rsync server" is
zsync: http://zsync.moria.org.uk/

"zsync is a file transfer program. It allows you to download a file from
a remote server, where you have a copy of an older version of the file
on your computer already. zsync downloads only the new parts of the
file. It uses the same algorithm as rsync. However, where rsync is
designed for synchronising data from one computer to another within an
organisation, zsync is designed for file distribution, with one file on
a server to be distributed to thousands of downloaders."

It seems like it would work really well, except that it would require
people to keep their old downloads; to efficiently upgrade to, say,
4.2.1, you would need the tarball from downloading 4.2.

It seems like a reasonable mix of regular file downloads and rsync.

Dan

--
--- Dan Drake
----- http://mathsci.kaist.ac.kr/~drake
-------

[William Stein]

So, I think this is all a great idea! To make it happen though,
somebody (not me) has to volunteer to bust their 'arse and make it
happen.

-- William

[Dan Drake]

On Mon, 23 Nov 2009 at 03:55PM -0800, William Stein wrote:
> > A compromise between "tarball on a webserver" and "rsync server" is
> > zsync: http://zsync.moria.org.uk/
> >
> > "zsync is a file transfer program. It allows you to download a file from
> > a remote server, where you have a copy of an older version of the file
> > on your computer already. zsync downloads only the new parts of the
> > file. It uses the same algorithm as rsync. However, where rsync is
> > designed for synchronising data from one computer to another within an
> > organisation, zsync is designed for file distribution, with one file on
> > a server to be distributed to thousands of downloaders."
> >
> > It seems like it would work really well, except that it would require
> > people to keep their old downloads; to efficiently upgrade to, say,
> > 4.2.1, you would need the tarball from downloading 4.2.
> >
> > It seems like a reasonable mix of regular file downloads and rsync.
> >
> > Dan
>
> So, I think this is all a great idea! To make it happen though,
> somebody (not me) has to volunteer to bust their 'arse and make it
> happen.

I just tested it, and using it should be super easy. Visit
http://sagenb.kaist.ac.kr/~drake/ and grab a tarball from there, say
sage-4.2.alpha0.tar. Install zsync, and then do

$ zsync -i sage-4.2.alpha0.tar http://sagenb.kaist.ac.kr/~drake/sage-4.2.tar.zsync

It will use the 4.2.alpha0 tarball as a source, then download the
necessary bits to give you a 4.2 tarball. If I'm reading the output
right, it used used 197 megabytes from the alpha0 tarball (so it
didn't download anything new) and only downloaded 74 megabytes. (Using
10^6 bytes = megabyte), so it only transferred about 27% of tarball
using nothing more than the webserver I am already running.

All you do is run "zsyncmake" on the file you want to efficiently serve
up. There are some options that I'll play with, but this could easily be
scripted. Then we have to educate users, which will likely be the harder
part if we want to use this.

[William Stein]

What do you envision users doing, exactly? Why not just make it so

sage -upgrade

is "educated"?

William

[Carlos Córdoba]

William,

I think what Dan is proposing is just to use an old tarball and an intelligent algorithm to auto-magically recreate the latest one. Then the user would have to uncompress it so he/she could use the new sage. I don't think this could be done inside sage itself.

2009/11/23 William Stein <wst...@gmail.com>

William

[Mike Hansen]

Another option might be using a unionfs overlay to monitor the files
that get installed / changed during the installation of an spkg. I'm
not sure about deleted files, but spkg which for example delete the
old copy before installation could have that functionality factored
out into a spkg-clean (or something of the like) file.
http://www.pierrox.net/trip/ is a package manager that does something
like this.

--Mike

[Carlos Córdoba]

Thanks for the tip Mike. I'll investigate this approach and report progress, if I make some.

Carlos

2009/11/24 Mike Hansen <mha...@gmail.com>

--

[Harald Schilly]

On Nov 24, 3:08 am, Dan Drake <dr...@kaist.edu> wrote:
> zsync

FWIW, I implemented zsync and it isn't good. It's probably the same
underlying issue as with rsync, where too many small connections try
to fetch small files but overall it takes much longer and the whole
traffic might be even worse. zsync issued warnings about block sizes,
i reduced their size and the zsync files got bigger than 20 mb.
Therefore I'll remove zsync from the mirror network.

H