Use mirrors

[Volker Braun]

As the centrally hosted sage download link at UW is going away, we need to use the mirrors for third-party packages soon. Bonus: faster downloads. 

Please review http://trac.sagemath.org/ticket/15642

This will require a system python install if you build from the git tree.

[leif]

Volker Braun wrote:
> This will require a system python install if you build from the git tree.

Will ATLAS then no longer depend on Sage's Python? :P

[Volker Braun]

It won't be changed on said ticket but in principle yes (needs some testing with python3 though)

[leif]

Volker Braun wrote:
> As the centrally hosted sage download link at UW is going away, we need
> to use the mirrors for third-party packages soon. Bonus: faster downloads.

If "soon" is soon, we should think about releasing a 6.6.1, as otherwise
installing packages will no longer work with the latest stable version.
(Setting SAGE_SERVER to specify a mirror was borked by hardcoding a
'packages/' folder which doesn't exist on any of the mirrors; they only
have 'spkg/'. #15642 now also fixes this, but is targeted at Sage 6.7.)


-leif

[William Stein]

On Sat, May 9, 2015 at 9:04 AM, leif <not.r...@online.de> wrote:
> Volker Braun wrote:
>> As the centrally hosted sage download link at UW is going away, we need
>> to use the mirrors for third-party packages soon. Bonus: faster downloads.
>
> If "soon" is soon, we should think about releasing a 6.6.1, as otherwise

"soon" is not that soon.

-- William

> installing packages will no longer work with the latest stable version.
> (Setting SAGE_SERVER to specify a mirror was borked by hardcoding a
> 'packages/' folder which doesn't exist on any of the mirrors; they only
> have 'spkg/'. #15642 now also fixes this, but is targeted at Sage 6.7.)
>
>
> -leif
>
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--
William (http://wstein.org)

[Dima Pasechnik]

How about keeping these downloads on github?
Perhaps adding its storage as an extra mirror - the one that would be
quite fast for many people, and not as much at mercy of mirror maintainers.if

In fact github can be a complete mirror if we split downloads larger than 1GB using rar or
something like this.

Dima

[leif]

Dima Pasechnik wrote:
> How about keeping these downloads on github?

Perhaps in addition.

> Perhaps adding its storage as an extra mirror - the one that would be
> quite fast for many people, and not as much at mercy of mirror
> maintainers.

Yeah, rather at the mercy of a single commercial company.

(And probably sooner or later of ISPs, in not violating network
neutrality, who knows... ;-) )


-leif

[Dima Pasechnik]

On Saturday, 9 May 2015 22:28:17 UTC+1, leif wrote:

Dima Pasechnik wrote:
> How about keeping these downloads on github?

Perhaps in addition.

> Perhaps adding its storage as an extra mirror - the one that would be
> quite fast for many people, and not as much at mercy of mirror
> maintainers.

Yeah, rather at the mercy of a single commercial company.

presently Sage releases are being done on Google Computiing Engine server
paid by Sagemath Inc., with borrowed money. It cost too much to have a Sage mirror
on this server, AFAIK, as bandwidth is not cheap. My suggestion looks a better kind
of setup in this sense. Note that there is not a single Sage mirror in USA at the moment - github would
fill this niche nicely.

I don't understand why a resource like github should be ignored in favour of ad hoc
solutions.

I was able to get a Sage mirror running in Singapore, but it's due to a goodwill of
one very busy sysadmin, and it seems to be over by now.
I tried to get a Sage mirror running in Oxford, but to no avail.
As we start getting funds from https://github.com/sagemath/grant-europe
(we're now at the stage of finishing grant formalities, so it should come, unless something
very unexpected happens) I can try again, arguing that it's important for the project.

[Volker Braun]

On Sunday, May 10, 2015 at 1:01:59 AM UTC+2, Dima Pasechnik wrote:

I don't understand why a resource like github should be ignored in favour of ad hoc
solutions. 

Its not an ad hoc solution, we are hosting an rsync server which is exactly how mirrors shoud work. Tell me how to have github rsync from our master and we'll be happy to add it.

[Dima Pasechnik]

On Sunday, 10 May 2015 00:14:13 UTC+1, Volker Braun wrote:

On Sunday, May 10, 2015 at 1:01:59 AM UTC+2, Dima Pasechnik wrote:

I don't understand why a resource like github should be ignored in favour of ad hoc
solutions. 

Its not an ad hoc solution, we are hosting an rsync server which is exactly how mirrors shoud work.

 I meant that finding and maintaining mirrors is an ad hoc part, and a painful one.
 

Tell me how to have github rsync from our master and we'll be happy to add it.

well, a mirror (perhaps one set up just for this purpose) should be able to push  files to github.

[William Stein]

On Sat, May 9, 2015 at 4:01 PM, Dima Pasechnik <dim...@gmail.com> wrote:
>
>
> On Saturday, 9 May 2015 22:28:17 UTC+1, leif wrote:
>>
>> Dima Pasechnik wrote:
>> > How about keeping these downloads on github?
>>
>> Perhaps in addition.
>>
>> > Perhaps adding its storage as an extra mirror - the one that would be
>> > quite fast for many people, and not as much at mercy of mirror
>> > maintainers.
>>
>> Yeah, rather at the mercy of a single commercial company.
>
>
> presently Sage releases are being done on Google Computiing Engine server
> paid by Sagemath Inc., with borrowed money. It cost too much to have a Sage

Minor change: the plan if nothing else works would be to use the
mentor part of the GSoC money (which will be ~$2K) to support this
hosting.

> mirror
> on this server, AFAIK, as bandwidth is not cheap. My suggestion looks a
> better kind
> of setup in this sense. Note that there is not a single Sage mirror in USA
> at the moment - github would
> fill this niche nicely.
>
> I don't understand why a resource like github should be ignored in favour of
> ad hoc solutions.

+1 to github hosting if it is possible.

>
> I was able to get a Sage mirror running in Singapore, but it's due to a
> goodwill of
> one very busy sysadmin, and it seems to be over by now.
> I tried to get a Sage mirror running in Oxford, but to no avail.
> As we start getting funds from https://github.com/sagemath/grant-europe
> (we're now at the stage of finishing grant formalities, so it should come,
> unless something
> very unexpected happens) I can try again, arguing that it's important for
> the project.
>
>
>
>
>>
>> (And probably sooner or later of ISPs, in not violating network
>> neutrality, who knows... ;-) )
>>
>>
>> -leif
>>
>>

[kcrisman]

I'm surprised that there wouldn't be any US institution willing to host a mirror.

[Samuel Lelievre]

kcrisman:

I'm surprised that there wouldn't be any US institution willing to host a mirror.

There is a mirror at MIT, both http and rsync:

    http://mirrors.mit.edu/sage/
    rsync://mirrors.mit.edu/sage

but is that the question or is the question about
where to host the master mirror?

Samuel

[Dima Pasechnik]

it seems that links like http://sagemath.org/download-source.html

don't show the full list of mirrors (e.g. the MIT one is up-to date).

 

Samuel

[leif]

Samuel Lelievre wrote:
> kcrisman:
>
> I'm surprised that there wouldn't be any US institution willing to
> host a mirror.
>
> There is a mirror at MIT, both http and rsync:
>
> http://mirrors.mit.edu/sage/
> rsync://mirrors.mit.edu/sage

... which raises the question whether there's also a publicly accessible
*static* list of Sage mirrors (probably more appropriate for casual use,
since we now benchmark the mirrors in sage-download-file ourselves).

http://www.sagemath.org/mirrors.html for example lists nine mirrors for
North America, of which only four appear to currently be available; at
least sage.scipy.org seems obsolete (it isn't set up at all). The
mirror at MIT mentioned above is not among these nine; Harvard is, but
appears to be down.


-leif

[Volker Braun]

Define "static list of mirrors". They are outside of our control.

On Sunday, May 10, 2015 at 4:10:28 PM UTC+2, leif wrote:

since we now benchmark the mirrors in sage-download-file ourselves).

We don't (until #15642 is reviewed)

[leif]

Volker Braun wrote:
> Define "static list of mirrors".

const char *default_mirror_list[48]=
{
"http://sagemath.polytechnic.edu.na/",
"ftp://ftp.sun.ac.za/pub/mirrors/www.sagemath.org/",
"http://sagemath.mirror.ac.za/",
"http://sagemath.mirror.tn/",
"http://ftp.leg.uct.ac.za/pub/packages/sage/",
"http://mirrors-usa.go-parts.com/sage/",
"http://modular.fas.harvard.edu/sage/",
"http://mirrors.hub.co/sage/",
"http://modular.math.jmu.edu/",
"http://sage.scipy.org/sage/",
"http://www.cecm.sfu.ca/sage/",
"http://sage.technocozy.com/",
"http://mirror.clibre.uqam.ca/sage/",
"http://mirrors.xmission.com/sage/",
"http://sagemath.c3sl.ufpr.br/",
"http://linorg.usp.br/sage/",
"http://sage.asis.io/",
"http://mirrors.hustunique.com/sagemath/",
"http://mirror.hust.edu.cn/sagemath/",
"http://ftp.iitm.ac.in/sage/",
"http://mirror.unej.ac.id/mirrors/sage/",
"http://ftp.kaist.ac.kr/sage/",
"http://jambu.spms.ntu.edu.sg/sage/",
"http://ftp.riken.jp/sagemath/",
"http://mirrors.tuna.tsinghua.edu.cn/sagemath/",
"http://mirrors.ustc.edu.cn/sagemath/",
"http://ftp.tsukuba.wide.ad.jp/software/sage/",
"http://ftp.yz.yamagata-u.ac.jp/pub/math/sage/",
"http://mirror.yandex.ru/mirrors/sage.math.washington.edu/",
"http://sage.yasar.edu.tr/",
"http://mirror.yongbok.net/sage/",
"http://mirror.aarnet.edu.au/pub/sage/",
"http://echidna.maths.usyd.edu.au/sage/",
"ftp://ftp.fu-berlin.de/unix/misc/sage/",
"http://sage.mirror.garr.it/mirrors/sage/",
"http://mirrors-uk.go-parts.com/sage/",
"http://mirrors-ru.go-parts.com/sage/",
"http://www-eu-1.sagedev.org/mirror/",
"http://www.mirrorservice.org/sites/www.sagemath.org/",
"http://sunsite.rediris.es/mirror/sagemath/",
"http://mirror.switch.ch/mirror/sagemath/",
"http://sage.Igor.onlineDirect.bg/",
"http://servingzone.com/mirrors/sage/",
"http://ftp.sh.cvut.cz/MIRRORS/sagemath/",
"http://mirrors.fe.up.pt/pub/sage/",
"http://www-ftp.lip6.fr/pub/math/sagemath/",
"http://ftp.ntua.gr/pub/sagemath/",
NULL
};

More seriously:

The ones currently *known* to us (perhaps excluding those that appear to
have definitely gone/have never(?) been set up like sage.scipy.org, but
including the ones that are currently not available).

> On Sunday, May 10, 2015 at 4:10:28 PM UTC+2, leif wrote:
>
> since we now benchmark the mirrors in sage-download-file ourselves).
>
>
> We don't (until #15642 is reviewed)

Until it is merged and released.

But anyone can pull the branch already; it appears to be functional (atm).

[Please pull, build, test, review and report back!]TM


-leif

[Volker Braun]

bump

[kcrisman]

If it's not on the download page, in my opinion it doesn't exist for "most" (Leif would surely say ^TM) Sage users. 

[leif]

kcrisman wrote:
> > I'm surprised that there wouldn't be any US institution
> willing to
> > host a mirror.
> >
> > There is a mirror at MIT, both http and rsync:
> >
> > http://mirrors.mit.edu/sage/

> > rsync://mirrors.mit.edu/sage <http://mirrors.mit.edu/sage>

>
> ... which raises the question whether there's also a publicly
> accessible
> *static* list of Sage mirrors (probably more appropriate for casual
> use,
> since we now benchmark the mirrors in sage-download-file ourselves).
>
> http://www.sagemath.org/mirrors.html
> <http://www.sagemath.org/mirrors.html> for example lists nine
> mirrors for
> North America, of which only four appear to currently be available; at

> least sage.scipy.org <http://sage.scipy.org> seems obsolete (it

> isn't set up at all). The
> mirror at MIT mentioned above is not among these nine; Harvard is, but
> appears to be down.
>
>
> If it's not on the download page, in my opinion it doesn't exist for
> "most" (Leif would surely say ^TM) Sage users.

MIT's for the elite only (hence not for "most" Sage users), so there's
nothing wrong with not listing it on the download page.


SCNR,

-leif