urpmi & metalinks in mandriva
Per Øyvind Karlsen
I'd just like to tell people that I've finally added metalink generation to urpmi. (re: https://qa.mandriva.com/show_bug.cgi?id=29161)
Since an alternate approach was chosen by Mandriva, GeoMcFly was discontinued (but luckily most ideas has been picked up by Peter Poeml for OpenSuSE download redirector, so efforts weren't all wasted:), the new approach will generate a local metalink in a similar way, but rather on client side with client's latitude and longitude coordinates determined by it's timezone, and server coordinates now stored in the mirror list.
It's just been commited to svn now and will need some final touch, but it should show up in cooker some time soon. :)
--
Regards,
Per Øyvind Karlsen
Anthony Bryan
Per, thanks for doing that & keeping us updated!
speaking of MirrorBrain/openSUSE download redirector, there's a new
PDF/video presentation at http://www.mirrorbrain.org/presentations
--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads
Sebastien WILLEMIJNS
> speaking of MirrorBrain/openSUSE download redirector, there's a new
> PDF/video presentation at http://www.mirrorbrain.org/presentations
argh ! mirrorbrain does not know metalink which even for a single file
permit file hash and more and more....
mhaahahahahahahahaahahahahhahahahahaaha
Peter Poeml
Please look closer:
http://download.opensuse.org/distribution/11.0/iso/dvd/openSUSE-11.0-DVD-i386.iso.metalink
I'm a bit puzzled what you mean.
The MirrorBrain supports hashes in metalinks. Sorry if I didn't explain
this very well.
Obviously, such a list of hashes cannot be generated in real-time,
because you can't read and hash a CD or DVD image in reasonable time,
while a client waits for the reply on its metalink request. Thus, the
hashes that the MirrorBrain injects into the metalinks can be prepared
beforehand, and there is a tool in the mirrorbrain svn which does this
and puts them on disk; and from there, Apache includes them later.
Since the hashes are optional, I don't generate them for each and every
small file on download.opensuse.org, especially since all RPM files are
digitally signed anyway; hashes that ensure transfer integrity are
contained within with RPM metadata already. But all metalinks *can* have
hashes, for example those on go-oo.mirrorbrain.org should be complete.
Peter
--
Contact: ad...@opensuse.org (a.k.a. ftpa...@suse.com)
#opensuse-mirrors on freenode.net
Info: http://en.opensuse.org/Mirror_Infrastructure
SUSE LINUX Products GmbH
Research & Development
Sebastien WILLEMIJNS
On Sat, 12 Jul 2008 14:45:26 +0200, "Peter Poeml" <po...@suse.de> said:
> I'm a bit puzzled what you mean.
>
> The MirrorBrain supports hashes in metalinks. Sorry if I didn't explain
> this very well.
you haven't understood i joke about metalink... in fact, i joke about
all
(big) links you purpose on your site aren't "metalinked" ;) that's all
;)
Peter Poeml
I must admit I still needed to think twice. But I think I now get it --
you refer to the fact that the large files (videos) linked on the
mirrorbrain project page are "simple" links, instead of metalinks?
Yes, entirely true... problem is, I don't think there are mirrors that
mirror those files anyway ;)
Sebastien WILLEMIJNS
On Sat, 12 Jul 2008 16:19:03 +0200, "Peter Poeml" <po...@suse.de> said:
> On Sat, Jul 12, 2008 at 03:03:42PM +0200, Sebastien WILLEMIJNS wrote:
> I must admit I still needed to think twice. But I think I now get it --
> you refer to the fact that the large files (videos) linked on the
> mirrorbrain project page are "simple" links, instead of metalinks?
> Yes, entirely true... problem is, I don't think there are mirrors that
> mirror those files anyway ;)
in fact my joke was to show this webpage does not contain the video URL
as a metalink URL too... so never mind
it is not important
Peter Poeml
On Thu, Jul 10, 2008 at 01:08:38AM +0200, Per Øyvind Karlsen wrote:
> Ahoi!
>
>
> I'd just like to tell people that I've finally added metalink generation to
> urpmi. (re: https://qa.mandriva.com/show_bug.cgi?id=29161)
Just so I understand this correctly: the server sends a "generic" mirror
list, which contains geographic coordinates, and the client locally
generates a metalink according to its own position?
That's pretty slick!
This would be just what I pictured for the openSUSE download client at
http://en.opensuse.org/Libzypp/Failover, just that I put that together
before I stumbled over metalinks, and I didn't know that they already
exist as a way to solve these issues :)
The route that we are heading with openSUSE is a bit different now,
however, we are rather going to serve real metalinks (specific to the
client request), and the client should be able to just use it.
However I think it would still be cool to transform the metalink locally
by e.g. injecting a local mirror (could be on the local subnet), which
is tried first. If the metalink is generated locally anyway, this is an
option that naturally offers itself, isn't it? Or how do you solve that
problem with urpmi?
Per Øyvind Karlsen
On Saturday 19 July 2008 01:10:18 Peter Poeml wrote:
> Ahoi Per!
>
> On Thu, Jul 10, 2008 at 01:08:38AM +0200, Per Øyvind Karlsen wrote:
> > Ahoi!
> >
> >
> > I'd just like to tell people that I've finally added metalink generation
> > to urpmi. (re: https://qa.mandriva.com/show_bug.cgi?id=29161)
>
> Just so I understand this correctly: the server sends a "generic" mirror
> list, which contains geographic coordinates, and the client locally
> generates a metalink according to its own position?
Yupp :)
>
> That's pretty slick!
>
> This would be just what I pictured for the openSUSE download client at
> http://en.opensuse.org/Libzypp/Failover, just that I put that together
> before I stumbled over metalinks, and I didn't know that they already
> exist as a way to solve these issues :)
>
> The route that we are heading with openSUSE is a bit different now,
> however, we are rather going to serve real metalinks (specific to the
> client request), and the client should be able to just use it.
>
> However I think it would still be cool to transform the metalink locally
> by e.g. injecting a local mirror (could be on the local subnet), which
> is tried first. If the metalink is generated locally anyway, this is an
> option that naturally offers itself, isn't it? Or how do you solve that
> problem with urpmi?
To be honest, I'm not entirely sure. I implemented aria2/metalink support and started work on GeoMcFly last year, but put the whole thing on hold due to concerns at Mandriva wrt. bandwidth usage and server load if generating metalinks on server side which led urpmi maintainer to implement "regular" mirror support in urpmi with priority given based on location.
I didn't get around to implement local metalink generation untill recently, which I felt like I had to finally get done since I knew it would be quite trivial with the new mirror list support, without caring to poke more around at code than getting this done (I really *hate* perl).
None the less, if not existing, such a feature would be quite useful and trivial to implement, maybe I should look into implement it if I stumble across some motivation some day, or at least suggest it as a feature to urpmi maintainer. :)
(sorry for the late reply btw.)