(forw) aria2c becoming the download backend in openSUSE
Peter Poeml
today, a long-awaited and longly prepared change hits openSUSE Factory
(Factory = the development branch of openSUSE).
aria2c is now the default download backend used by the package
management client. Thus, metalinks are used for everything, except some
security-critical files (which are delivered directly and shall not be
sent to mirrors, transparently negotiated).
I don't say this to boast how great openSUSE is, but rather because this
advance marks a special moment for me:
Exactly one year ago, I got mail from a guy named Anthony Bryan, who
spotted that I had mentioned metalinks in a talk at the Fosdem
conference. At the time, it was just something I heard about and
something that I intended to explore in my context. I had already
designed a plan for how to do a knowledge transfer the openSUSE download
client, that resembled Metalinks remarkably by nature.
This was the beginning of a great and intense collaboration; weeks later
I had implemented the metalink generator in MirrorBrain, reformulated
the plan for the download client to use Metalinks, a prototype became a
GSoC project, and much much more, most of which you have followed.
Mandriva and ArchLinux have since switched to use Metalinks as well, for
package updates, and Fedoras MirrorManager also uses them for some
purposes. We are working on an Internet draft etc.
I just find it great to see this happening, after this one year.
Thank you all!
Peter
Anthony Bryan
has it really been a year? :)
> From: Peter Poeml <po...@suse.de>
> To: opensuse...@opensuse.org
> Date: Wed, 25 Feb 2009 16:52:29 +0100
> Subject: Re: [opensuse-project] Proposal for a community based testing team
> On Wed, Feb 25, 2009 at 04:03:57PM +1000, Eric Springer wrote:
>> On Wed, Feb 25, 2009 at 10:30 AM, Gerald Pfeifer <g...@novell.com> wrote:
>> > What prevents you from using FACTORY?
>> >
>> > (Serious question. If there is anything, we should address it.)
>> >
>> > Gerald
>>
>> Yes, agreed.It needs to be easier. Easier to download. Easier to
>> update to. Easier to do updates, and those updates need to be more
>> reliable (in terms, of actually downloading).
>
> We are about to see significant improvement here; with changes
> implemented during the last weeks, and just hitting Factory now, Factory
> should be improved in several regards:
>
> * smaller deltas = less to download, due to "buildcompare" mechanism
> (comparing built packages with previous versions for actual changes)
> http://news.opensuse.org/2009/02/05/more-efficient-factory-development/
>
> * http://download.opensuse.org/factory/ keeps the previous package
> version available for a grace period. This gives ongoing
> updates/downloads a chance to complete successfully, even though
> Factory is being republished
> E.g. in http://download.opensuse.org/factory/repo/oss/suse/i586/
> you'll note the resulting package duplication
>
> * new, radically improved download backend in zypper/YaST,
> http://en.opensuse.org/Libzypp/Failover
> http://duncan.mac-vicar.com/blog/archives/507
>
> using aria2c and metalinks, which offers several helpful features:
> - without user interaction, it deals with mirrors that are
> incomplete, halfway updated or broken in some way, by simply
> switching to another one
> - it can download faster, using parallel connections to
> multiple mirrors and maximizing bandwidth utilization
> No more worries about the "best mirror"
> - error checking of downloads is already while in transit; not
> after downloading everything and then trying to use it.
>
> * mirror selection with stronger exploitation network locality
>
> If a client is in the same network as a mirror, it will automatically
> be assigned to that one mirror (not just looking on country level):
> http://mirrorbrain.org/news_items/2.6_network_topological_mirror_selection
>
> A further, planned, change is to increase world-wide availability of
> Factory by tighter integrating it publishing and its mirror syncing.
>
> This altogether should make Factory tracking a pleasant experience.
>
> At least the downloading part, which is the first step to using it ;)
>
>> [...] Then the system should
>> never be completely broken, and nor packages that aren't READY for
>> *testing* be pushed. (put them in their own repo, until it's deemed
>> that they're fine, and the maintainers are looking for *testing*).
>>
>> In short, try run factory as your main distro for a month -- and the
>> problems testers have should be pretty obvious =)
>
> Amen.
>
> Peter
> --
> Contact: ad...@opensuse.org (a.k.a. ftpa...@suse.com)
> #opensuse-mirrors on freenode.net
> Info: http://en.opensuse.org/Mirror_Infrastructure
>
> SUSE LINUX Products GmbH
> Research & Development
> --
> To unsubscribe, e-mail: opensuse-proje...@opensuse.org
> For additional commands, e-mail: opensuse-p...@opensuse.org
>
>
>
--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads
Tatsuhiro
I am also grad to hear that aria2 is in the backend of client.
Are you using adaptive URI selector of aria2 or more conventional one?
> > updates/downloads a chance to complete successfully, even though
> > Factory is being republished
>
> > * new, radically improved download backend in zypper/YaST,
> > http://en.opensuse.org/Libzypp/Failover
> > http://duncan.mac-vicar.com/blog/archives/507
>
> > using aria2c and metalinks, which offers several helpful features:
> > - without user interaction, it deals with mirrors that are
> > incomplete, halfway updated or broken in some way, by simply
> > switching to another one
> > - it can download faster, using parallel connections to
> > multiple mirrors and maximizing bandwidth utilization
> > No more worries about the "best mirror"
> > - error checking of downloads is already while in transit; not
> > after downloading everything and then trying to use it.
>
> > * mirror selection with stronger exploitation network locality
>
> > If a client is in the same network as a mirror, it will automatically
> > be assigned to that one mirror (not just looking on country level):
> > A further, planned, change is to increase world-wide availability of
> > Factory by tighter integrating it publishing and its mirror syncing.
>
> > This altogether should make Factory tracking a pleasant experience.
>
> > At least the downloading part, which is the first step to using it ;)
>
> >> [...] Then the system should
> >> never be completely broken, and nor packages that aren't READY for
> >> *testing* be pushed. (put them in their own repo, until it's deemed
> >> that they're fine, and the maintainers are looking for *testing*).
>
> >> In short, try run factory as your main distro for a month -- and the
> >> problems testers have should be pretty obvious =)
>
> > Amen.
>
> > Peter
> > --
> > Info:http://en.opensuse.org/Mirror_Infrastructure
>
> > SUSE LINUX Products GmbH
> > Research & Development
> > --
> > For additional commands, e-mail: opensuse-project+h...@opensuse.org
Nils
For openSUSE and for aria2 as well, which is a great client and will
now receive more attention.
And of course for metalink.
Nils
Sebastien WILLEMIJNS
On Wed, 25 Feb 2009 18:46:56 +0100, "Peter Poeml" <po...@suse.de> said:
> Thus, metalinks are used for everything, except some
> security-critical files (which are delivered directly and shall not be
> sent to mirrors, transparently negotiated).
I hope they are mirrors in all case for this kind of critical stuf ;)
Peter Poeml
Even if all files are mirrored, so they are widely available, still
there are situations where you don't want to download them from a
mirror, but rather from a trusted source. As metalinks can include the
hashes that allow verification of the content downloaded from mirror,
that solves it, but on the other hand those files (e.g. signature files)
are often so small that it is simply efficient to deliver them directly
than to create a metalink for them and send the client to further
round-trips to get the tiny file from a mirror. Sometimes the metalink
is even larger than the content in question, so it doesn't save
anything.
Peter
--
Contact: ad...@opensuse.org (a.k.a. ftpa...@suse.com)
Peter Poeml
> On Wed, Feb 25, 2009 at 07:44:38PM +0100, Sebastien WILLEMIJNS wrote:
> > On Wed, 25 Feb 2009 18:46:56 +0100, "Peter Poeml" <po...@suse.de> said:
> > > Thus, metalinks are used for everything, except some
> > > security-critical files (which are delivered directly and shall not be
> > > sent to mirrors, transparently negotiated).
> >
> > I hope they are mirrors in all case for this kind of critical stuf ;)
>
> Even if all files are mirrored, so they are widely available, still
> there are situations where you don't want to download them from a
> mirror, but rather from a trusted source. As metalinks can include the
> hashes that allow verification of the content downloaded from mirror,
> that solves it, but on the other hand those files (e.g. signature files)
> are often so small that it is simply efficient to deliver them directly
> than to create a metalink for them and send the client to further
> round-trips to get the tiny file from a mirror. Sometimes the metalink
> is even larger than the content in question, so it doesn't save
> anything.
>
> Peter
I actually wanted to mention the following link
http://www.cs.arizona.edu/people/justin/packagemanagersecurity/attacks-on-package-managers.html
but I forgot it. It discusses the issue in depth, with relation to Linux
package management clients, and attack scenarios.
Sebastien WILLEMIJNS
On Mon, 2 Mar 2009 14:34:15 +0100, "Peter Poeml" <po...@suse.de> said:
> On Wed, Feb 25, 2009 at 07:44:38PM +0100, Sebastien WILLEMIJNS wrote:
> > On Wed, 25 Feb 2009 18:46:56 +0100, "Peter Poeml" <po...@suse.de> said:
> > > Thus, metalinks are used for everything, except some
> > > security-critical files (which are delivered directly and shall not be
> > > sent to mirrors, transparently negotiated).
> >
> > I hope they are mirrors in all case for this kind of critical stuf ;)
>
> Even if all files are mirrored, so they are widely available, still
> there are situations where you don't want to download them from a
> mirror, but rather from a trusted source. As metalinks can include the
> hashes that allow verification of the content downloaded from mirror,
> that solves it, but on the other hand those files (e.g. signature files)
> are often so small that it is simply efficient to deliver them directly
> than to create a metalink for them and send the client to further
> round-trips to get the tiny file from a mirror. Sometimes the metalink
> is even larger than the content in question, so it doesn't save
> anything.
ok it is true but use a compressed metalink including a lot of files
permit to compress (redudant) mirrors informations and let only (hash)
file informations uncompressed ;)
by using a metalink you will be sure to have your tiny file (with hash
code) ;)
Sebastien WILLEMIJNS
> I actually wanted to mention the following link
> http://www.cs.arizona.edu/people/justin/packagemanagersecurity/attacks-on-package-managers.html
> but I forgot it. It discusses the issue in depth, with relation to Linux
> package management clients, and attack scenarios.
We known all online update can be a new kind of attack scenarios but it
is a little off-topic here i think because metalink is able to do other
downloads as core update OSes ;)
Peter Poeml
> ok it is true but use a compressed metalink including a lot of files
> permit to compress (redudant) mirrors informations and let only (hash)
> file informations uncompressed ;)
It's not *only* about size: Avoiding the need to generate a metalink and
to compress it lessens load on the server side. It is no work at all to
send out a tiny static file, compared to that.
> by using a metalink you will be sure to have your tiny file (with hash
> code) ;)
Good thought :) but, nope, the metalink could be forged just as well as
the tiny file itself (everybody can make the hash).
Only HTTPS helps against that, or an embedded PGP signature (which
requires the client to have the key and verify the content, obviously).
Peter Poeml
I beg to differ. I think it is not off-topic to discuss metalinks and
security in a context of online updates.
This for two reasons:
First, aria2c and metalinks is used by at least 3 or 4 major Linux
distros now. It is not a corner case. It might well be the largest
use of metalinks so far. But it is not a linux (or package management)
only topic.
In addition, online update is used by many applications, and it is often
automated. And most often, it is not done over secure channels. Many
projects don't even supply their software with signatures or hashes
provided (no .md5 or .sha1 files, no PGP signatures). Metalink usage is
considered by some at least. I think it is very relevant to have this in
mind and focus on some improvements here in the future. For example,
once OpenOffice uses MirrorBrain (and thus will offer metalinks), I want
them to also sign their code, and as a next step I would like to work on
their "update channel".
I recently used the built-in OOo updater and it downloaded a full OOo
package through conventional means. And I would certainly like to see
this happening via metalinks, because the update is not small in this
case, and in addition I would like to see it happening securely. Of
course, this is hypothetical so far, and something to work on for the
future.
So, security of software delivery certainly becomes more relevant once
metalinks are used at large scale, because the risk (and impact) of
problems rises respectively. It's not irrelevant to "smaller" use cases
though.
I don't want to bother the list with stuff that's not enough on-topic of
course. And I value your critique. Maybe I am too narrow-minded on some
things. I'll try to take care in the future :)
Thanks,
Sebastien WILLEMIJNS
On Mon, 2 Mar 2009 16:31:45 +0100, "Peter Poeml" <po...@suse.de> said:
> On Mon, Mar 02, 2009 at 03:29:39PM +0100, Sebastien WILLEMIJNS wrote:
> > by using a metalink you will be sure to have your tiny file (with hash
> > code) ;)
>
> Good thought :) but, nope, the metalink could be forged just as well as
> the tiny file itself (everybody can make the hash).
If remote server is hacked by non-scriptkiddies, no newbie or you is
able to detect it (i'm sure you do not read core updater logs every time
you launch it...) ... after, we can speak about an hacked version of
aria2 too... a man in the middle action between remote server and us...
and more and more...
Sebastien WILLEMIJNS
On Mon, 2 Mar 2009 17:49:59 +0100, "Peter Poeml" <po...@suse.de> said:
> On Mon, Mar 02, 2009 at 03:34:19PM +0100, Sebastien WILLEMIJNS wrote:
> So, security of software delivery certainly becomes more relevant once
> metalinks are used at large scale,
ok but where they are point of failures on a metalink ?
for me it can be a network trouble remote hack (apache) and/or server
hack (metalink software)
the most important fail is to grab original files... distributing sig
keys for every known distros OSes ? hmmmm...
Peter Poeml
On Fri, Feb 27, 2009 at 02:28:20AM -0800, Tatsuhiro wrote:
>
> Congratulations! Many Great improvements.
> I am also grad to hear that aria2 is in the backend of client.
> Are you using adaptive URI selector of aria2 or more conventional one?
So far, the conventional URI selector was in use. But I just noticed the
extreme advantage that the adaptive one brings about. I did some
experiments today. I hacked download.opensuse.org to return "broken
mirrors" when the client sends an "X-Broken-Mirrors: true" header. The
first mirrors at the top of metalinks will be bogus and broken URLs that
will lead to various failures. (And everybody is welcome to use that
server's metalinks for testing.)
And I realized very quickly that trying a series of unreachable mirrors,
one after the other, will make the client eventually succeed, but it
will take such a long time for each file to be downloaded that this is
leading nowhere, when dozens or hundreds of files are to be downloaded.
I had played with aria2c's server-stat-of and server-stat-if directive
before, but today was the first time I also added uri-selector=adaptive
which makes it actually use this saved data.
Wow, this is amazing. A difference like day and night.
What a fantastic ground that you, Aurelien Lefebvre and Pascal Rigaux
have layed here (and whoever else - I was just looking at the changelog).
Thanks!
Peter
--
Contact: ad...@opensuse.org (a.k.a. ftpa...@suse.com)
Tatsuhiro
adaptive URI selector. Actually it is particularly designed for that
purpose :)
> Info:http://en.opensuse.org/Mirror_Infrastructure
>
> SUSE LINUX Products GmbH
> Research & Development
>
> < 1K表示ダウンロード