Metalink for improving downloads with mirrors, checksums, and more

[Anthony Bryan]

Metalink is currently a community standard for improving downloads.
Mirrors, checksums, signatures, and other info are in an XML format.

About 50 apps support it, and many projects like Fedora, openSUSE,
OpenOffice.org, Ubuntu, cURL and other use it.

If one mirror stops working, download apps failover to others.
Downloads with transfer errors can easily be repaired. The easiest way
to try it out is with DownThemAll, a Firefox extension.

The bug for it is: https://bugzilla.mozilla.org/show_bug.cgi?id=331979

This is a request for involvement and review of current works in
progress, IETF Internet Drafts.
The first, a refinement of the earlier XML format, is farther along,
while the second, using HTTP headers, is just getting started.

http://tools.ietf.org/html/draft-bryan-metalink

http://tools.ietf.org/html/draft-bryan-metalinkhttp

which looks like:

Link: <http://www2.example.com/example.ext>; rel="duplicate"
Link: <ftp://ftp.example.com/example.ext>; rel="duplicate"
Link: <http://example.com/example.ext.torrent>; rel="describedby";
type="application/x-bittorrent"
Link: <http://example.com/example.ext.metalink>; rel="describedby";
type="application/metalink4+xml"
Link: <http://example.com/example.ext.asc>; rel="describedby";
type="application/pgp-signature"
Digest: SHA=thvDyvhfIqlvFe+A9MYgxAfm1q5=

Thanks!
--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads

[Shawn Wilsher]

On 9/28/09 1:47 PM, Anthony Bryan wrote:
> This is a request for involvement and review of current works in
> progress, IETF Internet Drafts.
> The first, a refinement of the earlier XML format, is farther along,
> while the second, using HTTP headers, is just getting started.
>
> http://tools.ietf.org/html/draft-bryan-metalink
>
> http://tools.ietf.org/html/draft-bryan-metalinkhttp

I think this looks great and addresses most (if not all) of the major
concerns I've heard in the past about Metalink. We could easily make an
implementation of this that hides the fact that we are using Metalink
from the user and to them it will just work.

Cheers,

Shawn

[Anthony Bryan]

Exactly, Shawn. Metalink features should be totally transparent to the
user. Failover to other mirrors, repair of corrupted downloads, etc.

Could you briefly go over the major concerns you've heard about
Metalink in the past? Maybe we could work on them now...

One of the aspects of Metalink/HTTP (
http://tools.ietf.org/html/draft-bryan-metalinkhttp ) I'd like
comments on is mirror network etag policy coordination.
Basically, if all mirrors MUST have the same etag policy, then we can
do some cool things (Early File Mismatch Detection).
In open source mirror networks, it seems like it will be hard to get
them all coordinated. Some mirrors mirror multiple projects
(OpenOffice.org, Apache, Mozilla files all mirrored on one server).
I'd like to have two levels, mirrors that are etag coordinated, and
mirrors that just mirror the files.

thanks!

[Shawn Wilsher]

On 10/13/09 9:24 AM, Anthony Bryan wrote:
> Could you briefly go over the major concerns you've heard about
> Metalink in the past? Maybe we could work on them now...

I don't recall any of the other ones at the moment except for how it
could be confusing to the user in the UI. This new proposal clearly
solves that.

Cheers,

Shawn

[Anthony Bryan]

Gotcha. Most download apps don't show a metalink download as anything
different than a regular download.

Some show a metalink icon so if a user knows what that is, they'd know
it's got mirrors & checksums being used behind the scenes but they
don't really see that - except for download accelerators that will
show separate segments and things like that.

DownThemAll! presents some of the information in a metalink to users
which is kinda neat but entirely optional:
http://www.metalinker.org/images/dta_ubuntu.png

[Anthony Bryan]

On Mon, Sep 28, 2009 at 3:47 PM, Anthony Bryan <anthon...@gmail.com> wrote:
> Metalink is currently a community standard for improving downloads.
> Mirrors, checksums, signatures, and other info are in an XML format.
>
> About 50 apps support it, and many projects like Fedora, openSUSE,
> OpenOffice.org, Ubuntu, cURL and other use it.
>
> If one mirror stops working, download apps failover to others.
> Downloads with transfer errors can easily be repaired. The easiest way
> to try it out is with DownThemAll, a Firefox extension.
>
> The bug for it is: https://bugzilla.mozilla.org/show_bug.cgi?id=331979
>

> This is a request for involvement and review of current works in
> progress, IETF Internet Drafts.
> The first, a refinement of the earlier XML format, is farther along,
> while the second, using HTTP headers, is just getting started.
>
> http://tools.ietf.org/html/draft-bryan-metalink

The Internet Draft version of Metalink (XML) is in IETF Last Call.

It would be great to have review from anyone, but browser people would
be especially nice.

http://tools.ietf.org/html/draft-bryan-metalink

[Anthony Bryan]

On Tue, Dec 22, 2009 at 10:32 PM, Anthony Bryan <anthon...@gmail.com> wrote:
> On Mon, Sep 28, 2009 at 3:47 PM, Anthony Bryan <anthon...@gmail.com> wrote:
>> Metalink is currently a community standard for improving downloads.
>> Mirrors, checksums, signatures, and other info are in an XML format.
>>
>> About 50 apps support it, and many projects like Fedora, openSUSE,
>> OpenOffice.org, Ubuntu, cURL and other use it.
>>
>> If one mirror stops working, download apps failover to others.
>> Downloads with transfer errors can easily be repaired. The easiest way
>> to try it out is with DownThemAll, a Firefox extension.
>>
>> The bug for it is: https://bugzilla.mozilla.org/show_bug.cgi?id=331979
>>
>> This is a request for involvement and review of current works in
>> progress, IETF Internet Drafts.
>> The first, a refinement of the earlier XML format, is farther along,
>> while the second, using HTTP headers, is just getting started.
>>
>> http://tools.ietf.org/html/draft-bryan-metalink
>
> The Internet Draft version of Metalink (XML) is in IETF Last Call.
>
> It would be great to have review from anyone, but browser people would
> be especially nice.

RFC 5854 on 'The Metalink Download Description Format' is out:
http://tools.ietf.org/html/rfc5854

Work continues on Metalink in HTTP headers:
http://tools.ietf.org/html/draft-bryan-metalinkhttp

A new draft on an FTP command for requesting the hash of a file is
also underway: http://tools.ietf.org/html/draft-bryan-ftp-hash

[Anthony Bryan]

On Thu, Jun 3, 2010 at 1:42 AM, Anthony Bryan <anthon...@gmail.com> wrote:
> On Tue, Dec 22, 2009 at 10:32 PM, Anthony Bryan <anthon...@gmail.com> wrote:
>> On Mon, Sep 28, 2009 at 3:47 PM, Anthony Bryan <anthon...@gmail.com> wrote:
>>> Metalink is currently a community standard for improving downloads.
>>> Mirrors, checksums, signatures, and other info are in an XML format.
>>>
>>> About 50 apps support it, and many projects like Fedora, openSUSE,
>>> OpenOffice.org, Ubuntu, cURL and other use it.
>>>
>>> If one mirror stops working, download apps failover to others.
>>> Downloads with transfer errors can easily be repaired. The easiest way
>>> to try it out is with DownThemAll, a Firefox extension.
>>>
>>> The bug for it is: https://bugzilla.mozilla.org/show_bug.cgi?id=331979
>>>
>>> This is a request for involvement and review of current works in
>>> progress, IETF Internet Drafts.
>>> The first, a refinement of the earlier XML format, is farther along,
>>> while the second, using HTTP headers, is just getting started.
>>>
>>> http://tools.ietf.org/html/draft-bryan-metalink
>>
>> The Internet Draft version of Metalink (XML) is in IETF Last Call.
>>
>> It would be great to have review from anyone, but browser people would
>> be especially nice.

"Metalink/HTTP: Mirrors and Cryptographic Hashes in HTTP Header
Fields" is in IETF Last Call.

it would be great to have feedback from browser vendors.

http://tools.ietf.org/html/draft-bryan-metalinkhttp

this is what it looks like:

Etag: "thvDyvhfIqlvFe+A9MYgxAfm1q5="
Link: <http://www2.example.com/example.ext>; rel=duplicate
Link: <ftp://ftp.example.com/example.ext>; rel=duplicate
Link: <http://example.com/example.ext.torrent>; rel=describedby;

type="application/x-bittorrent"
Link: <http://example.com/example.ext.metalink>; rel=describedby;
type="application/metalink4+xml"
Link: <http://example.com/example.ext.asc>; rel=describedby;
type="application/pgp-signature"

Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlO
DYzYjRjYzFhOTlmNTQwYzI2M2QwM2U2MQ==

[Anthony Bryan]

On Thu, Jan 27, 2011 at 2:26 PM, Anthony Bryan <anthon...@gmail.com> wrote:
> On Thu, Jun 3, 2010 at 1:42 AM, Anthony Bryan <anthon...@gmail.com> wrote:
>> On Tue, Dec 22, 2009 at 10:32 PM, Anthony Bryan <anthon...@gmail.com> wrote:
>>> On Mon, Sep 28, 2009 at 3:47 PM, Anthony Bryan <anthon...@gmail.com> wrote:
>>>> Metalink is currently a community standard for improving downloads.
>>>> Mirrors, checksums, signatures, and other info are in an XML format.
>>>>
>>>> About 50 apps support it, and many projects like Fedora, openSUSE,
>>>> OpenOffice.org, Ubuntu, cURL and other use it.
>>>>
>>>> If one mirror stops working, download apps failover to others.
>>>> Downloads with transfer errors can easily be repaired. The easiest way
>>>> to try it out is with DownThemAll, a Firefox extension.
>>>>
>>>> The bug for it is: https://bugzilla.mozilla.org/show_bug.cgi?id=331979
>>>>
>>>> This is a request for involvement and review of current works in
>>>> progress, IETF Internet Drafts.
>>>> The first, a refinement of the earlier XML format, is farther along,
>>>> while the second, using HTTP headers, is just getting started.
>>>>
>>>

>>> It would be great to have review from anyone, but browser people would
>>> be especially nice.
>
> "Metalink/HTTP: Mirrors and Cryptographic Hashes in HTTP Header
> Fields" is in IETF Last Call.

RFC 6249 Metalink/HTTP: Mirrors and Hashes describes providing
mirrors, hashes, signatures, P2P info, and more using HTTP Link header
fields.

this information is used by file transfer clients to complete
downloads as long as one source is available, to repair errors, and
automate other advanced features. it can also be used as a p2p
bootstrap for the web.

Metalink is supported by a large number of clients & used by a number
of OS distributions, LibreOffice, OpenOffice.org, and others.


http://tools.ietf.org/html/rfc6249

Link: <http://www2.example.com/example.ext>; rel=duplicate
Link: <ftp://ftp.example.com/example.ext>; rel=duplicate
Link: <http://example.com/example.ext.torrent>; rel=describedby;
type="application/x-bittorrent"

Link: <http://example.com/example.ext.meta4>; rel=describedby;

type="application/metalink4+xml"
Link: <http://example.com/example.ext.asc>; rel=describedby;
type="application/pgp-signature"
Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlO

I have also proposed a Downloads Enhanced Community Group at the W3C
for people who are interested in working on solutions for downloads.

http://www.w3.org/community/groups/proposed/#downloads