Fwd: HTTP Extensions for Simultaneous Download from Multiple Mirrors

[Anthony Bryan]

this is similar to some of metalink's features, but done in HTTP headers.

---------- Forwarded message ----------
From: Henrik Nordstrom <hen...@henriknordstrom.net>
Date: Mon, Jul 27, 2009 at 11:03 AM
Subject: Re: HTTP Extensions for Simultaneous Download from Multiple Mirrors
To: HTTP Working Group <ietf-h...@w3.org>

This draft made a bit of surprise appearance in the transport area
meeting today:

http://tools.ietf.org/html/draft-ford-http-multi-server

My initial reaction is lots of obvious overlap with other work and
misunderstandings of basic HTTP functions like ETag.

Basic motivation behind the work may be reasonable however.

I will try to catch the author for a more in-depth discussion shortly.

Other opinions?

Regards
Henrik

--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads

[Ant Bryan]

here are my very rough ideas for Metalink in HTTP headers

http://www.ietf.org/id/draft-bryan-metalinkhttp-00.txt

briefly, it's:

Link: <http://www2.example.com/example.ext>; rel="alternate";
Link: <ftp://ftp.example.com/example.ext>; rel="alternate";
Link: <http://example.com/example.ext.torrent>; rel="describedby";
type="torrent";
Link: <http://example.com/example.ext.asc>; rel="describedby";
type="application/pgp-signature";
Digest: SHA=thvDyvhfIqlvFe+A9MYgxAfm1q5=

[Bram Neijt]

Hi all,

Personally, I don't believe in HTTP headers as mirror descriptors. One
of the main reasons to use mirrors is to keep the load of the primary
server down. If you want to keep the load down, then you should not
send the whole file to every user you encounter, but to get the mirror
list the user will have to hit the link. One way would be to only
allow a HEAD request, but that seems idiotic to me because there is no
way to make sure people would only use the head request.

I've yet to come up with a problem this additional header complexity
would solve.

Bram

[Anthony Bryan]

I was just messing around & needed to put it up because someone posted
what I was writing & I didn't want it to look like I was copying them.

here are some other shortcomings:

Appendix B. What's different...?! (to be removed by RFC Editor before

publication)

...or missing, compared to the Metalink XML format
[draft-bryan-metalink] :

o (+) Reuses existing standards without defining much new stuff.
It's more of a collection/coordinated feature set.
o (+) No XML dependency.
o (-?) Tied to HTTP, not as generic. FTP/P2P clients won't be
using it unless they also support HTTP, unlike Metalink XML.
o (---) Requires changes to server software.
o (-?) Could require some coordination of all mirror servers for
all features, which may be difficult or impossible unless you are
in control of all servers on the mirror network.
o (-) Metalink XML could be created by user (or server, but server
component/changes not required).
o (-) Also, Metalink XML files are easily mirrored on all servers.
Even if usage in that case is not as transparent, it still gives
access to users at all mirrors to all download information with no
changes needed to the server.
o (-) Not portable/archivable/emailable. Not as easy for search
engines to index?
o (-) No way to show mirror/p2p priority or geographical location
(yet).
o (---) No chunk checksums/download repair (yet).
o (-) No checksums besides MD5/SHA-1 (yet).
o (-) Not as rich metadata.
o (-) Not able to add multiple files to a download queue or create
directory structure.



http://tools.ietf.org/html/draft-bryan-metalinkhttp-00#appendix-B

[Ant Bryan]

only very small changes: http://tools.ietf.org/html/draft-bryan-metalinkhttp

more descriptive title...
MetaLinkHeaders: Mirrors and Checksums in HTTP Headers

"duplicate" replaces "alternate"

Link: <http://www2.example.com/example.ext>; rel="duplicate";
Link: <ftp://ftp.example.com/example.ext>; rel="duplicate";

Link: <http://example.com/example.ext.torrent>; rel="describedby";
type="torrent";
Link: <http://example.com/example.ext.asc>; rel="describedby";
type="application/pgp-signature";
Digest: SHA=thvDyvhfIqlvFe+A9MYgxAfm1q5=

[Neil M.]

The HEAD request actually works well in my case and makes sense for
any download application. Metalink Checker is already performing a
HEAD request to see if the MIME Type transparent content negotiation
is implemented on the server. I should be able to use this same HEAD
request to grab LINK headers. After that it proceeds with a normal
GET in any case.

For Metalink HTTP with lots of mirrors (openoffice.org, for example)
that header is going to get really big with all those Link headers.
This is particularly bad if you are doing lots of partial file GET
requests (segmented downloads). Is there a way that we can turn
those on/off (whatever is inverse of default)? Maybe they are only
sent if the "Want-Digest" header is used? The current RFC draft does
not address this. Maybe that is a comment for the draft RFC for the
LINK header?

On Aug 28, 3:59 pm, Bram Neijt <bne...@gmail.com> wrote:
> Hi all,
>
> Personally, I don't believe in HTTP headers as mirror descriptors. One
> of the main reasons to use mirrors is to keep the load of the primary
> server down. If you want to keep the load down, then you should not
> send the whole file to every user you encounter, but to get the mirror
> list the user will have to hit the link. One way would be to only
> allow a HEAD request, but that seems idiotic to me because there is no
> way to make sure people would only use the head request.
>
> I've yet to come up with a problem this additional header complexity
> would solve.
>
> Bram
>

[Ant Bryan]

http://tools.ietf.org/html/draft-bryan-metalinkhttp updated

for anyone interested, you can try out Metalink in HTTP headers w/
Neil's bits from here:

http://metalinks.svn.sourceforge.net/viewvc/metalinks/checker/
http://metalinks.svn.sourceforge.net/viewvc/metalinks/webconvert/
(Python script to convert .metalink to Apache directives)

[Neil M.]

Here is also a test URL if you don't want to setup the server side but
are developing a client:

http://www.nabber.org/projects/appupdater/metalink/Appupdater-1.2.1.exe

The Link header will point you to a sourceforge URL for download.

Neil

[Ant Bryan]

great stuff, Neil.

newest draft adds ETags text from Henrik Nordstrom:

Metalink servers and their associated mirror servers SHOULD all
share
the same ETag policy, i.e. base it on the file contents (checksum)
and not server-unique filesystem metadata. The emitted ETag may be
implemented the same as the Instance Digest for simplicity.

[Tatsuhiro]

I just thought that it would be interesting to use MetaLinkHeader with
503 code(rfc2616 10.5.4 503 Service Unavailable).
The current HTTP spec says that Retry-After header can be used with
503 to tell the clients when the server can fulfill the request.
Instead of Retry-After, we presents Link header to the clients to
navigate them to another server. I think this is more practical and
convenient than using Retry-After if we have a mirror;Why we have to
wait a single server when another mirror is available?

[A. Bram Neijt]

Hi,

I didn't get the whole HTTP header, because I think that the server
hosting the metalink should not host the files it links to (because then
it can keep serving the metalink to clients even when all mirrors are
busy or down).

However, if you combine the HTTP header metalink with "Location", server
error or other status codes it might be ok. Still, I'm not convinced
that HTTP headers are a good solution and I don't think it is currently
a good idea to start pushing another mirroring method onto the world.

What are the big advantages over normal metalinks?
What is the main goal of the HTTP header extension?

Bram

[Anthony Bryan]

Bram, I view it as an experiment. we're not pushing it on anyone. but
who knows, we may stumble onto something?

Neil added support into checker in 25 new lines of code.

with very little effort, we have another IETF discussion going on
metalink. (what we need to discuss for a draft describing metalink XML
clients). as you can see from the appendix, I don't like it as much.

This draft, compared to the Metalink XML format [draft‑bryan‑metalink] :

* (+) Reuses existing HTTP standards without defining anything new
besides a Link Relation Type. It's more of a collection/coordinated
feature set.
* (?) The existing standards don't seem to be widely implemented.
* (+) No XML dependency.
* (?) Existing Metalink XML clients can be easily converted to
support this as well.
* (-?) Tied to HTTP, not as generic. FTP/P2P clients won't be

using it unless they also support HTTP, unlike Metalink XML.

* (---) Requires changes to server software.
* (-?) Could require some coordination of all mirror servers for

all features, which may be difficult or impossible unless you are in
control of all servers on the mirror network.

* (-) Requires server-side support. Metalink XML can be created by

user (or server, but server component/changes not required).

* (-) Also, Metalink XML files are easily mirrored on all servers.

Even if usage in that case is not as transparent, it still gives

access to users at all mirrors (FTP included) to all download

information with no changes needed to the server.

* (-) Not portable/archivable/emailable. Metalink XML is used to
import/export transfer queues. Not as easy for search engines to
index?
* (-) No way to show mirror/p2p geographical location (yet).
* (-) No checksums besides MD5/SHA-1 (yet).
* (-) Not as rich metadata.
* (-) Not able to add multiple files to a download queue or create
directory structure.

[Ant Bryan]

On Sep 19, 8:54 pm, Ant Bryan <anthonybr...@gmail.com> wrote:
> great stuff, Neil.
>
> newest draft adds ETags text from Henrik Nordstrom:

Henrik expanded the ETag text considerably. there's some great stuff
in svn now about how to detect file mismatches early on in transfer
with etags or instance digest. requirements to enable either on
mirrors could be a lot, but they are nice features.

[Anthony Bryan]

On Tue, Jul 28, 2009 at 5:37 PM, Anthony Bryan <anthon...@gmail.com> wrote:
> this is similar to some of metalink's features, but done in HTTP headers.
>
>
> ---------- Forwarded message ----------
> From: Henrik Nordstrom <hen...@henriknordstrom.net>
> Date: Mon, Jul 27, 2009 at 11:03 AM
> Subject: Re: HTTP Extensions for Simultaneous Download from Multiple Mirrors
> To: HTTP Working Group <ietf-h...@w3.org>
>
>
> This draft made a bit of surprise appearance in the transport area
> meeting today:
>
> http://tools.ietf.org/html/draft-ford-http-multi-server
>
> My initial reaction is lots of obvious overlap with other work and
> misunderstandings of basic HTTP functions like ETag.
>
> Basic motivation behind the work may be reasonable however.
>
> I will try to catch the author for a more in-depth discussion shortly.
>
> Other opinions?
>
> Regards
> Henrik

Henrik & Alan (author of draft-ford-http-multi-server above) have added to this.

http://tools.ietf.org/html/draft-bryan-metalinkhttp

Neil has added support to checker and some other stuff!

http://groups.google.com/group/metalink-discussion/web/metalinkhttp