VO Downloader ?

[Joe Hourclé]

I don't know what all's been going on with the HDMC discussions, as I
just see the mailing list, and let Joe Gurman handle the telecons.

I saw the mention of 'VO Downloader', and was curious as to what it
was.

... and more importantly, what protocols it'd support, and if it could
handle metalink files as input to download other stuff

http://en.wikipedia.org/wiki/Metalink
http://tools.ietf.org/html/rfc5854

(note -- metalink 4 was only released earlier this month, so there's a
reason most downloaders don't support it yet).


...

After all of my work on generating tarballs and zip archives on
demand, without staging is just causing too much trouble when I start
getting to the size we're dealing with for SDO, as we can't recover
from broken connections, or paused download (can't easily support the
HTTP Ranges header, when files are being generated each time, and you
can't be sure they're exactly the same each time).

Also, it's important for the downloader to support HTTP's Content-
Disposition header without trying to use 'HEAD' -- I tried playing
with passing a text file of URLs to wget, and it doesn't do it by
default .. and if you turn it on, it makes two downloads -- once to
examine the headers, then another pass to download the file. (they
should just load the headers into memory, parse the field, *then* open
the file for download .... or rename it once it's downloaded, but
asking a CGI to generate multi-GB files twice is just *bad*.)

-Joe H.
(yes, I'm not using my work account ... I'm not linking my google
account to all of my e-mail addresses, damnit)