Tatsuhiro's Metalink patch for curl
Anthony Bryan
http://curl.haxx.se/mail/archive-2008-12/att-0041/curl-7.19.2-metalink.patch
From: Tatsuhiro Tsujikawa
Date: Sat, 13 Dec 2008 00:31:29 +0900
Hi,
I made a patch to add very basic Metalink support to curl command-line
tool. The patch is made against curl-7.19.2. Although the patch is still
rather experimental, it provides basic fail over functionality.
Libmetalink library is used to parse Metalink XML file. I added
configure option for it in configure.ac.
Libmetalink is hosted at https://launchpad.net/libmetalink and there are
ubuntu package: https://launchpad.net/~bneijt/+archive
I added --metalink option which takes Metalink XML filename. If it is
given, curl parses the file and retrieves available URI.
If a problem occurs in a URL, curl tries next URL. In other words, it
currently supports fail over functionality.
I'm also planing to add checksum validation function.
While testing some metalinks, I found that I have to pass at least 2
arguments other than --metalink to get it work with Metalink: -C -, -f.
"-C -" is needed to continue the failed download. "-f" is needed to
avoid the curl to exit when it encounters error like 404(hmm, maybe -f
option can be omitted with further hacks).
For storing multiple URLs, I added alternate_url to struct getout.
alternate_url is a pointer to the array of URLs. In metalink download,
alternate_url is used instead of url member.
In order to use next URL when current URL is failed, I hacked 2nd URL
globing loop to iterate alternate_url. Because I thought Metalink file
really contains URLs need to be globed, I turned off URL globing in this
loop, but it may look a little bit ugly. Is there any good place to
iterate URLs for fail over?
Best regards,
Tatsuhiro Tsujikawa
--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads
Anthony Bryan
From: Dan Fandrich <dan_at_coneharvesters.com>
Date: Fri, 12 Dec 2008 11:45:15 -0800
On Sat, Dec 13, 2008 at 12:31:29AM +0900, Tatsuhiro Tsujikawa wrote:
> I made a patch to add very basic Metalink support to curl command-line
> tool. The patch is made against curl-7.19.2. Although the patch is still
> rather experimental, it provides basic fail over functionality.
This is a neat patch, but IMHO it doesn't belong in the curl command line
client. It adds functionality that goes well beyond what curl is designed
for and the UNIX philosophy behind it: to do one thing and do it well,
which is to transfer individual URLs from/to individual sites
and to allow all manner of tweaking that behaviour. That's why it
doesn't do HTML parsing, remote URL globbing, recursive transfers,
PAC proxy configuration, etc.
Metalink support is designed for something other than what curl's primary
mission is about. An indication of this is that of curl's 128 options, half
become irrelevant (and even more are marginally relevant) when doing
Metalink transfers. Adding two additional library dependencies to curl
is also an issue.
A curl-based Metalink downloader is a great idea, but as a separate program.
Of course, JMOOC, YMMV.
>>> Dan
From: Tatsuhiro Tsujikawa <tatsuhiro.t_at_gmail.com>
Date: Mon, 15 Dec 2008 00:55:29 +0900
Hi,
Dan Fandrich wrote:
> This is a neat patch, but IMHO it doesn't belong in the curl command line
> client. It adds functionality that goes well beyond what curl is designed
> for and the UNIX philosophy behind it: to do one thing and do it well,
> which is to transfer individual URLs from/to individual sites
> and to allow all manner of tweaking that behaviour. That's why it
> doesn't do HTML parsing, remote URL globbing, recursive transfers,
> PAC proxy configuration, etc.
>
> Metalink support is designed for something other than what curl's primary
> mission is about. An indication of this is that of curl's 128 options, half
> become irrelevant (and even more are marginally relevant) when doing
> Metalink transfers. Adding two additional library dependencies to curl
> is also an issue.
>
Hmm, it seems Metalink concept is an upper layer of what curl does.
If curl's mission objective includes UNIX philosophy, then it may not be
a good idea to include Metalink support in curl.
> A curl-based Metalink downloader is a great idea, but as a separate program.
> Of course, JMOOC, YMMV.
>
Do you think it is better to create, for example, a external script to
read Metalink and fed URL to curl?
Best regards,
Tatsuhiro Tsujikawa
From: Daniel Stenberg <daniel_at_haxx.se>
Date: Sun, 14 Dec 2008 20:02:53 +0100 (CET)
On Fri, 12 Dec 2008, Dan Fandrich wrote:
> This is a neat patch, but IMHO it doesn't belong in the curl command line
> client.
I think this is a philosophical dilemma: I agree that curl itself is not
really (traditionally) about doing stuff on top of the supported protocols,
but yet there are such systems that are clearly transport-related "systems"
that add value for transports that are exactly this. I then think of not only
metalink, but also bittorrent and probably other things.
I would personally like a transfer tool like curl that also supports these
"ontop" layers, even though they're strictly speaking above the "application
layer". I can see how I can either build my own build of curl that supports
these, or we could perhaps try to more officially work on a second tool that
is a curl++ (or perhaps durl ;-) which adds more on top of what the existing
curl does and works with.
I can't say I've made up my mind over how I'd like it to end up, but I do know
that I like metalink and bittorrent as download "helpers"...
--
/ daniel.haxx.se
From: Anthony Bryan <anthonybryan_at_gmail.com>
Date: Sun, 14 Dec 2008 14:53:22 -0500
> Message: 3
> Date: Fri, 12 Dec 2008 11:45:15 -0800
> From: Dan Fandrich <dan_at_coneharvesters.com>
> Subject: Re: Patch for Metalink Support
> To: curl-users_at_cool.haxx.se
> Message-ID: <20081212194513.GB12193_at_coneharvesters.com>
> Content-Type: text/plain; charset=us-ascii
>
> On Sat, Dec 13, 2008 at 12:31:29AM +0900, Tatsuhiro Tsujikawa wrote:
>> I made a patch to add very basic Metalink support to curl command-line
>> tool. The patch is made against curl-7.19.2. Although the patch is still
>> rather experimental, it provides basic fail over functionality.
>
> This is a neat patch, but IMHO it doesn't belong in the curl command line
> client. It adds functionality that goes well beyond what curl is designed
> for and the UNIX philosophy behind it: to do one thing and do it well,
> which is to transfer individual URLs from/to individual sites
> and to allow all manner of tweaking that behaviour. That's why it
> doesn't do HTML parsing, remote URL globbing, recursive transfers,
> PAC proxy configuration, etc.
Dan, thanks for taking the time to check out the patch.
obviously, your opinion is valued here. I understand where you're coming from...
the philosophy behind Metalink is that we want a download to succeed
if there is any way possible for it to. if a server goes down, switch
over to another. if there's an error in transfer, detect it and
re-get that chunk, etc.
where curl may be for tweaking & manual options galore, we want people
to have an error free completed download with no thought or
intervention necessary. :)
perhaps Metalink doesn't mesh up w/ curl. like you said, we're
different, but maybe there's a place where we intersect. maybe
metalink falls into "a busload of other useful tricks" category, from
the description of curl on the site.
as much as I'd like to see it in curl, I hope this isn't a round peg,
square hole deal. I think it'd be useful to some people. but I could
be wrong.
Daniel Stenberg has also blogged about this and added to the feature
bounty: http://daniel.haxx.se/blog/2008/11/09/metalink-in-curl-bounty/
> Metalink support is designed for something other than what curl's primary
> mission is about. An indication of this is that of curl's 128 options, half
> become irrelevant (and even more are marginally relevant) when doing
> Metalink transfers. Adding two additional library dependencies to curl
> is also an issue.
a great point! as I stated above, Metalink does aim to simplify
things. this may be different from curl's primary mission, but if curl
can also offered simplified completed transfers in many cases, then
that could be nice too.
I don't know what can be done about the two additional libraries, &
agree that's an issue.
> A curl-based Metalink downloader is a great idea, but as a separate program.
> Of course, JMOOC, YMMV.
Metadl ( http://hampus.vox.nu/metadl/ ), used in Wubi by Ubuntu is one
such Metalink downloader that uses libcurl.
--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads
Nicolas Alvarez
> This is a neat patch, but IMHO it doesn't belong in the curl command line
> client. It adds functionality that goes well beyond what curl is designed
> for and the UNIX philosophy behind it: to do one thing and do it well,
> which is to transfer individual URLs from/to individual sites
> and to allow all manner of tweaking that behaviour. That's why it
> doesn't do HTML parsing, remote URL globbing, recursive transfers,
> PAC proxy configuration, etc.
>
> Metalink support is designed for something other than what curl's primary
> mission is about. An indication of this is that of curl's 128 options,
> half become irrelevant (and even more are marginally relevant) when doing
> Metalink transfers. Adding two additional library dependencies to curl
> is also an issue.
Can I say "I told you so" or will I get flamed?
Anthony Bryan
no flames :)
it's kinda confusing when Daniel, curl author, writes
"The Metalink guys host a list of project ideas and one of those ideas
is to add metalink support to curl, and I recently bumped the stakes a
bit by raising the bounty with an additional 200 USD so that the offer
is now 500 USD for the person or team that brings the feature as
described.
My primary motivation for doing this is that I like the metalink idea
and I'd like to help making sure it gets used more widely."
http://daniel.haxx.se/blog/2008/11/09/metalink-in-curl-bounty/
Tatsuhiro
with Daniel.
Anthony Bryan
Date: Wed, 4 Feb 2009 23:10:15 +0100 (CET)
From: Daniel Stenberg <dan...@haxx.se>
Message-ID: <alpine.DEB.1.10.0...@yvahk2.pbagnpgbe.fr>
Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote:
> I made a patch to add very basic Metalink support to curl command-line tool.
> The patch is made against curl-7.19.2. Although the patch is still rather
> experimental, it provides basic fail over functionality.
have an initial request:
I'd like this patch to have as low impact as possible on the existing code so
I would really like to see perhaps a separate file added with the necessary
metalink support functions or similar, so that all devs and users who don't
care for metalink wouldn't have to see many traces of it.
--
/ daniel.haxx.se
Tatsuhiro
Daniel Stenberg wrote:
> On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote:
>
> I would like to play around with this patch and functionality a bit, but I have an initial request:
>
> I'd like this patch to have as low impact as possible on the existing code so I would really like to see perhaps a separate file added with the necessary metalink support functions or similar, so that all devs and users who don't care for metalink wouldn't have to see many traces of it.
>
OK. In my first patch, all metalink functions were integrated in
main.c. I'll make new patch that call metalink function in another
file after options are parsed.
Best regards,
Tatsuhiro Tsujikawa
On 2月6日, 午前1:19, Anthony Bryan <anthonybr...@gmail.com> wrote:
> good news?
>
> Date: Wed, 4 Feb 2009 23:10:15 +0100 (CET)
> From: Daniel Stenberg <dan...@haxx.se>
> Subject: Re: Patch for Metalink Support
> To: the curl tool <curl-us...@cool.haxx.se>
Anthony Bryan
Tatsuhiro has revived his 3 year old patch & it's very nice to be able
to use metalink with curl!
the discussion is happening on the curl-users list, & I don't think it
will make it into the next version, but hopefully the one after that.
curl's metalink support uses libmetalink, which Tatsuhiro has just
applied a few fixes to.
I'm thinking it might be good to have meta4/RFC 5854 support in
libmetalink soon too
> You received this message because you are subscribed to the Google Groups "Metalink Discussion" group.
> To post to this group, send email to metalink-...@googlegroups.com
> To unsubscribe from this group, send email to metalink-discus...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/metalink-discussion?hl=en
> -~----------~----~----~----~------~----~------~--~---
Anthony Bryan
>> Message-ID: <alpine.DEB.1.10.0902042227120.2058@yvahk2.pbagnpgbe.fr>
> To post to this group, send email to metalink-discussion@googlegroups.com
> To unsubscribe from this group, send email to metalink-discussion+unsub...@googlegroups.com
Anthony Bryan
Date: Thu, 24 May 2012 18:28:13 +0200 (CEST)
Title: after 7.26.0
Hi friends
As we've now gotten rid of the release, we once gain open the flood gates for
new features to enter. We will now allow new features getting in during the
coming month.
I especially want to point out and work with the following features to get
merged soon:
1 - the new auth callback for libcurl, by Joe Mason
2 - metalink support for curl, by Tatsuhiro Tsujikaw (note that this is not
for libcurl)
3 - schannel support for libcurl, by Marc Hoersken
... but I'm not closing the door for others or other features, so please show
us what you have as soon as possible if you want it get merged for the next
release.
Thanks for flying curl!
-- / daniel.haxx.se ------------------------------------------------------------------- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Anthony Bryan
metalink support is in mainline curl!curl 7.26 was just released, so the next release should have it & be out in two months!1) first get libmetalinkhttps://launchpad.net/libmetalinkyou can get the latest withbzr branch lp:libmetalink2) get curl withgit clone git://github.com/bagder/curl.git
then
./buildconf./configure --with-libmetalinkmake
Anthony Bryan
> metalink support is in mainline curl!
> curl 7.26 was just released, so the next release should have it & be out in
> two months!
> Daniel Stenberg @bagder
>
> at 23:24, I committed the 21 changes to #curl that introduced#metalink
> support
>
>
> 1) first get libmetalink
> https://launchpad.net/libmetalink
>
> you can get the latest with
> bzr branch lp:libmetalink
>
> 2) get curl with
>
> git clone git://github.com/bagder/curl.git
>
>
> then
>
> ./buildconf
> ./configure --with-libmetalink
> make
>
>
> (Tatsuhiro helped me, I needed './configure
> PKG_CONFIG_PATH=/usr/local/lib/pkgconfig --with-libmetalink' because I
> upgraded to Fedora 17)
>
> 3) you can test it out with
> curl URL_to_remote_Metalink3_file or curl --metalink local.metalink
curl --metalink URL_to_remote_Metalink (3 or 4, yay!) or curl
--metalink file://local.metalink
please try it out from the latest libmetalink & curl repo. I think
we're about 2 weeks away from a release of curl 7.27 with metalink
support!
Anthony Bryan
From: Daniel Stenberg <dan...@haxx.se>
To: Curl Announce -- curl users <curl-...@cool.haxx.se>,
curl-a...@cool.haxx.se, libcurl hacking
<curl-l...@cool.haxx.se>
Subject: ANNOUNCE: curl and libcurl 7.27.0
Message-ID: <alpine.DEB.2.00.1...@tvnag.unkk.fr>
Content-Type: text/plain; charset="iso-8859-15"; Format="flowed"
Friends!
Hurry on over to http://curl.haxx.se/download.html and download the fresh
release! This time with new features and two new SSL libraries supported.
Curl and libcurl 7.27.0
Public curl releases: 128
Command line options: 152
curl_easy_setopt() options: 199
Public functions in libcurl: 58
Known libcurl bindings: 39
Contributors: 953
This release includes the following changes:
o nss: use human-readable error messages provided by NSS
o added --metalink for metalink download support [5]
o pop3: Added support for sasl plain text authentication
o pop3: Added support for sasl login authentication
o pop3: Added support for sasl ntlm authentication
o pop3: Added support for sasl cram-md5 authentication
o pop3: Added support for sasl digest-md5 authentication
o pop3: Added support for apop authentication
o Added support for Schannel (Native Windows) SSL/TLS encryption [2]
o Added support for Darwin SSL (Native Mac OS X and iOS) [6]
o http: print reason phrase from HTTP status line on error [8]
This release includes the following bugfixes:
o pop3: Fixed the issue of having to supply the user name for all requests
o configure: fix LDAPS disabling related misplaced closing parenthesis
o cmdline: made -D option work with -O and -J
o configure: Fix libcurl.pc and curl-config generation for static MingW*
cross builds
o ssl: fix duplicated SSL handshake with multi interface and proxy [1]
o winbuild: Fix Makefile.vc ignoring USE_IPV6 and USE_IDN flags
o OpenSSL: support longer certificate subject names [3]
o openldap: OOM fixes
o log2changes.pl: fix the Version output
o lib554.c: use curl_formadd() properly [4]
o urldata.h: fix cyassl build clash with wincrypt.h
o cookies: changed the URL in the cookiejar headers [7]
o http-proxy: keep CONNECT connections alive (for NTLM)
o NTLM SSPI: fixed to work with unicode user names and passwords
o OOM fix in the curl tool when cloning cmdline options
o fixed some examples to use curl_global_init() properly
o cmdline: stricter numerical option parser
o HTTP HEAD: don't force-close after response-headers
o test231: fix wrong -C use
o docs: switch to proper UTF-8 for text file encoding
o keepalive: DragonFly uses milliseconds [9]
o HTTP Digest: Client's "qop" value should not be quoted
o make distclean works again
This release includes the following known bugs:
o see docs/KNOWN_BUGS (http://curl.haxx.se/docs/knownbugs.html)
This release would not have looked like this without help, code, reports and
advice from friends like these:
Anthony Bryan, Guenter Knauf, Kamil Dudka, Steve Holme, Tatsuhiro Tsujikawa,
Yang Tse, Gokhan Sengun, Marc Hoersken, Ghennadi Procopciuc, Gisle Vanem,
Mark Salisbury, Anthony G. Basile, Enrico Scholz, Robert B. Harris,
Neil Bowers, Marcel Raad, Christian H?gele, Philip Craig, Nick Zitzmann,
Eelco Dolstra, Anton Yabchinskiy, Santhana Todatry, John Marino
Thanks! (and sorry if I forgot to mention someone)
References to bug reports and discussions on issues:
[1] = https://bugzilla.redhat.com/788526
[2] = http://daniel.haxx.se/blog/2012/06/12/schannel-support-in-libcurl/
[3] = http://curl.haxx.se/bug/view.cgi?id=3533045
[4] = http://curl.haxx.se/mail/lib-2012-06/0001.html
[5] = http://daniel.haxx.se/blog/2012/06/03/curling-the-metalink/
[6] = http://daniel.haxx.se/blog/2012/06/28/darwin-native-ssl-for-curl/
[7] = http://daniel.haxx.se/blog/2012/07/08/curls-new-http-cookies-docs/
[8] = https://bugzilla.redhat.com/676596
[9] = http://curl.haxx.se/bug/view.cgi?id=3546257
--
/ daniel.haxx.se
Anthony Bryan
http://en.wikipedia.org/wiki/Metalink
Thank you to curl community, especially Daniel Stenberg who accepted Metalink patch, Yang Tse who quickly merged our patch and provided security enhancement ideas and many test cases for Metalink feature and Guenter Knauf who provided many libmetalink enhancement patches which fixes many portability issues.
Since Metalink feature in curl is optional at the moment, I'll write a simple tutorial to get it work. First grab libmetalink-0.1.0 tar ball.
https://launchpad.net/libmetalink/trunk/libmetalink-0.1.0/+download/libmetalink-0.1.0.tar.bz2
(Please note that libmetalink-0.0.3 does not work with curl).
Libmetalink depends on Expat XML library. If you are using Ubuntu, install libexpat-dev package. Unpack libmetalink tar ball and configure and make install.
Probably one of most appealing point of Metalink is automatic checksum validation. To do this, curl must be built with OpenSSL or GnuTLS.
For OpenSSL, install libssl-dev.
After libmetalink was installed, download curl 7.27.0 source tar ball.
Unpack it and run configure script with --with-libmetalink option.
Make sure that metalink support is enabled. If it is enabled, you can see the following line in the output of configure script:
metalink support: enabled
Then run make to build curl tool binary.
To download a file using Metalink feature, provide URI to Metalink file.
Curl download page offers several Metalink downloads. Let's use it as an example. Use --metalink option to enable Metalink feature.
$ src/curl --metalink 'http://curl.haxx.se/metalink.cgi?curl=tar.bz2'
Metalink: parsing (http://curl.haxx.se/metalink.cgi?curl=tar.bz2) metalink/XML...
Metalink: parsing (http://curl.haxx.se/metalink.cgi?curl=tar.bz2) OK
[1/12]: http://curl.freeby.pctools.cl/download/curl-7.27.0.tar.bz2 --> curl-7.27.0.tar.bz2
--curl--http://curl.freeby.pctools.cl/download/curl-7.27.0.tar.bz2
Metalink: fetching (curl-7.27.0.tar.bz2) from (http://curl.freeby.pctools.cl/download/curl-7.27.0.tar.bz2)...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2418k 100 2418k 0 0 191k 0 0:00:12 0:00:12 --:--:-- 134k
Metalink: fetching (curl-7.27.0.tar.bz2) from (http://curl.freeby.pctools.cl/download/curl-7.27.0.tar.bz2) OK
Metalink: validating (curl-7.27.0.tar.bz2)...
Metalink: validating (curl-7.27.0.tar.bz2) OK
OK, I'll explain what happened behind the scene.
First, curl downloaded metalink XML file pointed by URI 'http://curl.haxx.se/metalink.cgi?curl=tar.bz2'.
The content of XML file was not saved on the disk. It was parsed on the fly.
After the XML file was downloaded and parsed, curl retrieves mirror URIs from the parsed result and download the file pointed by it. In this example, the Metalink XML file contains 12 URIs. These URIs are used as fallback.
As a first attempt, curl used http://curl.freeby.pctools.cl/download/curl-7.27.0.tar.bz2 to download curl 7.27.0 tar ball. Unlike normal curl behavior, curl saves the file using the filename described in Metalink XML file. In this example, the filename is "curl-7.27.0.tar.bz2". -O option is not necessary.
After download completed, curl automatically calculated checksum of the download file and compared it with the correct one described in Metalink XML file and surely they were matched.
On Friday, July 27, 2012 6:27:50 PM UTC-4, Anthony Bryan wrote:
Date: Fri, 27 Jul 2012 23:23:51 +0200 (CEST)
From: Daniel Stenberg <dan...@haxx.se>
To: Curl Announce -- curl users <curl-...@cool.haxx.se>,
curl-a...@cool.haxx.se, libcurl hacking
<curl-l...@cool.haxx.se>
Subject: ANNOUNCE: curl and libcurl 7.27.0
Message-ID: <alpine.DEB.2.00.1207270058470.22192@tvnag.unkk.fr>
Content-Type: text/plain; charset="iso-8859-15"; Format="flowed"
Friends!
Hurry on over to http://curl.haxx.se/download.html and download the fresh
release! This time with new features and two new SSL libraries supported.
Curl and libcurl 7.27.0