From: Tatsuhiro Tsujikawa Date: Sat, 13 Dec 2008 00:31:29 +0900
Hi,
I made a patch to add very basic Metalink support to curl command-line tool. The patch is made against curl-7.19.2. Although the patch is still rather experimental, it provides basic fail over functionality.
I added --metalink option which takes Metalink XML filename. If it is given, curl parses the file and retrieves available URI. If a problem occurs in a URL, curl tries next URL. In other words, it currently supports fail over functionality. I'm also planing to add checksum validation function.
While testing some metalinks, I found that I have to pass at least 2 arguments other than --metalink to get it work with Metalink: -C -, -f. "-C -" is needed to continue the failed download. "-f" is needed to avoid the curl to exit when it encounters error like 404(hmm, maybe -f option can be omitted with further hacks).
For storing multiple URLs, I added alternate_url to struct getout. alternate_url is a pointer to the array of URLs. In metalink download, alternate_url is used instead of url member. In order to use next URL when current URL is failed, I hacked 2nd URL globing loop to iterate alternate_url. Because I thought Metalink file really contains URLs need to be globed, I turned off URL globing in this loop, but it may look a little bit ugly. Is there any good place to iterate URLs for fail over?
Best regards,
Tatsuhiro Tsujikawa
-- (( Anthony Bryan ... Metalink [ http://www.metalinker.org ] )) Easier, More Reliable, Self Healing Downloads
From: Dan Fandrich <dan_at_coneharvesters.com> Date: Fri, 12 Dec 2008 11:45:15 -0800
On Sat, Dec 13, 2008 at 12:31:29AM +0900, Tatsuhiro Tsujikawa wrote: > I made a patch to add very basic Metalink support to curl command-line > tool. The patch is made against curl-7.19.2. Although the patch is still > rather experimental, it provides basic fail over functionality.
This is a neat patch, but IMHO it doesn't belong in the curl command line client. It adds functionality that goes well beyond what curl is designed for and the UNIX philosophy behind it: to do one thing and do it well, which is to transfer individual URLs from/to individual sites and to allow all manner of tweaking that behaviour. That's why it doesn't do HTML parsing, remote URL globbing, recursive transfers, PAC proxy configuration, etc.
Metalink support is designed for something other than what curl's primary mission is about. An indication of this is that of curl's 128 options, half become irrelevant (and even more are marginally relevant) when doing Metalink transfers. Adding two additional library dependencies to curl is also an issue.
A curl-based Metalink downloader is a great idea, but as a separate program. Of course, JMOOC, YMMV.
Dan Fandrich wrote: > This is a neat patch, but IMHO it doesn't belong in the curl command line > client. It adds functionality that goes well beyond what curl is designed > for and the UNIX philosophy behind it: to do one thing and do it well, > which is to transfer individual URLs from/to individual sites > and to allow all manner of tweaking that behaviour. That's why it > doesn't do HTML parsing, remote URL globbing, recursive transfers, > PAC proxy configuration, etc.
> Metalink support is designed for something other than what curl's primary > mission is about. An indication of this is that of curl's 128 options, half > become irrelevant (and even more are marginally relevant) when doing > Metalink transfers. Adding two additional library dependencies to curl > is also an issue.
Hmm, it seems Metalink concept is an upper layer of what curl does. If curl's mission objective includes UNIX philosophy, then it may not be a good idea to include Metalink support in curl.
> A curl-based Metalink downloader is a great idea, but as a separate program. > Of course, JMOOC, YMMV.
Do you think it is better to create, for example, a external script to read Metalink and fed URL to curl?
Best regards,
Tatsuhiro Tsujikawa
From: Daniel Stenberg <daniel_at_haxx.se> Date: Sun, 14 Dec 2008 20:02:53 +0100 (CET)
On Fri, 12 Dec 2008, Dan Fandrich wrote: > This is a neat patch, but IMHO it doesn't belong in the curl command line > client.
I think this is a philosophical dilemma: I agree that curl itself is not really (traditionally) about doing stuff on top of the supported protocols, but yet there are such systems that are clearly transport-related "systems" that add value for transports that are exactly this. I then think of not only metalink, but also bittorrent and probably other things.
I would personally like a transfer tool like curl that also supports these "ontop" layers, even though they're strictly speaking above the "application layer". I can see how I can either build my own build of curl that supports these, or we could perhaps try to more officially work on a second tool that is a curl++ (or perhaps durl ;-) which adds more on top of what the existing curl does and works with.
I can't say I've made up my mind over how I'd like it to end up, but I do know that I like metalink and bittorrent as download "helpers"...
-- / daniel.haxx.se
From: Anthony Bryan <anthonybryan_at_gmail.com> Date: Sun, 14 Dec 2008 14:53:22 -0500
> Message: 3 > Date: Fri, 12 Dec 2008 11:45:15 -0800 > From: Dan Fandrich <dan_at_coneharvesters.com> > Subject: Re: Patch for Metalink Support > To: curl-users_at_cool.haxx.se > Message-ID: <20081212194513.GB12193_at_coneharvesters.com> > Content-Type: text/plain; charset=us-ascii
> On Sat, Dec 13, 2008 at 12:31:29AM +0900, Tatsuhiro Tsujikawa wrote: >> I made a patch to add very basic Metalink support to curl command-line >> tool. The patch is made against curl-7.19.2. Although the patch is still >> rather experimental, it provides basic fail over functionality.
> This is a neat patch, but IMHO it doesn't belong in the curl command line > client. It adds functionality that goes well beyond what curl is designed > for and the UNIX philosophy behind it: to do one thing and do it well, > which is to transfer individual URLs from/to individual sites > and to allow all manner of tweaking that behaviour. That's why it > doesn't do HTML parsing, remote URL globbing, recursive transfers, > PAC proxy configuration, etc.
Dan, thanks for taking the time to check out the patch. obviously, your opinion is valued here. I understand where you're coming from...
the philosophy behind Metalink is that we want a download to succeed if there is any way possible for it to. if a server goes down, switch over to another. if there's an error in transfer, detect it and re-get that chunk, etc. where curl may be for tweaking & manual options galore, we want people to have an error free completed download with no thought or intervention necessary. :)
perhaps Metalink doesn't mesh up w/ curl. like you said, we're different, but maybe there's a place where we intersect. maybe metalink falls into "a busload of other useful tricks" category, from the description of curl on the site.
as much as I'd like to see it in curl, I hope this isn't a round peg, square hole deal. I think it'd be useful to some people. but I could be wrong.
> Metalink support is designed for something other than what curl's primary > mission is about. An indication of this is that of curl's 128 options, half > become irrelevant (and even more are marginally relevant) when doing > Metalink transfers. Adding two additional library dependencies to curl > is also an issue.
a great point! as I stated above, Metalink does aim to simplify things. this may be different from curl's primary mission, but if curl can also offered simplified completed transfers in many cases, then that could be nice too.
I don't know what can be done about the two additional libraries, & agree that's an issue.
> A curl-based Metalink downloader is a great idea, but as a separate program. > Of course, JMOOC, YMMV.
Metadl ( http://hampus.vox.nu/metadl/ ), used in Wubi by Ubuntu is one such Metalink downloader that uses libcurl.
-- (( Anthony Bryan ... Metalink [ http://www.metalinker.org ] )) Easier, More Reliable, Self Healing Downloads
> I made a patch to add very basic Metalink support to curl command-line > tool. The patch is made against curl-7.19.2. Although the patch is still > rather experimental, it provides basic fail over functionality.
> I added --metalink option which takes Metalink XML filename. If it is > given, curl parses the file and retrieves available URI. > If a problem occurs in a URL, curl tries next URL. In other words, it > currently supports fail over functionality. > I'm also planing to add checksum validation function.
> While testing some metalinks, I found that I have to pass at least 2 > arguments other than --metalink to get it work with Metalink: -C -, -f. > "-C -" is needed to continue the failed download. "-f" is needed to > avoid the curl to exit when it encounters error like 404(hmm, maybe -f > option can be omitted with further hacks).
> For storing multiple URLs, I added alternate_url to struct getout. > alternate_url is a pointer to the array of URLs. In metalink download, > alternate_url is used instead of url member. > In order to use next URL when current URL is failed, I hacked 2nd URL > globing loop to iterate alternate_url. Because I thought Metalink file > really contains URLs need to be globed, I turned off URL globing in this > loop, but it may look a little bit ugly. Is there any good place to > iterate URLs for fail over?
Anthony Bryan wrote: > This is a neat patch, but IMHO it doesn't belong in the curl command line > client. It adds functionality that goes well beyond what curl is designed > for and the UNIX philosophy behind it: to do one thing and do it well, > which is to transfer individual URLs from/to individual sites > and to allow all manner of tweaking that behaviour. That's why it > doesn't do HTML parsing, remote URL globbing, recursive transfers, > PAC proxy configuration, etc.
> Metalink support is designed for something other than what curl's primary > mission is about. An indication of this is that of curl's 128 options, > half become irrelevant (and even more are marginally relevant) when doing > Metalink transfers. Adding two additional library dependencies to curl > is also an issue.
> Anthony Bryan wrote: >> This is a neat patch, but IMHO it doesn't belong in the curl command line >> client. It adds functionality that goes well beyond what curl is designed >> for and the UNIX philosophy behind it: to do one thing and do it well, >> which is to transfer individual URLs from/to individual sites >> and to allow all manner of tweaking that behaviour. That's why it >> doesn't do HTML parsing, remote URL globbing, recursive transfers, >> PAC proxy configuration, etc.
>> Metalink support is designed for something other than what curl's primary >> mission is about. An indication of this is that of curl's 128 options, >> half become irrelevant (and even more are marginally relevant) when doing >> Metalink transfers. Adding two additional library dependencies to curl >> is also an issue.
> Can I say "I told you so" or will I get flamed?
no flames :)
it's kinda confusing when Daniel, curl author, writes
"The Metalink guys host a list of project ideas and one of those ideas is to add metalink support to curl, and I recently bumped the stakes a bit by raising the bounty with an additional 200 USD so that the offer is now 500 USD for the person or team that brings the feature as described.
My primary motivation for doing this is that I like the metalink idea and I'd like to help making sure it gets used more widely."
> > Anthony Bryan wrote:
> >> This is a neat patch, but IMHO it doesn't belong in the curl command line
> >> client. It adds functionality that goes well beyond what curl is designed
> >> for and the UNIX philosophy behind it: to do one thing and do it well,
> >> which is to transfer individual URLs from/to individual sites
> >> and to allow all manner of tweaking that behaviour. That's why it
> >> doesn't do HTML parsing, remote URL globbing, recursive transfers,
> >> PAC proxy configuration, etc.
> >> Metalink support is designed for something other than what curl's primary
> >> mission is about. An indication of this is that of curl's 128 options,
> >> half become irrelevant (and even more are marginally relevant) when doing
> >> Metalink transfers. Adding two additional library dependencies to curl
> >> is also an issue.
> > Can I say "I told you so" or will I get flamed?
> no flames :)
> it's kinda confusing when Daniel, curl author, writes
> "The Metalink guys host a list of project ideas and one of those ideas
> is to add metalink support to curl, and I recently bumped the stakes a
> bit by raising the bounty with an additional 200 USD so that the offer
> is now 500 USD for the person or team that brings the feature as
> described.
> My primary motivation for doing this is that I like the metalink idea
> and I'd like to help making sure it gets used more widely."
Date: Wed, 4 Feb 2009 23:10:15 +0100 (CET)
From: Daniel Stenberg <dan...@haxx.se>
Subject: Re: Patch for Metalink Support
To: the curl tool <curl-us...@cool.haxx.se>
Message-ID: <alpine.DEB.1.10.0902042227120.2058@yvahk2.pbagnpgbe.fr>
Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote:
> I made a patch to add very basic Metalink support to curl command-line tool.
> The patch is made against curl-7.19.2. Although the patch is still rather
> experimental, it provides basic fail over functionality.
I would like to play around with this patch and functionality a bit, but I
have an initial request:
I'd like this patch to have as low impact as possible on the existing code so
I would really like to see perhaps a separate file added with the necessary
metalink support functions or similar, so that all devs and users who don't
care for metalink wouldn't have to see many traces of it.
>> > Anthony Bryan wrote:
>> >> This is a neat patch, but IMHO it doesn't belong in the curl command line
>> >> client. It adds functionality that goes well beyond what curl is designed
>> >> for and the UNIX philosophy behind it: to do one thing and do it well,
>> >> which is to transfer individual URLs from/to individual sites
>> >> and to allow all manner of tweaking that behaviour. That's why it
>> >> doesn't do HTML parsing, remote URL globbing, recursive transfers,
>> >> PAC proxy configuration, etc.
>> >> Metalink support is designed for something other than what curl's primary
>> >> mission is about. An indication of this is that of curl's 128 options,
>> >> half become irrelevant (and even more are marginally relevant) when doing
>> >> Metalink transfers. Adding two additional library dependencies to curl
>> >> is also an issue.
>> > Can I say "I told you so" or will I get flamed?
>> no flames :)
>> it's kinda confusing when Daniel, curl author, writes
>> "The Metalink guys host a list of project ideas and one of those ideas
>> is to add metalink support to curl, and I recently bumped the stakes a
>> bit by raising the bounty with an additional 200 USD so that the offer
>> is now 500 USD for the person or team that brings the feature as
>> described.
>> My primary motivation for doing this is that I like the metalink idea
>> and I'd like to help making sure it gets used more widely."
Daniel Stenberg wrote: > On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote:
> I would like to play around with this patch and functionality a bit, but I have an initial request:
> I'd like this patch to have as low impact as possible on the existing code so I would really like to see perhaps a separate file added with the necessary metalink support functions or similar, so that all devs and users who don't care for metalink wouldn't have to see many traces of it.
OK. In my first patch, all metalink functions were integrated in main.c. I'll make new patch that call metalink function in another file after options are parsed.
Best regards,
Tatsuhiro Tsujikawa
On 2月6日, 午前1:19, Anthony Bryan <anthonybr...@gmail.com> wrote:
> Date: Wed, 4 Feb 2009 23:10:15 +0100 (CET) > From: Daniel Stenberg <dan...@haxx.se> > Subject: Re: Patch for Metalink Support > To: the curl tool <curl-us...@cool.haxx.se> > Message-ID: <alpine.DEB.1.10.0902042227120.2...@yvahk2.pbagnpgbe.fr> > Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
> On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote: > > I made a patch to add very basic Metalink support to curl command-line tool. > > The patch is made against curl-7.19.2. Although the patch is still rather > > experimental, it provides basic fail over functionality.
> I would like to play around with this patch and functionality a bit, but I > have an initial request:
> I'd like this patch to have as low impact as possible on the existing code so > I would really like to see perhaps a separate file added with the necessary > metalink support functions or similar, so that all devs and users who don't > care for metalink wouldn't have to see many traces of it.
> --
> / daniel.haxx.se
> On Wed, Dec 17, 2008 at 6:55 AM, Tatsuhiro <tatsuhir...@gmail.com> wrote:
> > Hmm, maybe other curl devs have not yet shared the idea about Metalink > > with Daniel.
> > On 12月17日, 午前2:22, "Anthony Bryan" <anthonybr...@gmail.com> wrote: > >> On Mon, Dec 15, 2008 at 6:05 PM, Nicolas Alvarez
> >> > Anthony Bryan wrote: > >> >> This is a neat patch, but IMHO it doesn't belong in the curl command line > >> >> client. It adds functionality that goes well beyond what curl is designed > >> >> for and the UNIX philosophy behind it: to do one thing and do it well, > >> >> which is to transfer individual URLs from/to individual sites > >> >> and to allow all manner of tweaking that behaviour. That's why it > >> >> doesn't do HTML parsing, remote URL globbing, recursive transfers, > >> >> PAC proxy configuration, etc.
> >> >> Metalink support is designed for something other than what curl's primary > >> >> mission is about. An indication of this is that of curl's 128 options, > >> >> half become irrelevant (and even more are marginally relevant) when doing > >> >> Metalink transfers. Adding two additional library dependencies to curl > >> >> is also an issue.
> >> > Can I say "I told you so" or will I get flamed?
> >> no flames :)
> >> it's kinda confusing when Daniel, curl author, writes
> >> "The Metalink guys host a list of project ideas and one of those ideas > >> is to add metalink support to curl, and I recently bumped the stakes a > >> bit by raising the bounty with an additional 200 USD so that the offer > >> is now 500 USD for the person or team that brings the feature as > >> described.
> >> My primary motivation for doing this is that I like the metalink idea > >> and I'd like to help making sure it gets used more widely."
Tatsuhiro has revived his 3 year old patch & it's very nice to be able
to use metalink with curl!
the discussion is happening on the curl-users list, & I don't think it
will make it into the next version, but hopefully the one after that.
curl's metalink support uses libmetalink, which Tatsuhiro has just
applied a few fixes to.
I'm thinking it might be good to have meta4/RFC 5854 support in
libmetalink soon too
On Fri, Feb 6, 2009 at 8:26 PM, Tatsuhiro <tatsuhir...@gmail.com> wrote:
> Good news!
> Daniel Stenberg wrote:
>> On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote:
>> I would like to play around with this patch and functionality a bit, but I have an initial request:
>> I'd like this patch to have as low impact as possible on the existing code so I would really like to see perhaps a separate file added with the necessary metalink support functions or similar, so that all devs and users who don't care for metalink wouldn't have to see many traces of it.
> OK. In my first patch, all metalink functions were integrated in
> main.c. I'll make new patch that call metalink function in another
> file after options are parsed.
> Best regards,
> Tatsuhiro Tsujikawa
> On 2月6日, 午前1:19, Anthony Bryan <anthonybr...@gmail.com> wrote:
>> good news?
>> Date: Wed, 4 Feb 2009 23:10:15 +0100 (CET)
>> From: Daniel Stenberg <dan...@haxx.se>
>> Subject: Re: Patch for Metalink Support
>> To: the curl tool <curl-us...@cool.haxx.se>
>> Message-ID: <alpine.DEB.1.10.0902042227120.2...@yvahk2.pbagnpgbe.fr>
>> Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
>> On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote:
>> > I made a patch to add very basic Metalink support to curl command-line tool.
>> > The patch is made against curl-7.19.2. Although the patch is still rather
>> > experimental, it provides basic fail over functionality.
>> I would like to play around with this patch and functionality a bit, but I
>> have an initial request:
>> I'd like this patch to have as low impact as possible on the existing code so
>> I would really like to see perhaps a separate file added with the necessary
>> metalink support functions or similar, so that all devs and users who don't
>> care for metalink wouldn't have to see many traces of it.
>> --
>> / daniel.haxx.se
>> On Wed, Dec 17, 2008 at 6:55 AM, Tatsuhiro <tatsuhir...@gmail.com> wrote:
>> > Hmm, maybe other curl devs have not yet shared the idea about Metalink
>> > with Daniel.
>> > On 12月17日, 午前2:22, "Anthony Bryan" <anthonybr...@gmail.com> wrote:
>> >> On Mon, Dec 15, 2008 at 6:05 PM, Nicolas Alvarez
>> >> > Anthony Bryan wrote:
>> >> >> This is a neat patch, but IMHO it doesn't belong in the curl command line
>> >> >> client. It adds functionality that goes well beyond what curl is designed
>> >> >> for and the UNIX philosophy behind it: to do one thing and do it well,
>> >> >> which is to transfer individual URLs from/to individual sites
>> >> >> and to allow all manner of tweaking that behaviour. That's why it
>> >> >> doesn't do HTML parsing, remote URL globbing, recursive transfers,
>> >> >> PAC proxy configuration, etc.
>> >> >> Metalink support is designed for something other than what curl's primary
>> >> >> mission is about. An indication of this is that of curl's 128 options,
>> >> >> half become irrelevant (and even more are marginally relevant) when doing
>> >> >> Metalink transfers. Adding two additional library dependencies to curl
>> >> >> is also an issue.
>> >> > Can I say "I told you so" or will I get flamed?
>> >> no flames :)
>> >> it's kinda confusing when Daniel, curl author, writes
>> >> "The Metalink guys host a list of project ideas and one of those ideas
>> >> is to add metalink support to curl, and I recently bumped the stakes a
>> >> bit by raising the bounty with an additional 200 USD so that the offer
>> >> is now 500 USD for the person or team that brings the feature as
>> >> described.
>> >> My primary motivation for doing this is that I like the metalink idea
>> >> and I'd like to help making sure it gets used more widely."
>> >> --
>> >> (( Anthony Bryan ... Metalink [http://www.metalinker.org]
>> >> )) Easier, More Reliable, Self Healing Downloads
>> --
>> (( Anthony Bryan ... Metalink [http://www.metalinker.org]
>> )) Easier, More Reliable, Self Healing Downloads
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "Metalink Discussion" group.
> To post to this group, send email to metalink-discussion@googlegroups.com
> To unsubscribe from this group, send email to metalink-discussion+unsubscribe@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/metalink-discussion?hl=en > -~----------~----~----~----~------~----~------~--~---
-- (( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads
On Sunday, April 29, 2012 3:39:54 AM UTC-4, Anthony Bryan wrote:
> so, quite exciting news on the curl front...
> Tatsuhiro has revived his 3 year old patch & it's very nice to be able > to use metalink with curl!
> the discussion is happening on the curl-users list, & I don't think it > will make it into the next version, but hopefully the one after that.
> curl's metalink support uses libmetalink, which Tatsuhiro has just > applied a few fixes to. > I'm thinking it might be good to have meta4/RFC 5854 support in > libmetalink soon too
> On Fri, Feb 6, 2009 at 8:26 PM, Tatsuhiro <tatsuhir...@gmail.com> wrote: > > Good news!
> > Daniel Stenberg wrote: > >> On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote:
> >> I would like to play around with this patch and functionality a bit, > but I have an initial request:
> >> I'd like this patch to have as low impact as possible on the existing > code so I would really like to see perhaps a separate file added with the > necessary metalink support functions or similar, so that all devs and users > who don't care for metalink wouldn't have to see many traces of it.
> > OK. In my first patch, all metalink functions were integrated in > > main.c. I'll make new patch that call metalink function in another > > file after options are parsed.
> > Best regards,
> > Tatsuhiro Tsujikawa
> > On 2月6日, 午前1:19, Anthony Bryan <anthonybr...@gmail.com> wrote: > >> good news?
> >> Date: Wed, 4 Feb 2009 23:10:15 +0100 (CET) > >> From: Daniel Stenberg <dan...@haxx.se> > >> Subject: Re: Patch for Metalink Support > >> To: the curl tool <curl-us...@cool.haxx.se> > >> Message-ID: <alpine.DEB.1.10.0902042227120.2...@yvahk2.pbagnpgbe.fr> > >> Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
> >> On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote: > >> > I made a patch to add very basic Metalink support to curl > command-line tool. > >> > The patch is made against curl-7.19.2. Although the patch is still > rather > >> > experimental, it provides basic fail over functionality.
> >> I would like to play around with this patch and functionality a bit, > but I > >> have an initial request:
> >> I'd like this patch to have as low impact as possible on the existing > code so > >> I would really like to see perhaps a separate file added with the > necessary > >> metalink support functions or similar, so that all devs and users who > don't > >> care for metalink wouldn't have to see many traces of it.
> >> --
> >> / daniel.haxx.se
> >> On Wed, Dec 17, 2008 at 6:55 AM, Tatsuhiro <tatsuhir...@gmail.com> > wrote:
> >> > Hmm, maybe other curl devs have not yet shared the idea about > Metalink > >> > with Daniel.
> >> > On 12月17日, 午前2:22, "Anthony Bryan" <anthonybr...@gmail.com> wrote: > >> >> On Mon, Dec 15, 2008 at 6:05 PM, Nicolas Alvarez
> >> >> > Anthony Bryan wrote: > >> >> >> This is a neat patch, but IMHO it doesn't belong in the curl > command line > >> >> >> client. It adds functionality that goes well beyond what curl is > designed > >> >> >> for and the UNIX philosophy behind it: to do one thing and do it > well, > >> >> >> which is to transfer individual URLs from/to individual sites > >> >> >> and to allow all manner of tweaking that behaviour. That's why it > >> >> >> doesn't do HTML parsing, remote URL globbing, recursive > transfers, > >> >> >> PAC proxy configuration, etc.
> >> >> >> Metalink support is designed for something other than what curl's > primary > >> >> >> mission is about. An indication of this is that of curl's 128 > options, > >> >> >> half become irrelevant (and even more are marginally relevant) > when doing > >> >> >> Metalink transfers. Adding two additional library dependencies to > curl > >> >> >> is also an issue.
> >> >> > Can I say "I told you so" or will I get flamed?
> >> >> "The Metalink guys host a list of project ideas and one of those > ideas > >> >> is to add metalink support to curl, and I recently bumped the stakes > a > >> >> bit by raising the bounty with an additional 200 USD so that the > offer > >> >> is now 500 USD for the person or team that brings the feature as > >> >> described.
> >> >> My primary motivation for doing this is that I like the metalink > idea > >> >> and I'd like to help making sure it gets used more widely."
> >> -- > >> (( Anthony Bryan ... Metalink [http://www.metalinker.org] > >> )) Easier, More Reliable, Self Healing Downloads > > --~--~---------~--~----~------------~-------~--~----~ > > You received this message because you are subscribed to the Google > Groups "Metalink Discussion" group. > > To post to this group, send email to > metalink-discussion@googlegroups.com > > To unsubscribe from this group, send email to > metalink-discussion+unsubscribe@googlegroups.com > > For more options, visit this group at > http://groups.google.com/group/metalink-discussion?hl=en > > -~----------~----~----~----~------~----~------~--~---
> -- > (( Anthony Bryan ... Metalink [ http://www.metalinker.org ] > )) Easier, More Reliable, Self Healing Downloads
From: Daniel Stenberg <daniel_at_haxx.se<daniel_at_haxx.se?Subject=Re:%20after%207.26.0>
Date: Thu, 24 May 2012 18:28:13 +0200 (CEST)
Title: after 7.26.0
Hi friends
As we've now gotten rid of the release, we once gain open the flood gates for new features to enter. We will now allow new features getting in during the coming month.
I especially want to point out and work with the following features to get merged soon:
1 - the new auth callback for libcurl, by Joe Mason
2 - metalink support for curl, by Tatsuhiro Tsujikaw (note that this is not for libcurl)
3 - schannel support for libcurl, by Marc Hoersken
... but I'm not closing the door for others or other features, so please show us what you have as soon as possible if you want it get merged for the next release.
> On Sunday, April 29, 2012 3:39:54 AM UTC-4, Anthony Bryan wrote:
>> so, quite exciting news on the curl front...
>> Tatsuhiro has revived his 3 year old patch & it's very nice to be able >> to use metalink with curl!
>> the discussion is happening on the curl-users list, & I don't think it >> will make it into the next version, but hopefully the one after that.
>> curl's metalink support uses libmetalink, which Tatsuhiro has just >> applied a few fixes to. >> I'm thinking it might be good to have meta4/RFC 5854 support in >> libmetalink soon too
>> On Fri, Feb 6, 2009 at 8:26 PM, Tatsuhiro <tatsuhir...@gmail.com> wrote: >> > Good news!
>> > Daniel Stenberg wrote: >> >> On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote:
>> >> I would like to play around with this patch and functionality a bit, >> but I have an initial request:
>> >> I'd like this patch to have as low impact as possible on the existing >> code so I would really like to see perhaps a separate file added with the >> necessary metalink support functions or similar, so that all devs and users >> who don't care for metalink wouldn't have to see many traces of it.
>> > OK. In my first patch, all metalink functions were integrated in >> > main.c. I'll make new patch that call metalink function in another >> > file after options are parsed.
>> > Best regards,
>> > Tatsuhiro Tsujikawa
>> > On 2月6日, 午前1:19, Anthony Bryan <anthonybr...@gmail.com> wrote: >> >> good news?
>> >> Date: Wed, 4 Feb 2009 23:10:15 +0100 (CET) >> >> From: Daniel Stenberg <dan...@haxx.se> >> >> Subject: Re: Patch for Metalink Support >> >> To: the curl tool <curl-us...@cool.haxx.se> >> >> Message-ID: <alpine.DEB.1.10.0902042227120.2...@yvahk2.pbagnpgbe.fr> >> >> Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
>> >> On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote: >> >> > I made a patch to add very basic Metalink support to curl >> command-line tool. >> >> > The patch is made against curl-7.19.2. Although the patch is still >> rather >> >> > experimental, it provides basic fail over functionality.
>> >> I would like to play around with this patch and functionality a bit, >> but I >> >> have an initial request:
>> >> I'd like this patch to have as low impact as possible on the existing >> code so >> >> I would really like to see perhaps a separate file added with the >> necessary >> >> metalink support functions or similar, so that all devs and users who >> don't >> >> care for metalink wouldn't have to see many traces of it.
>> >> --
>> >> / daniel.haxx.se
>> >> On Wed, Dec 17, 2008 at 6:55 AM, Tatsuhiro <tatsuhir...@gmail.com> >> wrote:
>> >> > Hmm, maybe other curl devs have not yet shared the idea about >> Metalink >> >> > with Daniel.
>> >> > On 12月17日, 午前2:22, "Anthony Bryan" <anthonybr...@gmail.com> wrote: >> >> >> On Mon, Dec 15, 2008 at 6:05 PM, Nicolas Alvarez
>> >> >> > Anthony Bryan wrote: >> >> >> >> This is a neat patch, but IMHO it doesn't belong in the curl >> command line >> >> >> >> client. It adds functionality that goes well beyond what curl is >> designed >> >> >> >> for and the UNIX philosophy behind it: to do one thing and do it >> well, >> >> >> >> which is to transfer individual URLs from/to individual sites >> >> >> >> and to allow all manner of tweaking that behaviour. That's why >> it >> >> >> >> doesn't do HTML parsing, remote URL globbing, recursive >> transfers, >> >> >> >> PAC proxy configuration, etc.
>> >> >> >> Metalink support is designed for something other than what >> curl's primary >> >> >> >> mission is about. An indication of this is that of curl's 128 >> options, >> >> >> >> half become irrelevant (and even more are marginally relevant) >> when doing >> >> >> >> Metalink transfers. Adding two additional library dependencies >> to curl >> >> >> >> is also an issue.
>> >> >> > Can I say "I told you so" or will I get flamed?
>> >> >> "The Metalink guys host a list of project ideas and one of those >> ideas >> >> >> is to add metalink support to curl, and I recently bumped the >> stakes a >> >> >> bit by raising the bounty with an additional 200 USD so that the >> offer >> >> >> is now 500 USD for the person or team that brings the feature as >> >> >> described.
>> >> >> My primary motivation for doing this is that I like the metalink >> idea >> >> >> and I'd like to help making sure it gets used more widely."
>> >> -- >> >> (( Anthony Bryan ... Metalink [http://www.metalinker.org] >> >> )) Easier, More Reliable, Self Healing Downloads >> > --~--~---------~--~----~------------~-------~--~----~ >> > You received this message because you are subscribed to the Google >> Groups "Metalink Discussion" group. >> > To post to this group, send email to >> metalink-discussion@googlegroups.com >> > To unsubscribe from this group, send email to >> metalink-discussion+unsubscribe@googlegroups.com >> > For more options, visit this group at >> http://groups.google.com/group/metalink-discussion?hl=en >> > -~----------~----~----~----~------~----~------~--~---
>> -- >> (( Anthony Bryan ... Metalink [ http://www.metalinker.org ] >> )) Easier, More Reliable, Self Healing Downloads
metalink support is in mainline curl!
curl 7.26 was just released, so the next release should have it & be out in two months!
*Daniel Stenberg* @*bagder* <https://twitter.com/#!/bagder>
> From: Daniel Stenberg <daniel_at_haxx.se<daniel_at_haxx.se?Subject=Re:%20after%207.26.0>
> Date: Thu, 24 May 2012 18:28:13 +0200 (CEST)
> Title: after 7.26.0
> Hi friends
> As we've now gotten rid of the release, we once gain open the flood gates > for > new features to enter. We will now allow new features getting in during > the > coming month.
> I especially want to point out and work with the following features to get > merged soon:
> 1 - the new auth callback for libcurl, by Joe Mason
> 2 - metalink support for curl, by Tatsuhiro Tsujikaw (note that this is > not > for libcurl)
> 3 - schannel support for libcurl, by Marc Hoersken
> ... but I'm not closing the door for others or other features, so please > show > us what you have as soon as possible if you want it get merged for the > next > release.
>> On Sunday, April 29, 2012 3:39:54 AM UTC-4, Anthony Bryan wrote:
>>> so, quite exciting news on the curl front...
>>> Tatsuhiro has revived his 3 year old patch & it's very nice to be able >>> to use metalink with curl!
>>> the discussion is happening on the curl-users list, & I don't think it >>> will make it into the next version, but hopefully the one after that.
>>> curl's metalink support uses libmetalink, which Tatsuhiro has just >>> applied a few fixes to. >>> I'm thinking it might be good to have meta4/RFC 5854 support in >>> libmetalink soon too
>>> On Fri, Feb 6, 2009 at 8:26 PM, Tatsuhiro <tatsuhir...@gmail.com> >>> wrote: >>> > Good news!
>>> > Daniel Stenberg wrote: >>> >> On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote:
>>> >> I would like to play around with this patch and functionality a bit, >>> but I have an initial request:
>>> >> I'd like this patch to have as low impact as possible on the existing >>> code so I would really like to see perhaps a separate file added with the >>> necessary metalink support functions or similar, so that all devs and users >>> who don't care for metalink wouldn't have to see many traces of it.
>>> > OK. In my first patch, all metalink functions were integrated in >>> > main.c. I'll make new patch that call metalink function in another >>> > file after options are parsed.
>>> > Best regards,
>>> > Tatsuhiro Tsujikawa
>>> > On 2月6日, 午前1:19, Anthony Bryan <anthonybr...@gmail.com> wrote: >>> >> good news?
>>> >> Date: Wed, 4 Feb 2009 23:10:15 +0100 (CET) >>> >> From: Daniel Stenberg <dan...@haxx.se> >>> >> Subject: Re: Patch for Metalink Support >>> >> To: the curl tool <curl-us...@cool.haxx.se> >>> >> Message-ID: <alpine.DEB.1.10.0902042227120.2...@yvahk2.pbagnpgbe.fr> >>> >> Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
>>> >> On Sat, 13 Dec 2008, Tatsuhiro Tsujikawa wrote: >>> >> > I made a patch to add very basic Metalink support to curl >>> command-line tool. >>> >> > The patch is made against curl-7.19.2. Although the patch is still >>> rather >>> >> > experimental, it provides basic fail over functionality.
>>> >> I would like to play around with this patch and functionality a bit, >>> but I >>> >> have an initial request:
>>> >> I'd like this patch to have as low impact as possible on the existing >>> code so >>> >> I would really like to see perhaps a separate file added with the >>> necessary >>> >> metalink support functions or similar, so that all devs and users who >>> don't >>> >> care for metalink wouldn't have to see many traces of it.
>>> >> --
>>> >> / daniel.haxx.se
>>> >> On Wed, Dec 17, 2008 at 6:55 AM, Tatsuhiro <tatsuhir...@gmail.com> >>> wrote:
>>> >> > Hmm, maybe other curl devs have not yet shared the idea about >>> Metalink >>> >> > with Daniel.
>>> >> > On 12月17日, 午前2:22, "Anthony Bryan" <anthonybr...@gmail.com> wrote: >>> >> >> On Mon, Dec 15, 2008 at 6:05 PM, Nicolas Alvarez
>>> >> >> > Anthony Bryan wrote: >>> >> >> >> This is a neat patch, but IMHO it doesn't belong in the curl >>> command line >>> >> >> >> client. It adds functionality that goes well beyond what curl >>> is designed >>> >> >> >> for and the UNIX philosophy behind it: to do one thing and do >>> it well, >>> >> >> >> which is to transfer individual URLs from/to individual sites >>> >> >> >> and to allow all manner of tweaking that behaviour. That's why >>> it >>> >> >> >> doesn't do HTML parsing, remote URL globbing, recursive >>> transfers, >>> >> >> >> PAC proxy configuration, etc.
>>> >> >> >> Metalink support is designed for something other than what >>> curl's primary >>> >> >> >> mission is about. An indication of this is that of curl's 128 >>> options, >>> >> >> >> half become irrelevant (and even more are marginally relevant) >>> when doing >>> >> >> >> Metalink transfers. Adding two additional library dependencies >>> to curl >>> >> >> >> is also an issue.
>>> >> >> > Can I say "I told you so" or will I get flamed?
>>> >> >> "The Metalink guys host a list of project ideas and one of those >>> ideas >>> >> >> is to add metalink support to curl, and I recently bumped the >>> stakes a >>> >> >> bit by raising the bounty with an additional 200 USD so that the >>> offer >>> >> >> is now 500 USD for the person or team that brings the feature as >>> >> >> described.
>>> >> >> My primary motivation for doing this is that I like the metalink >>> idea >>> >> >> and I'd like to help making sure it gets used more widely."
>>> >> -- >>> >> (( Anthony Bryan ... Metalink [http://www.metalinker.org] >>> >> )) Easier, More Reliable, Self Healing Downloads >>> > --~--~---------~--~----~------------~-------~--~----~ >>> > You received this message because you are subscribed to the Google >>> Groups "Metalink Discussion" group. >>> > To post to this group, send email to >>> metalink-discussion@googlegroups.com >>> > To unsubscribe from this group, send email to >>> metalink-discussion+unsubscribe@googlegroups.com >>> > For more options, visit this group at >>> http://groups.google.com/group/metalink-discussion?hl=en >>> > -~----------~----~----~----~------~----~------~--~---
>>> -- >>> (( Anthony Bryan ... Metalink [ http://www.metalinker.org ] >>> )) Easier, More Reliable, Self Healing Downloads
On Tue, May 29, 2012 at 9:20 PM, Anthony Bryan <anthonybr...@gmail.com> wrote:
> metalink support is in mainline curl!
> curl 7.26 was just released, so the next release should have it & be out in
> two months!
> Daniel Stenberg @bagder
> at 23:24, I committed the 21 changes to #curl that introduced#metalink
> support
Hurry on over to http://curl.haxx.se/download.html and download the fresh
release! This time with new features and two new SSL libraries supported.
Curl and libcurl 7.27.0
Public curl releases: 128
Command line options: 152
curl_easy_setopt() options: 199
Public functions in libcurl: 58
Known libcurl bindings: 39
Contributors: 953
This release includes the following changes:
o nss: use human-readable error messages provided by NSS
o added --metalink for metalink download support [5]
o pop3: Added support for sasl plain text authentication
o pop3: Added support for sasl login authentication
o pop3: Added support for sasl ntlm authentication
o pop3: Added support for sasl cram-md5 authentication
o pop3: Added support for sasl digest-md5 authentication
o pop3: Added support for apop authentication
o Added support for Schannel (Native Windows) SSL/TLS encryption [2]
o Added support for Darwin SSL (Native Mac OS X and iOS) [6]
o http: print reason phrase from HTTP status line on error [8]
This release includes the following bugfixes:
o pop3: Fixed the issue of having to supply the user name for all requests
o configure: fix LDAPS disabling related misplaced closing parenthesis
o cmdline: made -D option work with -O and -J
o configure: Fix libcurl.pc and curl-config generation for static MingW*
cross builds
o ssl: fix duplicated SSL handshake with multi interface and proxy [1]
o winbuild: Fix Makefile.vc ignoring USE_IPV6 and USE_IDN flags
o OpenSSL: support longer certificate subject names [3]
o openldap: OOM fixes
o log2changes.pl: fix the Version output
o lib554.c: use curl_formadd() properly [4]
o urldata.h: fix cyassl build clash with wincrypt.h
o cookies: changed the URL in the cookiejar headers [7]
o http-proxy: keep CONNECT connections alive (for NTLM)
o NTLM SSPI: fixed to work with unicode user names and passwords
o OOM fix in the curl tool when cloning cmdline options
o fixed some examples to use curl_global_init() properly
o cmdline: stricter numerical option parser
o HTTP HEAD: don't force-close after response-headers
o test231: fix wrong -C use
o docs: switch to proper UTF-8 for text file encoding
o keepalive: DragonFly uses milliseconds [9]
o HTTP Digest: Client's "qop" value should not be quoted
o make distclean works again
This release would not have looked like this without help, code, reports and
advice from friends like these:
Anthony Bryan, Guenter Knauf, Kamil Dudka, Steve Holme, Tatsuhiro Tsujikawa,
Yang Tse, Gokhan Sengun, Marc Hoersken, Ghennadi Procopciuc, Gisle Vanem,
Mark Salisbury, Anthony G. Basile, Enrico Scholz, Robert B. Harris,
Neil Bowers, Marcel Raad, Christian H?gele, Philip Craig, Nick Zitzmann,
Eelco Dolstra, Anton Yabchinskiy, Santhana Todatry, John Marino
Thanks! (and sorry if I forgot to mention someone)
References to bug reports and discussions on issues:
Tatsuhiro's explanation & help compiling curl w/ metalink support:
Curl 7.27.0 was out few days ago. This release is important to us because it is the first public curl release which supports Metalink. http://en.wikipedia.org/wiki/Metalink Thank you to curl community, especially Daniel Stenberg who accepted Metalink patch, Yang Tse who quickly merged our patch and provided security enhancement ideas and many test cases for Metalink feature and Guenter Knauf who provided many libmetalink enhancement patches which fixes many portability issues.
Since Metalink feature in curl is optional at the moment, I'll write a simple tutorial to get it work. First grab libmetalink-0.1.0 tar ball. https://launchpad.net/libmetalink/trunk/libmetalink-0.1.0/+download/l... (Please note that libmetalink-0.0.3 does not work with curl). Libmetalink depends on Expat XML library. If you are using Ubuntu, install libexpat-dev package. Unpack libmetalink tar ball and configure and make install.
Probably one of most appealing point of Metalink is automatic checksum validation. To do this, curl must be built with OpenSSL or GnuTLS. For OpenSSL, install libssl-dev.
After libmetalink was installed, download curl 7.27.0 source tar ball. Unpack it and run configure script with --with-libmetalink option. Make sure that metalink support is enabled. If it is enabled, you can see the following line in the output of configure script:
metalink support: enabled
Then run make to build curl tool binary.
To download a file using Metalink feature, provide URI to Metalink file. Curl download page offers several Metalink downloads. Let's use it as an example. Use --metalink option to enable Metalink feature.
OK, I'll explain what happened behind the scene. First, curl downloaded metalink XML file pointed by URI ' http://curl.haxx.se/metalink.cgi?curl=tar.bz2'. The content of XML file was not saved on the disk. It was parsed on the fly. After the XML file was downloaded and parsed, curl retrieves mirror URIs from the parsed result and download the file pointed by it. In this example, the Metalink XML file contains 12 URIs. These URIs are used as fallback. As a first attempt, curl used http://curl.freeby.pctools.cl/download/curl-7.27.0.tar.bz2 to download curl 7.27.0 tar ball. Unlike normal curl behavior, curl saves the file using the filename described in Metalink XML file. In this example, the filename is "curl-7.27.0.tar.bz2". -O option is not necessary. After download completed, curl automatically calculated checksum of the download file and compared it with the correct one described in Metalink XML file and surely they were matched.
> Hurry on over to http://curl.haxx.se/download.html and download the fresh > release! This time with new features and two new SSL libraries supported.
> Curl and libcurl 7.27.0
> This release includes the following changes:
> o nss: use human-readable error messages provided by NSS > o added --metalink for metalink download support [5]
