Any plans to support "Keep-Alive" in http.tcl package ??
bb
http::geturl, to support "Keep_alive". This means that after setting
up the initial TCP connection to the web server all subsequent HTTP
GETs will go over that connection. Currently, the package only
supports basic HTTP 1.0 functionality so that for every GET a new TCP
connection is established.
The problem with doing this is that I will need to insure that my
version of http.tcl is installed on any Linux box I want to use.
Is anyone aware of:
1. Whether this has already been done?
2. Will http.tcl ever support HTTP 1.0 Keep-Alive?
3. Will http.tcl ever support HTTP 1.1?
Thanks.
bb
Pat Thoyts
There has been a lot of discussion about this during the past
week. The current result of this is summarized at
http://tclsoap.sf.net/http.html
In short - we have done some work on KeepAlive and a certain
amount of HTTP/1.1 support. More needs to be done.
--
Pat Thoyts http://www.zsplat.freeserve.co.uk/resume.html
To reply, rot13 the return address or read the X-Address header.
PGP fingerprint 2C 6E 98 07 2C 59 C8 97 10 CE 11 E6 04 E0 B9 DD
lvi...@yahoo.com
According to bb <bbea...@desanasystems.com>:
:Is anyone aware of:
:
:1. Whether this has already been done?
yes, several people are aware of this. If you visit http://groups.google.com
and search the past 2 weeks or so of comp.lang.tcl for http keep alive,
you should find pointers to a patch to provide this functionality.
--
"I know of vanishingly few people ... who choose to use ksh." "I'm a minority!"
<URL: mailto:lvi...@cas.org> <URL: http://www.purl.org/NET/lvirden/>
Even if explicitly stated to the contrary, nothing in this posting
should be construed as representing my employer's opinions.
Phil Ehrens
more on TclCurl, which offers a great deal more bang for the
buck.
Pat Thoyts
>As cute as the Tcl HTTP package is, I wish people would work
>more on TclCurl, which offers a great deal more bang for the
>buck.
>
So promote it a bit. Why it it nice to use? Perhaps an example or two
showing how simple it makes complex tasks? I'm sure you're right but
I'm also sure I may never get around to looking at the package unless
I'm shown it's going to be worth my while.
Of course, now I _do_ think it's likely to be worth my
while. Still...
Phil Ehrens
for the package, it's so darn sweet it'll give you toothache.
You want more? Go fish. I'm having too much fun on
alt.collecting.pens-pencils, where I feel more appreciated.
(QOTW, Cameron?)
Phil
Marc Spitzer
> Well, once you get past the badly written configure/make
> for the package, it's so darn sweet it'll give you toothache.
>
> You want more? Go fish. I'm having too much fun on
> alt.collecting.pens-pencils, where I feel more appreciated.
>
> (QOTW, Cameron?)
>
> Phil
>
Well in your previous post(below) you made a sales pitch, tclcurl
is better then tcl http. You had at least one person interested
enough to ask you to provide enough info to close the sale, why
did you tell him to close the sale himself? It generally does
not work. You could have had your first sale on tclcurl and
then he could have also been saying how great tclcurl is and he
could convert his friends ...
marc
Andres Garcia
> showing how simple it makes complex tasks? I'm sure you're right but
> I'm also sure I may never get around to looking at the package unless
> I'm shown it's going to be worth my while.
Since I am the mantainer of TclCurl I will try do to that:
- Imagine you want to automate posting a form:
curl::transfer -url 127.0.0.1/cgi-bin/post1.tcl \
-httppost [list name "firstName" contents "Andres" contenttype text/plain"] \
-httppost [list name "lastName" contents "Garcia"] \
-httppost [list name "file" file "data.txt" contenttype text/plain"] \
-httppost [list name "submit" contents "send"]
- Or you need cookies:
curl::transfer -url 127.0.0.1 \
-cookiefile "/home/Andres/TclCurl/cookies.txt"
TclCurl, or rather libcurl, will take care of keeping a Netscape style
cookie file, putting all the cookies it receives there and using them
when needed. You can also use the real Netscape file if it already has
the coookies you need.
- It supports http 1.1, so if you want to resume downloading a file where
it was interrupted:
curl::transfer -url http://www.something.con/anything.tgz \
-resumefrom 5000
- If you have a ssl-enabled libcurl, you can use https as easily as http, I
don't have an example handy though.
- You want ftp:
curl::transfer -file tcl8.4a3.tar.gz \
-url ftp://ftp.etsimo.uniovi.es/pub/tcl/tcl8_4/tcl8.4a3.tar.gz \
- If you need to send your own headers:
curl::transfer -url "127.0.0.1" -verbose 1 \
-httpheader [list "hola: hello" "adios: goodbye"]
- If you want TclCurl to follow redirections on its own:
curl::transfer -url 127.0.0.1 -followlocation 1
- And certainly, libcurl supports keep-alive
One feature that the http package has and TclCurl lacks, and
it is a real showstopper, is non-blocking connects, but there
is some code in libcurl's cvs that I hope will solve this.
Well, if you would like to take a look, you can find it at:
http://personal1.iddeo.es/andresgarci/tclcurl/english/
where you will find a feature list.
Andres
Cameron Laird
Andres Garcia <fan...@retemail.es> wrote:
>Pat Thoyts wrote in message
>> So promote it a bit. Why it it nice to use? Perhaps an example or two
>> showing how simple it makes complex tasks? I'm sure you're right but
>> I'm also sure I may never get around to looking at the package unless
>> I'm shown it's going to be worth my while.
>
>Since I am the mantainer of TclCurl I will try do to that:
>
> - Imagine you want to automate posting a form:
.
.
> TclCurl, or rather libcurl, will take care of keeping a Netscape style
> cookie file, putting all the cookies it receives there and using them
> when needed. You can also use the real Netscape file if it already has
> the coookies you need.
>
> - It supports http 1.1, so if you want to resume downloading a file where
> it was interrupted:
.
.
Now you've got me interested. Do you have any
sense of how easy TclCurl make it to automate
such a task as mass-download of a HotMail folder?
That's a concrete example of a broadly-useful
category--'takes some cookies, a bit of POSTing,
Expect-style result pattern-matching, ...
--
Cameron Laird <Cam...@Lairds.com>
Business: http://www.Phaseit.net
Personal: http://starbase.neosoft.com/~claird/home.html
Jochem Huhmann
> Now you've got me interested. Do you have any
> sense of how easy TclCurl make it to automate
> such a task as mass-download of a HotMail folder?
> That's a concrete example of a broadly-useful
> category--'takes some cookies, a bit of POSTing,
> Expect-style result pattern-matching, ...
I'm doing such right now and TclCurl is nice to have. I think that some
framework for such things would be a great application for Tcl. Also for
Tk, since it is easy then to wrap a nice and cross-platform GUI client
around "Web Applications".
The most ugly part is the HTML parsing. Pattern-matching is a PITA, it
would be much nicer to turn the HTML into some valid XML and use Tcl-DOM
(or tDOM) to extract the information out of it.
Jochem
--
This message is printed on recycled electrons.
Rolf Ade
Jochem Huhmann <j...@gmx.net> wrote:
>
>The most ugly part is the HTML parsing. Pattern-matching is a PITA, it
>would be much nicer to turn the HTML into some valid XML and use Tcl-DOM
>(or tDOM) to extract the information out of it.
The new tDOM release (which for sure will be there some day in the
future ;-)) includes an HTML reader. With that, you don't have to turn
the HTML into wellformed XML. Just feed the HTML into the parser and
you will get a DOM tree, which you can query with xpath
expressions. Jochen showed exactly this (among others) in his talk at
the second European Tcl/Tk User Meeting, this summer.
The HTML reader is already included in the (a bit outdated)
pre-release snapshot tDOM-0.64alpha.tar.bz at
http://groups.yahoo.com/group/tdom
rolf
Jochem Huhmann
> The new tDOM release (which for sure will be there some day in the
> future ;-)) includes an HTML reader. With that, you don't have to turn
> the HTML into wellformed XML. Just feed the HTML into the parser and
> you will get a DOM tree, which you can query with xpath
> expressions. Jochen showed exactly this (among others) in his talk at
> the second European Tcl/Tk User Meeting, this summer.
Great, thanks!
> The HTML reader is already included in the (a bit outdated)
> pre-release snapshot tDOM-0.64alpha.tar.bz at
> http://groups.yahoo.com/group/tdom
Even better ;-)
Jochem
--
"Unless, of course, there are better solutions. Anyone have one?" --
Michael Hill
Andres Garcia
> Now you've got me interested. Do you have any
> sense of how easy TclCurl make it to automate
> such a task as mass-download of a HotMail folder?
> That's a concrete example of a broadly-useful
> category--'takes some cookies, a bit of POSTing,
> Expect-style result pattern-matching, ...
This may sound silly but actually I have never used
TclCurl for making a POST request and I don't have
a hotmail account, so I can only guess.
I am certain though that it can be done as there
are a couple of scripts that use curl in order to
read Hotmail.
Anyway, it would be something like this.
Since we are going to download several pages from the
same server we'd better use different commands than
in my simpler examples, this other way we will
use persistant connections, so:
set curlHandle [curl::init]
Now we have to configure the first transfer:
$curlHandle configure -url http://www.hotmail.com/login.cgi \
-ccokiefile hotCookies.txt -post 1 \
-postfields $dataToPost \
-useragent "Mozilla/4.0 (compatible; MSIE 4.01; Windows 95)" \
-followlocation 1 -file mail.html
And we perform the transfer
$curlHandle perform
Except for the posting data which is reset betwen transfers the rest of
the options will keep, so it is probably a good idea to change at least two
of them now:
$curlHanlde configure -post 0 -followlocation 0
After parsing the file to get the links for the mail, you download them
one by one:
$curlHandle configure -url http://www.hotmail.com/firstMail.html \
-file firstMail.html
$curlHandle perform
$curlHandle configure -url http://www.hotmail.com/secondMail.html \
-file secondMail.html
$curlHandle perform
.
.
.
Instead of putting them in file you could give the name of variable,
to show them in a Tk window or whatever, this works even if they
include binary data.
When you have all of them you close the connection:
$curlHandle cleanup
An explanation of the first configure:
The url would be login cgi used by the site, you would need to look it
up in the login page.
The cookiefile is where all cookies sent by Hotmail will be stored,
as I said, all the other performs will use this file until you
change the option explicitly.
The posting method is different from the one in the other example,
because that one is for multipart/formdata post, while this is for the
application/x-www-form-urlencoded kind which is more common.
The dataToPost would b e a string with the data encoded for
posting, you can encode using the curl::escape command.
The user agent option is used in case hotmail is picky with clients,
I don't know if it really is.
I guess the login script will redirect you to a page with your mail
so I added the followlocation option
And the file tells where the save the page.
TclCurl won't parse the HTML, I have a program to download
complete web sites for which I did a parser using regexp, it
is tied to that program but it wouldn't be hard to change it for
this as long as Hotmail doesn't use Javascript or something
such.
I never did a program that worked first time and this is
more guessing than anything, but it shouldn't be too
different.
There is one thing though, TclCurl is still alpha, it seems
to work quite well, but that is likely to be because it hasn't
been used very much yet.
Andres
lvi...@yahoo.com
According to Cameron Laird <cla...@starbase.neosoft.com>:
:sense of how easy [...] make it to automate
:such a task as mass-download of a HotMail folder?
:That's a concrete example of a broadly-useful
:category--'takes some cookies, a bit of POSTing,
:Expect-style result pattern-matching, ...
Or maybe an extension that would generalize use of search engines similar
to how Perl has the WWW::Search::* routines... one can just use a simple
command line and search various engines with the same basic command line...
Michael A. Cleverly
> According to Cameron Laird <cla...@starbase.neosoft.com>:
> :sense of how easy [...] make it to automate
> :such a task as mass-download of a HotMail folder?
> :That's a concrete example of a broadly-useful
> :category--'takes some cookies, a bit of POSTing,
> :Expect-style result pattern-matching, ...
>
> Or maybe an extension that would generalize use of search engines similar
> to how Perl has the WWW::Search::* routines... one can just use a simple
> command line and search various engines with the same basic command line...
I made a "websearch" type package my New Years Day project. I've put up a
page on the Wiki with the code:
I used the http package, not TclCurl. In this case, I guess a C
implementation might buy a few ms of speed, but the "hard" part isn't
fetching the html from the web, but rather, scraping the data out of it.
(And that's inheirently fragile, since search engines periodically
change their html layout.)
Cameron's other example (downloading a hotmail folder) ought to be the
same kind of thing; TclCurl, with out-of-the-box cookie support probably
has a slight leg up on the http package, but most of the hard work would
be in parsing MS's html, and keeping that current as it changes.
Michael
Cameron Laird
Nice work, Michael.
I'll ask again of the Curl fans: what are the specific
advantages? Is it the cookie support? Is there something
more than that?
lvi...@yahoo.com
Cameron, I'm not greatly familar with the package. It's my understanding,
from reading the bits and pieces, that curl provides for others what
Tcl provides via http and a few of the other modules. Not a lot more.
It's one 'claim to fame' is that the W3C is pushing on the Curl library
to become a 'standard' in a manner similar to what they were doing with
libwww back in the old days...
Andres Garcia
> I'll ask again of the Curl fans: what are the specific
> advantages? Is it the cookie support? Is there something
> more than that?
Let's see, the speed of a C extension is certainly not an
advantage, no matter how slow a program is, it will be
fast as lighting compared to the Internet, besides, if we
were in a hurry we wouldn't be using a scripting language.
So the advantage of TclCurl over the http package is
that it includes several features that the http package
lacks, as usual, none of these features is important except
when you need them.
So, if you insist on a list of features you can check them here:
http://personal1.iddeo.es/andresgarci/tclcurl/english/
you are likely to know more about the pure Tcl clients that I do,
so you can see for yourself if there is any advantage at all.
On the other hand, Larry mentioned that the w3c is pushing
the use of libcurl, this is the first time I hear anything about
them using or even considering using libcurl over libwww. So,
are you sure?
Andres
Pat Thoyts
>> I'll ask again of the Curl fans: what are the specific
>> advantages? Is it the cookie support? Is there something
>> more than that?
>
>Let's see, the speed of a C extension is certainly not an
>advantage, no matter how slow a program is, it will be
>fast as lighting compared to the Internet, besides, if we
>were in a hurry we wouldn't be using a scripting language.
>
[snip]
My feeling here is that TclCurl looks extremely useful for doing
internet work. However as it is a C extension and various categories
of Tcl user are unable or unwilling to add in external C extensions I
think it is still worth enhancing the Tcl-only http package as has
previously been discussed.
Basically if you need cookie support or keepalive or HTTP/1.1 support
_now_ then TclCurl provides this. I see this a bit like tcllib's
base64 module. If I have the Trf package then I'll use Trf's base64
command otherwise I'll default to tcllib's version. I figure to do the
same for TclCurl.
I also plan to carry on (when I find the time) looking at extending
the Tcl http package towards HTTP/1.1 support. I'm going to look at
providing TclCurl support within TclSOAP's HTTP transport too.
lvi...@yahoo.com
According to Andres Garcia <fan...@spam.retemail.es>:
:On the other hand, Larry mentioned that the w3c is pushing
:the use of libcurl, this is the first time I hear anything about
:them using or even considering using libcurl over libwww. So,
:are you sure?
Perhaps I misunderstood statements i have read over the past few months.
However, comments such as
<URL: http://www.google.com/search?q=cache:5pOhlnloRz4C:lists.w3.org/Archives/Public/www-patentpolicy-comment/2001Oct/1139.html+curl&hl=en&start=3 >
lead me to believe that work with people at the heart of Curl appear to
be working with advancing this technology in the web. I seem to recall
reading something at slashdot/wired/cnet/computerworld or elsewhere that
gave me some kind of 'consortium' endorsement .
Wow. After writing the above, I did more research. And I'm really confused.
I read: <URL: http://www.cag.lcs.mit.edu/curl/ > and what I read about is
a 'programming language' for writing web pages. How does this correspond
to TclCurl?
David Bleicher
>
> Wow. After writing the above, I did more research. And I'm really
confused.
> I read: <URL: http://www.cag.lcs.mit.edu/curl/ > and what I read about is
> a 'programming language' for writing web pages. How does this correspond
> to TclCurl?
>
It doesn't. TclCurl is a TCL wrapper around libcurl which, in turn, is a
subproject of "cURL" which can be found at http://curl.haxx.se/ "cURL's"
purpose in life is to simplify file transfer via URL syntax for a variety of
protocols (e.g. HTTP, FTP, Gopher, etc.). The "cURL" program is a command
line application to do just that. "libcurl" is the C/C++ library for
programmatic access to the functions of cURL, and is the basis for the
command line program. TclCurl is the TCL extension that allows TCL to
access libcurl. (Thanks Andres!)
The article you reference above refers to work done at MIT on a completely
unrelated topic that just happens to have the same name. As mentioned in
the link you found (and at the commercial site www.curl.com) the MIT
research work was commercialized by a bunch of the MIT folk who founded Curl
Corporation to sell it. You can think of "Curl" (the commercial language)
as another scripting language in a vein similar TCL itself (although not as
nice IMHO :). The two initiatives (cURL and Curl) have nothing in common
other than the spelling of their respective names.
cURL is a handy command line program. I use it frequently when I need to
quickly transfer a large number of (especially sequential) files via HTTP or
FTP. It has nice features for resume, stats, and progress display. It
handles SSL and certificate tasks (via OpenSSL) very nicely, and I used it
even more frequently before I got my hands on the "tls" extension for TCL.
TclCurl is also handy. By combining libcurl with TCL, TclCurl provides a
very friendly, "high level" set of interfaces to a range of protocols.
David
---
--
-