ANNOUNCE: www 2.0 released

[Schelte]

www is a package for performing http requests. It is intended to be used
in situations where the http package is too cumbersome to use (which is
pretty much always) and tclcurl is overkill.

The basic way to obtain a web resource using the www package is as follows:

if {[catch {www get http://www.tcl.tk/} data info]} {
puts stderr "Failed to get the web page"
} else {
puts $data
}

The package takes care of redirects, retries, cookies, and proxies and
only uses the standard way of throwing errors. The command returns the
document body. Metadata is available through return options.

The package also provides extensions to handle WebSockets and HTTP/2.
For HTTP/2 over TLS a patch is needed for TclTLS. I have filed a ticket
with the patch to the TclTLS tracker
(https://core.tcl-lang.org/tcltls/tktview?name=e1f9a21c67). Hopefully it
will be picked up soon.

For more information on www, please visit the chiselapp repo at
https://chiselapp.com/user/schelte/repository/www


Enjoy,
Schelte.

[Dave]

All the symbolic links are playing havoc on windows.

Using msys2 with the TAR file, I get a bunch of errors when doing 'tar
-zxf':
tar: www/www-2.0.tm: Cannot create symlink to 'www.tcl': No such file or
directory

Using 7z with the TAR file I get the errors:
Can not create symbolic link : Acess is denied. :
C:\users\<user>\AppData\Local\Temp\7zE069A5F1\www\www-2.0.tm

Using 7z with the ZIP file creates a normal file with the symlink target
as its contents.

[Schelte]

On 07/09/2021 23:17, Dave wrote:
> All the symbolic links are playing havoc on windows.
>

Well, that's annoying. Then I'm unsure how to handle Tcl modules in a
VCS. Each version of a TM will have a different file name. If that means
you also have to add a new copy of the file to the repository for each
version, then there's not much version control going on. What's the
suggested way to handle this?


Schelte.

[Dave]

Windows does support both hard (must be on same drive letter) or
symlinks (win7 requires admin privilege, don't know about win10, etc.).

Possibly an install script of some sort? With that you could avoid
multiple copies of the same file in the fossil repository. The script
could be very simple since, for www, there are not that many files
affected and the extra disk space is irrelevant.


--
computerjock AT mail DOT com

[Rolf Ade]

Schelte <nos...@wanadoo.nl> writes:
> www is a package for performing http requests. It is intended to be
> used in situations where the http package is too cumbersome to use
> (which is pretty much always) and tclcurl is overkill.

I wholeheartedly agree with "the http package is cumbersome to use".

> [...]

> For more information on www, please visit the chiselapp repo at
> https://chiselapp.com/user/schelte/repository/www

Great read. Will give it a try. It may be the answer.

> Enjoy,

I hope so.

> Schelte.

rolf

[Harald Oehlmann]

Hi Schelte,

I appreciate, thank you !
Great that you even care about DNS block by delegating it to a different
thread.
It was always an aim by Reinhard and myself make this non-blocking in
the core... So much to do.... Do you support iocp by Ashok ?

Thanks again,
Harald

[Harald Oehlmann]

Hi Schelte,
I would also appreciate, if Ashoks TWAPI would be usable for the TLS part.
Thanks,
Harald

[Schelte]

On 07/10/2021 15:44, Harald Oehlmann wrote:
>> Do you support iocp by Ashok ?
>>

> I would also appreciate, if Ashoks TWAPI would be usable for the TLS part.

Both of the packages you inquire about are Windows only. I don't use
Windows myself, so I can't say for sure if they will work in combination
with the www package.

The www package provides a way to register a command for implementing a
scheme. That command is expected to apply a channel transformation on
top of a socket after it has been opened. Reading the documentation, it
seems that the iocp_inet package replaces the standard Tcl socket
command. That seems like it could work quite seamlessly. The first
problem is loading the package in the helper thread. There currently is
no support for doing something like that. I also wonder if transferring
an iocp channel between threads has ever been tested.

The TWAPI documentation makes me believe that the twapi_crypto starttls
command should be usable for doing https with TWAPI. That should only
require a short proc that is installed using `www register`.


Schelte.

[Harald Oehlmann]

Sorry for the questions. I was not aware that you don't use Windows.

By the way, I don't understand the usage as backround command.
It means, that a:
catch {www get ...}
will call vwait and events are executed, right ?

Great package and thank you,
Harald

[Schelte]

On 07/10/2021 18:52, Harald Oehlmann wrote:
> By the way, I don't understand the usage as backround command.
> It means, that a:
> catch {www get ...}
> will call vwait and events are executed, right ?
>

That is correct. But that is really just a safety net for when the
command is invoked from direct code. The suggested way is to perform www
calls in a coroutine.


Schelte.

[Harald Oehlmann]

Dear Schelte,
thank you for the valuable answer.

Thanks,
Harald

[Ashok]

On 10/7/2021 9:20 PM, Schelte wrote:
> The www package provides a way to register a command for implementing a
> scheme. That command is expected to apply a channel transformation on
> top of a socket after it has been opened. Reading the documentation, it
> seems that the iocp_inet package replaces the standard Tcl socket
> command. That seems like it could work quite seamlessly. The first
> problem is loading the package in the helper thread. There currently is
> no support for doing something like that. I also wonder if transferring
> an iocp channel between threads has ever been tested.

iocp passes the entire socket test suite, the http test suite as well as
the tcltls tests. However, the only thread transfer test in those is
socket-13.1 so it is a bit light (but no lighter than ::socket :-) to
say the least. If I get some time, I'll add the www test suite to the
iocp tests as well.

>
> The TWAPI documentation makes me believe that the twapi_crypto starttls
> command should be usable for doing https with TWAPI. That should only
> require a short proc that is installed using `www register`.
>
>
> Schelte.

I think that should work "in principle" but there is a difference
between twapi::tls, which validates against the Windows certificate
store versus tcltls which uses a specified file/dir path. The
corresponding options may need some tweaking. Again, I'll try if I get
some time on the weekend.

BTW, are you the same Schelte as sbron on the wiki?

/Ashok

[Schelte]

On 08/10/2021 09:44, Ashok wrote:
> I'll try if I get some time on the weekend.
>

Thanks. It will be interesting to learn what you find.

> BTW, are you the same Schelte as sbron on the wiki?

Yes, that's me.


Schelte.

[Ashok]

On 10/8/2021 4:26 PM, Schelte wrote:
> On 08/10/2021 09:44, Ashok wrote:
>> I'll try if I get some time on the weekend.
>>
> Thanks. It will be interesting to learn what you find.
>

Well, did not get very far I'm afraid (using iocp in lieu of Tcl sockets
as well as TLS). Aside from the file link issues on Windows, I was not
able to get the register command to work at all. I think it is not being
passed to the helper thread but did not have time to look closely.

>> BTW, are you the same Schelte as sbron on the wiki?
>
> Yes, that's me.
>

I was asking because coincidentally I ran into your version of
getopt-like packages on the Wiki while looking for that class of
functionality. Very nicely done, full featured, expected behaviors
supported and a very intuitive switch-like Tcl-ish interface. My pick
for Editor's choice :-)

/Ashok

[Ashok]

On 9/8/2021 5:13 AM, Rolf Ade wrote:
>
> Schelte <nos...@wanadoo.nl> writes:
>> www is a package for performing http requests. It is intended to be
>> used in situations where the http package is too cumbersome to use
>> (which is pretty much always) and tclcurl is overkill.
>
> I wholeheartedly agree with "the http package is cumbersome to use".
>

I understand the comment about the built-in http package being awkward /
cumbersome for many use cases, mostly historical hold overs, but I have
to point out a commonly overlooked strength - its test suite in its
current incarnation is orders of magnitude more comprehensive than any
of the competing http clients. I *think* it was Keith Nash (apologies if
I'm wrong) who updated it in the recent past and clearly a lot of effort
went into it. The quality of the test suite is a major consideration for me.

/Ashok

[Schelte]

On 18/10/2021 17:45, Ashok wrote:
> I understand the comment about the built-in http package being awkward /
> cumbersome for many use cases, mostly historical hold overs, but I have
> to point out a commonly overlooked strength - its test suite in its
> current incarnation is orders of magnitude more comprehensive than any
> of the competing http clients.

As far as I could quickly tell, there are about twice as many tests for
the http package as there are for the www package. Not exactly "orders
of magnitude". But please let me know if there is anything you feel is
missing from the test suite of the www package.

Despite the more extensive test suite of the http package, it still has
a number of bugs/issues. Which is one of the reasons I chose to write
the www package from scratch, rather than build it on top of the http
package.


Schelte.

[Schelte]

On 18/10/2021 17:38, Ashok wrote:
> Well, did not get very far I'm afraid (using iocp in lieu of Tcl sockets
> as well as TLS). Aside from the file link issues on Windows, I was not
> able to get the register command to work at all. I think it is not being
> passed to the helper thread but did not have time to look closely.
>

As I mentioned in an earlier post: "The www package provides a way to

register a command for implementing a scheme. That command is expected
to apply a channel transformation on top of a socket after it has been

opened." So this mechanism cannot be used in its current form to open a
socket using a command other than [socket]. Which is why I also
remarked: "The first problem is loading the package in the helper

thread. There currently is no support for doing something like that."

So for now, probably the only way to test the package with iocp would be
to modify the source code to load the package in the helper thread.

I hope I have addressed the file link issue for the base www package.
The same still needs to be done for the sub-packages.

> I was asking because coincidentally I ran into your version of
> getopt-like packages on the Wiki while looking for that class of
> functionality. Very nicely done, full featured, expected behaviors
> supported and a very intuitive switch-like Tcl-ish interface. My pick
> for Editor's choice :-)
>

Thanks. I use it myself in most applications I make.


Schelte.

[Ashok]

On 10/19/2021 3:42 AM, Schelte wrote:
> As far as I could quickly tell, there are about twice as many tests for
> the http package as there are for the www package. Not exactly "orders
> of magnitude". But please let me know if there is anything you feel is
> missing from the test suite of the www package.

Just to be clear, I certainly did not mean to denigrate other http
implementations with respect to testing or quality. That was not my
intent. Rather I meant to highlight at least one positive aspect of the
http package.

The Tcl http suite contains over 2000 tests. You might not have noticed
because they are spread over multiple test files and will not run
without multiple constraints enabled (and will take more than an hour to
run). But more than just the number of tests my use of "comprehensive"
related not just to tests but to number of scenarios covered given the
number of HTTP option *combinations* possible - methods(get,post
etc.)*keepalive*pipelining*encoding*contentlength*deflate|gzip|uncompressed*chunking
variations*mime types*HTTP codes*error conditions. That's an impossible
number to test! Perhaps a look at http.test http11.test and
httpPipeline.test might help identify missing (if any) test cases for
www and whether they are worth adding/spending time on.

>
> Despite the more extensive test suite of the http package, it still has
> a number of bugs/issues. Which is one of the reasons I chose to write
> the www package from scratch, rather than build it on top of the http
> package.
>

Quite possible though in my usage I've found the http package has
greatly improved between 2.7 and the current version.

Again, I'm not knocking the other http packages which I also use. They
are undoubtedly more convenient and featureful and much appreciated. I'm
merely drawing attention to what I see as a strength in the Tcl http one.

/Ashok

[Schelte]

On 19/10/2021 00:13, Schelte wrote:
> So for now, probably the only way to test the package with iocp would be
> to modify the source code to load the package in the helper thread.
>

I have added a new configuration option to the www package. You should
now (hopefully) be able to use the iocp socket command by doing
something like this at the start of your script:

package require www
www configure -socketcmd [list apply [list {path args} {
set ::auto_path $path
package require iocp_inet
tailcall socket {*}$args
}] $auto_path]

Clearly, you can simplify this a bit if the iocp_inet package is in the
default auto_path.

Probably needles to say: The socket command replacement is executed in a
separate thread, so it has to be self-contained. You can't use any procs
defined in the main thread.


Schelte

[Ashok]

Great. I'll give it a shot when I have some time.

/Ashok

[apn]

On 10/20/2021 6:29 PM, Schelte wrote:

> I have added a new configuration option to the www package. You should
> now (hopefully) be able to use the iocp socket command by doing
> something like this at the start of your script:
>
> package require www
> www configure -socketcmd [list apply [list {path args} {
>     set ::auto_path $path
>     package require iocp_inet
>     tailcall socket {*}$args
> }] $auto_path]
>

Finally got around to this and it works as advertised.

/Ashok

[Schelte]

On 10/11/2021 17:55, apn wrote:
> Finally got around to this and it works as advertised.

Excellent. Thanks for checking.


Schelte