Tcl Http for Proxy

[Alexandru]

Hi,

Does the http package allow to communicate to a server which only accepts https requests over a proxy server that communicates over http with the client?

So I'm a client inside a local network protexted by a proxy. I want to http get a resource on a https server. But I need to send the request through the proxy, which only accepts http requests.

Is this possible with the Tcl's package http and tls?

Thanks!
Alexandru

[Harald Oehlmann]

Alexandru, Ashok,

well, here is where I am with the issue:
http://wiki.tcl.tk/2627
Basically use the tls package in case of a proxy.

I discussed this with Ashok two times.
He made this mark for himself:
https://sourceforge.net/p/twapi/bugs/162/
and here is the tcllib ticket:
https://core.tcl.tk/tcllib/info/e72cf5e815088248
All thanks to Ashok for all his work. This is not a reminder, just be
happy and anything is ok.

Thanks,
Harald

[Alexandru]

Hi Harald,

So you are saying, that Twapi cannot be used to solve this issue. Instead I should use the TLS and autoproxy package, according to your solution on the wiki. Is this correct?

Thanks!

[Harald Oehlmann]

Am 13.04.2018 um 18:41 schrieb Alexandru:
> So you are saying, that Twapi cannot be used to solve this issue. Instead I should use the TLS and autoproxy package, according to your solution on the wiki. Is this correct?

Yes, that is my knowledge.

Harald

[Alexandru]

Could you explain the role of ::autoproxy::tunnel_connect? Your are not using this in your example. When is ::autoproxy::tunnel_connect needed? What it does different than ::autoproxy::tls_connect or ::autoproxy::tls_socket. I read the manual but still not sure I understand.

[Harald Oehlmann]

Sorry, no idea... I followed an example, that is all.
Harald

[Alexandru]

I just went through the code of autoproxy and found this:

autoproxy::tunnel_connect
this command just opens the socket through the proxy to the target machine -- no SSL/TLS negotiation is done yet.
The command is called by ::autoproxy::tls_connect if the proxy host is specified in the autoproxy_init command.

::autoproxy::tls_connect
calls tunnel_connect if proxy host is set.

::autoproxy::tls_socket
is the actual command to communicate through the proxy or directly by calling tls_connect or ::tls::socket.

By the way, your code has an error I think:

By taking a look at the tls_socket command (see below), the last two arguments are simply ignored. So in your example:

http::register https 443 [list autoproxy::tls_socket -tls1 1]

the "-tls1 1" option will be ignored. Am I rigth?


proc ::autoproxy::tls_socket {args} {
variable options

# Look into the http package for the actual target. If a proxy is in use then
# The function appends the proxy host and port and not the target.

upvar host uhost port uport
set args [lrange $args 0 end-2]
lappend args $uhost $uport

set s [eval [linsert $args 0 tls_connect]]

# record the tls connection status in the http state array.
upvar state state
tls::handshake $s
set state(tls_status) [tls::status $s]

return $s
}

[Ashok]

Ah, I'd completely forgotten about this, I think in part because looking
at the code at the time, it was more an issue with autoproxy being
hardcoded to use the TLS extension commands and not so much with twapi
missing some needed function, and I didn't want to / have the time to
get into yet another package.

If someone can set me up with an accessible proxy server (no clue what
that involves), I can try modifying autoproxy.

/Ashok

[Alexandru]

Hi Ashok,

I use one of the free proxy adresses out there. Here is my complete code for testing the proxy. It works for me like this, although the proxy oftens times out (I don't know why...):

package require http
package require tls
package require autoproxy

console show
puts "Please wait. This could take some time. If connection times out, please reapeat."
update

set proxy_host 92.222.83.160
set proxy_port 80
set proxy_user ""
set proxy_pass ""
set proxy_ssl2 0
set proxy_ssl3 0
set proxy_tls1 0
set proxy_tls1.1 0
set proxy_tls1.2 0
set proxy_CAfile ""

set options {}
set cafile [string map {\\ /} $proxy_CAfile]
if {$cafile!=""} {
lappend options -request 1
lappend options -require 1
}
if {$proxy_ssl2==1} {
lappend options -ssl2 $proxy_ssl2
}
if {$proxy_ssl3==1} {
lappend options -ssl3 $proxy_ssl3
}
if {$proxy_tls1==1} {
lappend options -tls1 $proxy_tls1
}
if {${proxy_tls1.1}==1} {
lappend options -tls1.1 ${proxy_tls1.1}
}
if {${proxy_tls1.2}==1} {
lappend options -tls1.2 ${proxy_tls1.2}
}

catch {::http::unregister https}
if {$proxy_host=="" || $proxy_port==""} {
::autoproxy::init
} else {
::autoproxy::configure -proxy_host $proxy_host -proxy_port $proxy_port
if {$proxy_user!=""} {
::autoproxy::configure -basic -username $proxy_user -password $proxy_pass
}
}
set tlssocket [::http::register https 443 [list autoproxy::tls_socket {*}$options $proxy_host $proxy_port]]

set token [::http::geturl https://www.meshparts.de/cgi-bin/IPC?cmd=::meshparts::ServerTest]
upvar #0 $token state
set ncode [lindex $state(http) 1]
puts ncode=$ncode
if {[string index $ncode 0]==2} {
set result [string trimleft $state(body)]
puts $result
}

[Harald Oehlmann]

Dear Ashok,

thank ylou for the E-Mail. I would propose to use a proxy executable
like FreeProxy and test with that locally. The advantage is, that you
also have local log-files and the proxy may give you feed-back.

Here is a thread discussing about possibilities:
https://superuser.com/questions/335123/simple-web-proxy-server-for-windows

By the way, to use TOR, you always need a proxy.

Thank you,
Harald

[Alexandru]

Am Freitag, 13. April 2018 18:31:03 UTC+2 schrieb Harald Oehlmann:

Now, the communication through a free proxy works.
The next problem is, that the actual proxy of a customer needs autentication.
It seems that the username and password are not sent by http package. I get the error:

Proxy Authentication Required
while executing
"::http::geturl $thisurl {*}$options"

According to the autoproxy package, we can specify a user name and password when configuring the proxy:

::autoproxy::configure -proxy_host $proxy_host
::autoproxy::configure -proxy_port $proxy_port

if {$proxy_user!=""} {
::autoproxy::configure -basic -username $proxy_user -password $proxy_pass
}

... which I did... So the question is, why do I still get this error??

Many thanks.
Alexandru

[Donal K. Fellows]

On 16/04/2018 13:17, Alexandru wrote:
> So the question is, why do I still get this error??

They might be requiring a different authentication scheme than simple
Basic Auth? Alas, it's not something I'm experienced with.

Donal.
--
Donal Fellows — Tcl user, Tcl maintainer, TIP editor.

[Ashok]

OK, I've checked changes into the autoproxy repository branch with the
required changes to have autoproxy::tls_socket work with twapi's TLS
implementation. The modified autoproxy.tcl file is at

http://core.tcl.tk/tcllib/artifact/0e4f60a6180e96dd

Doc has also been updated. Usage is something like

package require autoproxy
package require twapi
autoproxy::init
autoproxy::configure -tls_package twapi; # Use twapi instead of tls
http::register https 443 autoproxy::tls_socket
http::geturl....blah blah

Tested with one proxy server from one client system :-)

/Ashok

[Harald Oehlmann]

Wow !
Light speed and incredibly good.
Test will follow.

Thanks,
Harald

[Harald Oehlmann]

Am 17.04.2018 um 16:19 schrieb Ashok:

> OK, I've checked changes into the autoproxy repository branch with the
> required changes to have autoproxy::tls_socket work with twapi's TLS
> implementation. The modified autoproxy.tcl file is at
>
> http://core.tcl.tk/tcllib/artifact/0e4f60a6180e96dd
>
> Doc has also been updated. Usage is something like
>
> package require autoproxy
> package require twapi
> autoproxy::init
> autoproxy::configure -tls_package twapi; # Use twapi instead of tls
> http::register https 443 autoproxy::tls_socket
> http::geturl....blah blah
>
> Tested with one proxy server from one client system :-)

Take care, Ashok, version number is updated, do "fossil pull".

Thanks,
Harald

[Alexandru]

Hi Ashok,

Many thanks for the fast reaction and good work.
I tested you changes to autoporxy and it works.

I had to do two minor changes:
1. In pkgIndex.tcl package provide 1.5.3 changed to 1.7 (no idea why I had 1.5.3)
2. In

set tlssocket [::http::register https 443 [list autoproxy::tls_socket {*}$options $proxy_host $proxy_port]]

I had to remove the last to arguments:
set tlssocket [::http::register https 443 [list autoproxy::tls_socket {*}$options]]
I the prev version the last two arguments were necessary, although I never logically understood why. So now it's better anyway.


Regards
Alexandru

[Harald Oehlmann]

Am 17.04.2018 um 18:24 schrieb Alexandru:
> I had to do two minor changes:

> 2. In
> set tlssocket [::http::register https 443 [list autoproxy::tls_socket {*}$options $proxy_host $proxy_port]]
> I had to remove the last to arguments:
> set tlssocket [::http::register https 443 [list autoproxy::tls_socket {*}$options]]
> I the prev version the last two arguments were necessary, although I never logically understood why. So now it's better anyway.

I hope, this was due to your older former version 1.5.7 of autoproxy.
I did not require this in 1.6 and it would be a bug if it would be required.

Glad to hear that it works and is tested.

Thank you,
Harald

[Harald Oehlmann]

Am 17.04.2018 um 16:19 schrieb Ashok:

> Tested with one proxy server from one client system :-)

I have also tested it and it works for me.
Thanks !

I have no idea how any change gets into the final distribution.
I have E-Mail Andreas for another modification (log::logsubst extension)
and he has put it on his list.
I may send him an E-Mail again...

Thank you all,
Harald

[Harald Oehlmann]

Am 18.04.2018 um 09:05 schrieb Harald Oehlmann:
> Am 17.04.2018 um 16:19 schrieb Ashok:
>> Tested with one proxy server from one client system :-)

Ashok,

I have a very strange issue for the non-proxy case.

So:

package require autoproxy
::autoproxy::init
package require twapi_crypto
autoproxy::configure -tls_package twapi

http::register https 443 ::autoproxy::tls_socket

% parray autoproxy::options
autoproxy::options(authProc) =
autoproxy::options(basic) =
autoproxy::options(no_proxy) =
autoproxy::options(proxy_host) =
autoproxy::options(proxy_port) = 80
autoproxy::options(tls_package) = twapi

This should be identical to
http::register https 443 ::twapi::tls_socket

But I get an error in the geturl call in the first case.
And I dont find out what is happenning...

The command is:
http::geturl
{https://accessgudid.nlm.nih.gov/api/v1/devices/lookup.xml?di=J019ABC1234}

In the first case, I get the error dict:
-code 1
-level 0
-errorstack {
INNER {invokeStk1 http::geturl
https://accessgudid.nlm.nih.gov/api/v1/devices/lookup.xml?di=J019ABC1234}
CALL {HTTPGet
https://accessgudid.nlm.nih.gov/api/v1/devices/lookup.xml?di=J019ABC1234}}
-errorcode NONE
-errorinfo {Invalid SSPI security context handle
while executing
"http::geturl $URL"}
-errorline 2

In the second case, all works o.k. (correct status 404 is returned which
is ok).

I tried to find out what is happenning.
So, I have put a puts in:
proc ::autoproxy::tls_connect {args} {
...
if {$options(tls_package) eq "twapi"} {
puts >>>[linsert $args 0 ::twapi::tls_socket]<<<
set s [eval [linsert $args 0 ::twapi::tls_socket]]


It returns:
>>>::twapi::tls_socket -async accessgudid.nlm.nih.gov 443<<<

Many tries led to the fact, that the new code in autoproxy::tls_socket
is the issue.
If I add a "return $s" just after the call of tls_connect, the error is
gone.

E.g.:

proc ::autoproxy::tls_socket {args} {
variable options

# Look into the http package for the actual target. If a proxy is in
use then
# The function appends the proxy host and port and not the target.

upvar host uhost port uport
set args [lrange $args 0 end-2]
lappend args $uhost $uport

set s [eval [linsert $args 0 tls_connect]]

return $s
}

I think, this part is anyway only to get some info into the state array...
Without that, anything works for me....

Thank you,
Harald

[Ashok]

I see the problem, the tls version of tls_socket blocks till the
connection is open even if the -async option is specified (as has always
been the case). The twapi version of tls_socket does not block if -async
is specified, thus an attempt to get tls status fails since the
connection handshake is not complete yet.

Thanks for the detailed debugging output, really helps.

I'll look into a fix. Personally, I feel the socket should not block if
-async is specified (similar to the ::socket behaviour) but changing
that would mean a change from existing behaviour.

/Ashok

[Harald Oehlmann]

Am 19.04.2018 um 09:48 schrieb Ashok:
> I'll look into a fix. Personally, I feel the socket should not block if
> -async is specified (similar to the ::socket behaviour) but changing
> that would mean a change from existing behaviour.

I can confirm that there are plans by Reinhard (Unix, project leader)
and myself to make socket connect totally non blocking. What is actually
blocking is the DNS lookup.
The loop which checks all possible IP-Addresses of a name is implemented
non-blocking.

I can also confirm that there is no action in this project currently.

I would also love to include David Graveraux iocpsock
http://core.tcl.tk/tcl/tktview?name=fc3c425cde

So much to do...

Thanks,
Harald

[Harald Oehlmann]

The problematic code is only about the state array of http.
It would be ok (for me) to just put nothing into the state array.

Thanks, Harald

Am 19.04.2018 um 09:48 schrieb Ashok:

[Alexandru]

Hey guys,

What about a proxy that requires other type of autentication that basic? Do I have do change this in the source code of the autoproxy package (or add a new opotion)? Will this work?

Thanks!
Alexandru

[Ashok]

I've updated the branch with the fix to skip setting tls_status if
negotiation is not yet complete.

/Ashok

[Ashok]

On 4/19/2018 5:58 PM, Alexandru wrote:> Hey guys,

>
> What about a proxy that requires other type of autentication that basic? Do I have do change this in the source code of the autoproxy package (or add a new opotion)? Will this work?
>
> Thanks!
> Alexandru
>

Sorry, don't really know how proxy authentication works.

/Ashok

[Alexandru]

Hi Ashok,

Here comes the genius of Twapi: The customer tested it and it works! Seems like your Twapi takes care of everything, including the autentication.

Thank you!
Alexandru

[pal...@yahoo.com]

I'm happy to take credit but I doubt it is deserved! TWAPI has no support or code for http authentication at all so it was either a SSL/TLS issue or something else changed with the customer. Either way, glad it is working.

/Ashok

[Alexandru]

Confirmed... I just find out from the customer how it still works: He has to open some web page in the internet explorer (or other browser) and then he can start my app. So the browser does the autentication and this last a while. During this time, other apps can go connect to the proxy without autentication.

So the question remains, in general, how one can set up autentication for proxy in Tcl. This feature is very less discussed today although most of the midle and large companies use a proxy.

Regards
Alexandru

[Harald Oehlmann]

Alexandru, Ashok,

Andreas is asking for a test in tcllib with twapi:

https://core.tcl.tk/tcllib/info/e72cf5e815088248

Who volonteers to make the test?

Should I jump in ?

Thanks,
Harald

[Alexandru]

It's not clear to me what should be tested. I use Twapi and autoproxy in the official release of my app which is also used by our customers. So it works actually. But I'm ready to do other kind of tests if necessary.

[Ashok]

On 5/4/2018 4:53 PM, Harald Oehlmann wrote:
> Alexandru, Ashok,
>
> Andreas is asking for a test in tcllib with twapi:
>
> https://core.tcl.tk/tcllib/info/e72cf5e815088248
>
> Who volonteers to make the test?
>
> Should I jump in ?
>
> Thanks,
> Harald
>
>

I had already tested after Andreas asked and sent him mail. My guess is
he has not seen it. I'll ping him again

/Ashok

[Alexandru]

Another thing that's not clear to me is how to use proxy autentication when using autoproxy and Twapi.

[Harald Oehlmann]

I suggest to comment the ticket.
So others (like me) do not worry ;-)

Thanks,
Harald

[Harald Oehlmann]

Doesen't it work the normal autoproxy way ?

E.g.
::autoproxy::Init
::autoproxy::configure -username me -password you

[Alexandru]

No it doesn't work because the autentication type is not basic and I cann't change this in autoproxy. Besides, it must go through Twapi. The problem with the autentication is solved by not setting the autentication at all. Instead the user must "simply" start Internet Explorer. I guess the IE does the autentication and the proxy thinks that the twapi requests are from IE or something similar.

[Ashok]

Andreas has committed the changes to the repository trunk.

/Ashok

[Ashok]

TWAPI only establishes a TLS channel. It knows nothing about what goes
over that channel, http or anything else and this issues related to http
authentication are unlikely to be related to twapi.

Having said that, the fact that you say that it works after IE
authentication seem to me to indicate that the authentication is not
really HTTP authentication at all but some kind of "proxy firewall"
which permits connections from any application once one application
coming from that system has authenticated. I can't think of how any pure
http protocol based auth mechanism would exhibit that behaviour.

/Ashok

[Alexandru]

Well, in IE the user sets up the autentication and IE then autenticates the connection of the system. I would be nice, if we had someting similar in Tcl... I'm not saying it should be Twapi, just that it's necessary as it is not possible with what we have now in Tcl.