https through proxy with libwww
Evgeny Stepanov
I need to access https pages and work with forms there. I tried like
that:
use strict;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent('Mozilla/5.0');
$ua->proxy(['https', 'http', 'ftp'] => 'http://my.proxy.addr:3128');
my $req = HTTP::Request->new(GET => 'https://some.secured.host/');
my $res = $ua->request($req);
if ($res->is_success) {
print $res->as_string;
}
else {
print "Failed: ", $res->status_line, "\n";
}
but it gives me
Failed: 501 Not Implemented
This piece of code works just fine without proxy defined, but i need
proxy. As far as i understood, https through proxy is not implemented
yet. Maybe you guys know another way to get it working with https
through proxy?
---
best regards
Evgeny Stepanov
Robert Barta
> use strict;
> use LWP::UserAgent;
>
> my $ua = LWP::UserAgent->new;
> $ua->agent('Mozilla/5.0');
> $ua->proxy(['https', 'http', 'ftp'] => 'http://my.proxy.addr:3128');
> my $req = HTTP::Request->new(GET => 'https://some.secured.host/');
> my $res = $ua->request($req);
> if ($res->is_success) {
> print $res->as_string;
> }
> else {
> print "Failed: ", $res->status_line, "\n";
> }
>
> but it gives me
> Failed: 501 Not Implemented
> This piece of code works just fine without proxy defined, but i need
> proxy. As far as i understood, https through proxy is not implemented
> yet. Maybe you guys know another way to get it working with https
> through proxy?
Evgeny,
What you are trying to do is to initiate a 'GET' while most proxies
would expect method 'CONNECT'. With a GET the proxy would have to open
an SSL connection. With an end-to-end technology like SSL this is
pretty senseless. How should a proxy check the server cert or how
should the proxy send a client cert?
Sometimes it helps to read the docs. In Crypt::SSLeay it says
Crypt::SSLeay Proxy Support
For native Crypt::SSLeay proxy support of https requests, you
need to set an environment variable HTTPS_PROXY to your proxy
server & port, as in:
# PROXY SUPPORT
$ENV{HTTPS_PROXY} = 'http://proxy_hostname_or_ip:port';
$ENV{HTTPS_PROXY} = '127.0.0.1:8080';
ride or break the Crypt::SSLeay support, so do not mix the two.
Use of the HTTPS_PROXY environment variable in this way is
similar to LWP::UserAgent->env_proxy() usage, but calling that
method will likely over
And, voila, in the proxy logs you should see
==> /var/log/squid/access.log <==
1111195481.685 656 192.168..X.XX TCP_MISS/200 8934 CONNECT some.secured.host:443 - DIRECT/XXXXXXXXX -
\rho
Evgeny Stepanov
Friday, March 18, 2005, 8:34:01 PM, вы написали:
AS> Have you tried using NET::SSLeay module within the script?
no, i have Crypt::SSLeay, it was written in the docs that LWP would
use this package. In the same readme was mentioned that LWP could use
NET::SSLeay, but it has no direct bindings for LWP. Whatever that
means, i don't know much of perl, so i have no idea how to use
NET::SSLeay in my script. Maybe some piece of example code will help.
Thanks in advance!
Evgeny Stepanov
RB> Sometimes it helps to read the docs. In Crypt::SSLeay it says
yes, i've read LWP docs but not Crypt::SSLeay :-) Thanks for pointing
me to the right place!
RB> Crypt::SSLeay Proxy Support
RB> For native Crypt::SSLeay proxy support of https requests, you
RB> need to set an environment variable HTTPS_PROXY to your proxy
RB> server & port, as in:
RB> # PROXY SUPPORT
RB> $ENV{HTTPS_PROXY} = 'http://proxy_hostname_or_ip:port';
wow! that works! Thanks, robert!
Andrea Setti
I have an issue to submit to you:
When i loop through WWW::Mechanize->{res}->{_headers} like the following
snippet:
my $headers = $mech->{res}->{_headers};
while ( (my $key, my $value) = each (%$headers)) {
print $key . ': ' . "$value\n";}
}
I get some values as an array:
For instance the 'content-type' key, the 'links' key and the 'expires' key
are seen as an array, with one or more values each (almost) identical.
In example, i have 'content-type' key with two values: [0] =
'text/html;charset=iso-8859-1' and [1] = ' text/html;charset=iso-8859-1'
Please note the space at the beginning of value [1].
How does it happens? It happens when i load a php self-referencing page for
Tutos planner login, on other sites it behaves correctly (without any
hash-of-array).
Thank you all for helping,
Andrea Setti.
Robert Barta
> Hi List.
>
> I have an issue to submit to you:
>
> When i loop through WWW::Mechanize->{res}->{_headers} like the following
> snippet:
>
> my $headers = $mech->{res}->{_headers};
> while ( (my $key, my $value) = each (%$headers)) {
> print $key . ': ' . "$value\n";}
> }
Did you check whether there are actually API calls to do what
you need to do? If I see a component _something in a hash, this
usually tells me to keep my fingers off.
\rho
Andrea Setti
and forward them to the browser...
I'm trying to make a single sign-on script, and i need to do the following
steps:
1. fetch the original login page and submit username & password
2. submit the login page and fetch from the HTTP::Headers all the cookies
that i need to forward to the browser
3. forward to the browser all the cookies / cache informations / other
headers and the content of the authenticated page.
The problem is that i am not able to forward a full HTTP::Message to the
browser, it does not work (ie: $my_HTMessage->as_string();)
So, the only way i have found to forward the cookies to the browser is to
get them from the $mech->(res)->_headers, and print them in stdout building
my own headers followed by the authenticated content...
if anyone has a better idea, please let me know...
Thank you for supporting!
-----Messaggio originale-----
Da: Robert Barta [mailto:rho@mando] Per conto di Robert Barta
Inviato: giovedì 24 marzo 2005 03.49
A: Andrea Setti
Cc: lib...@perl.org; 'John J Lee'
Oggetto: Re: Strange HTTP::Headers behaviour in
WWW::Mechanize->{res}->{_headers} hash
Robert Barta
> Yes, i know, but i didn't found any API that gets the headers from a page
> and forward them to the browser...
>
> I'm trying to make a single sign-on script, and i need to do the following
> steps:
>
> 1. fetch the original login page and submit username & password
> 2. submit the login page and fetch from the HTTP::Headers all the cookies
> that i need to forward to the browser
> 3. forward to the browser all the cookies / cache informations / other
> headers and the content of the authenticated page.
If it is only the cookie stuff then reading
man lwpcook
may help. Remember, that WWW::Mechanize subclasses LWP.
Otherwise I am not sure what headers WHICH THE SERVER sends. you
should put into a message WHICH then THE CLIENT SENDS. There are not
too many occasions where you would do that...
\rho
> -----Messaggio originale-----
> Da: Robert Barta [mailto:rho@mando] Per conto di Robert Barta
> Inviato: gioved? 24 marzo 2005 03.49
> A: Andrea Setti
> Cc: lib...@perl.org; 'John J Lee'
> Oggetto: Re: Strange HTTP::Headers behaviour in
> WWW::Mechanize->{res}->{_headers} hash
Andrea Setti
>
> man lwpcook
>
> may help. Remember, that WWW::Mechanize subclasses LWP.
I have created my mech obj as this: my $mech =
WWW::Mechanize->new(cookie_jar => {});
This would create an empty cookie jar that can be used to fetch cookie
infos.
But it is not so convenient when you need also other parameters from the
headers, i think it could be easier to build a brand new HTTP::Header with
all the needed infos...
> Otherwise I am not sure what headers WHICH THE SERVER sends. you should
put into a message WHICH then THE CLIENT SENDS. There are not too many
occasions > where you would do that...
Well, for instance, i need to fetch all the "link" header, that informs the
bowser to use a certain css stuff, links etc...
Maybe would be necessary to fetch also the cache-control header, the
transfer-encoding...
What i would like to create is a "fake" login page, with no need of
submitting username and password. This could be very useful in a portal
enviromnent...
-----Messaggio originale-----
Da: Robert Barta [mailto:rho@mando] Per conto di Robert Barta
Inviato: giovedì 24 marzo 2005 10.27
A: Andrea Setti
Cc: lib...@perl.org
Oggetto: Re: R: Strange HTTP::Headers behaviour in
Robert Barta
> I have created my mech obj as this: my $mech =
> WWW::Mechanize->new(cookie_jar => {});
> This would create an empty cookie jar that can be used to fetch cookie
> infos.
> But it is not so convenient when you need also other parameters from the
> headers, i think it could be easier to build a brand new HTTP::Header with
> all the needed infos...
>
> > Otherwise I am not sure what headers WHICH THE SERVER sends. you should
> put into a message WHICH then THE CLIENT SENDS. There are not too many
> occasions > where you would do that...
>
> Well, for instance, i need to fetch all the "link" header, that informs the
> bowser to use a certain css stuff, links etc...
You talk about
<link rel="stylesheet" media="screen" href="...." type="text/css" >
headers? These are HTML headers, NOT HTTP headers.
> What i would like to create is a "fake" login page, with no need of
> submitting username and password. This could be very useful in a portal
> enviromnent...
Hmmm.
\rho
> -----Messaggio originale-----
> Da: Robert Barta [mailto:rho@mando] Per conto di Robert Barta
> Inviato: gioved? 24 marzo 2005 10.27