Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

LWP hangs forever with get

60 views
Skip to first unread message

Jesper Persson

unread,
May 21, 2014, 4:37:12 AM5/21/14
to libwww
---------------------
use LWP::Simple;
---------------------
the perl process never returns.

lwp version: "VERSION: 6.06"

Summary of my perl5 (revision 5 version 18 subversion 2) configuration:
  Platform:
    osname=MSWin32, osvers=6.2, archname=MSWin32-x64-multi-thread
    uname='Win32 strawberry-perl 5.18.2.1 #1 Tue Jan  7 22:32:35 2014 x64'

Best Regards
Jesper Persson

Daniel de Oliveira Mantovani

unread,
May 21, 2014, 8:12:58 AM5/21/14
to Jesper Persson, libwww
Hi, Jesper. It's happen because it should happen. You don't have the
error because you are not handle error.

This happen because LWP has a timeout to wait the server response,
after this timeout it will return a timeout error. In your case, you
don't handle the error the only stuff you get is nothing.

This URL "https://www.butler.edu/boa/default.aspx" has two problems:

1 - The certificate authority is not trusted
2 - Never (i.e until timeout) response to the client

If you are inside an intranet and you can access this site from your
browser and you know ins't cache. Start handle with error, and show it
to us.

To avoid "certificate authority not trusted" you can use:

BEGIN {
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
}


Lab:~ danielmantovani$ wget "https://www.butler.edu/boa/default.aspx"
--2014-05-21 09:04:08-- https://www.butler.edu/boa/default.aspx
Resolving www.butler.edu (www.butler.edu)... 159.242.130.240
Connecting to www.butler.edu (www.butler.edu)|159.242.130.240|:443... connected.
ERROR: The certificate of ‘www.butler.edu’ is not trusted.
ERROR: The certificate of ‘www.butler.edu’ hasn't got a known issuer.

Lab:~ danielmantovani$ wget "https://www.butler.edu/boa/default.aspx"
--no-check-certificate
--2014-05-21 09:04:39-- https://www.butler.edu/boa/default.aspx
Resolving www.butler.edu (www.butler.edu)... 159.242.130.240
Connecting to www.butler.edu (www.butler.edu)|159.242.130.240|:443... connected.
WARNING: The certificate of ‘www.butler.edu’ is not trusted.
WARNING: The certificate of ‘www.butler.edu’ hasn't got a known issuer.
HTTP request sent, awaiting response... Read error (Success.) in headers.
Retrying.
--

-dom

--

Daniel de Oliveira Mantovani
Business Analytic Specialist
Perl Evangelist /Astrophysics hobbyist.
+55 11 9 8538-9897
XOXO

Jesper Persson

unread,
May 21, 2014, 12:57:58 PM5/21/14
to Daniel de Oliveira Mantovani, libwww
this perl code
------------------------------------------
use LWP::Simple qw($ua get);
$ua->timeout(5);
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
------------------------------------------

just tried fresh perl installs.

strawberry perl 5.16.3.1:  doesnt hang.

strawberry perl 5.18.0.1:  doesnt hang.

strawberry perl 5.18.1.1:  doesnt hang.

strawberry perl 5.18.2.1:  hangs.

strawberry perl 5.18.2.2:  hangs.

/Jesper

Tom Hukins

unread,
May 21, 2014, 1:03:09 PM5/21/14
to lib...@perl.org
On Wed, May 21, 2014 at 06:57:58PM +0200, Jesper Persson wrote:
> this perl code
> ------------------------------------------
> use LWP::Simple qw($ua get);
> $ua->timeout(5);
> $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
> $contents = get("https://www.butler.edu/boa/default.aspx");
> ------------------------------------------
>
> just tried fresh perl installs.
>
> strawberry perl 5.16.3.1: doesnt hang.
>
> strawberry perl 5.18.0.1: doesnt hang.
>
> strawberry perl 5.18.1.1: doesnt hang.
>
> strawberry perl 5.18.2.1: hangs.
>
> strawberry perl 5.18.2.2: hangs.

I expect you're using different versions of LWP with these different
versions of Perl and you're encountering the change documented at
https://metacpan.org/pod/LWP#PERL_LWP_SSL_VERIFY_HOSTNAME

Tom

Jesper Persson

unread,
May 21, 2014, 3:10:14 PM5/21/14
to Tom Hukins, libwww
Well I do set the $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; so I guess the change shouldn't affect me.

perl code
-----------------------------------
use LWP::Simple qw($ua get);
use IO::Socket::SSL qw(debug3);
$ua->timeout(5);
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 1;
-----------------------------------

with the perl version that doesnt hang:

strawberry perl 5.18.1.1
libwww-perl-6.05

DEBUG: .../IO/Socket/SSL.pm:1805: new ctx 39861360
DEBUG: .../IO/Socket/SSL.pm:449: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:451: socket connected
DEBUG: .../IO/Socket/SSL.pm:469: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:504: using SNI with hostname www.butler.edu
DEBUG: .../IO/Socket/SSL.pm:527: set socket to non-blocking to enforce timeout=5
DEBUG: .../IO/Socket/SSL.pm:540: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:550: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:560: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:570: handshake failed because socket did not became ready
DEBUG: .../IO/Socket/SSL.pm:1842: free ctx 39861360 open=39861360
DEBUG: .../IO/Socket/SSL.pm:1847: free ctx 39861360 callback
DEBUG: .../IO/Socket/SSL.pm:1850: OK free ctx 39861360

NO HANG :-)

with the perl version that hangs:

strawberry 5.18.2.2
libwww-perl-6.05
DEBUG: .../IO/Socket/SSL.pm:2137: new ctx 49554496
DEBUG: .../IO/Socket/SSL.pm:478: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:480: socket connected
DEBUG: .../IO/Socket/SSL.pm:498: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:528: using SNI with hostname www.butler.edu
DEBUG: .../IO/Socket/SSL.pm:566: set socket to non-blocking to enforce timeout=5
DEBUG: .../IO/Socket/SSL.pm:579: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:589: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:599: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:619: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:2098: ok=1 cert=53220176
DEBUG: .../IO/Socket/SSL.pm:2098: ok=1 cert=60570800
DEBUG: .../IO/Socket/SSL.pm:2098: ok=1 cert=60570608
DEBUG: .../IO/Socket/SSL.pm:1339: scheme=www cert=60570608
DEBUG: .../IO/Socket/SSL.pm:1348: identity=www.butler.edu cn=www.butler.edu alt=2 www.butler.edu 2 butler.edu
DEBUG: .../IO/Socket/SSL.pm:579: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:589: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:599: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:619: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:579: Net::SSLeay::connect -> 1
DEBUG: .../IO/Socket/SSL.pm:634: ssl handshake done

HANGS FOREVER :-(

/Jesper

Victor Efimov

unread,
May 21, 2014, 3:37:22 PM5/21/14
to Daniel de Oliveira Mantovani, Jesper Persson, libwww
2014-05-21 16:12 GMT+04:00 Daniel de Oliveira Mantovani
<daniel.olive...@gmail.com>:
> Hi, Jesper. It's happen because it should happen. You don't have the
> error because you are not handle error.
>
> This happen because LWP has a timeout to wait the server response,
> after this timeout it will return a timeout error. In your case, you
> don't handle the error the only stuff you get is nothing.

But that fact that LWP has timeout, means it _should_ _not_ _hang forever_

Daniel de Oliveira Mantovani

unread,
May 21, 2014, 3:41:27 PM5/21/14
to Victor Efimov, Jesper Persson, libwww
I *know it*, I understood in first place, that he was getting undef
from "get" function and not hang forever.

Jesper Persson

unread,
May 26, 2014, 2:47:35 AM5/26/14
to libwww
The "hang" happens in c:\strawberry\perl\vendor\lib\Net\HTTP\Methods.pm in the procedure
sub read_response_headers
where it calls 
my($status, $eol) = my_readline($self, 'Status');

Its actually not a hang because perl is looping in my_readline where it keeps repeating the code:

READ:
            {
                die "read timeout" unless $self->can_read;
                my $n = $self->sysread($_, 1024, length);
                unless (defined $n) {
                    redo READ if $!{EINTR} || $!{EAGAIN};

Should that be possible? Maybe the procedure could have a max number of "redo READ"'s or a timer to give up after some time?

Best regards
Jesper Persson

Steffen Ullrich

unread,
Jun 1, 2014, 5:55:42 PM6/1/14
to lib...@perl.org

>
> READ:
> {
> die "read timeout" unless $self->can_read;
> my $n = $self->sysread($_, 1024, length);
> unless (defined $n) {
> redo READ if $!{EINTR} || $!{EAGAIN};

The server makes a connection reset if you try to access
https://www.butler.edu/boa/default.aspx. On unix this will set $! to ECONNRESET,
and for unknown reason this will not set $! on Windows with Strawberry
perl5.18.2 , but only returns "SSL read error" from the SSL layer.

Thus $! will stay at EINTR from the previous error and it will loop forever.

IO::Socket::SSL 1.992 (just released) works around this problem by resetting $!
before doing I/O. This way $! will no longer stay at $!, but will be undef and
thus the loop will exit, albeit with no specific error. The real fix would be of
course to set $! to ECONNRESET in the underlying IO::Socket layer if the TCP
connection was reset.

Steffen


Jesper Persson

unread,
Jun 2, 2014, 11:55:46 AM6/2/14
to Steffen Ullrich, libwww
Hi Steffen,

thank you for taking your time to fix/workaround this.

I have just updated IO::Socket::SSL to version 1.992 and I can verify that the "hang" no longer happens.

Best Regards
Jesper
0 new messages