Need advice with Mojo::IOLoop::Delay and segmentation faults

[Scott]

I maintain sites that use email validation services through a http api. I use the Mojolicious delay non-blocking code. In the last day or so, I have had my delay processes hanging resulting in 502 errors (hypnotoad reverse proxy with nginx) and at time the http service returning numerous blank records. I blamed them at first, but when I switched to non-blocking code, it worked fine. I boiled down my code to a command line program:

Mojo::IOLoop::Delay->new->steps(

  sub {

    my $delay = shift;

    $ua->get( $bv . $_ => $delay->begin ) for @emails;

  },

  sub {

    my $delay = shift;

    for my $tx ( @_ ) {

      my $res = $tx->res->json;

      say Dumper $res;

    }

  }

)->wait;

At times it works great, and at other times I get errors:

Segmentation fault (core dumped)

and

*** Error in `perl': double free or corruption (out): 0x00007f9c40000910 ***

Aborted (core dumped)

Some of you may know exactly what the problem is, but I can only guess. Is this something to do with Perl itself? For a workstation and server I use Ubuntu 14.04 and it is happening on both. I update as updates happen. I noticed in the apt log that libc6, libc6-dev updated a few days back. Since I use perlbrew, do I need to recompile my perl after certain packages are updated? Currentlty, I am using the most up-to-date Mojolicious. I read the changes before updating and test out on my workstation first, but I have not had a Perl problem before and I am not sure I am managing things correctly.

Thanks for any advice,

Scott K.

[sri]

Is this something to do with Perl itself?

That would be my first guess, but it could also be a bug in an XS module.

--

sebastian

[Scott]

Thanks Sebastian for the quick response.

I have recompiled perl with perlbrew (5.20.0 > 5.20.1) on a fully up-to-date Kubuntu 14.04 and nothing has changed, same errors as before. Sam on the server which is perl 8.18.2. The entire script is:

#!/usr/bin/env perl

use Modern::Perl '2013';

use Mojo::UserAgent;

use Data::Dumper;

use Mojo::IOLoop::Delay;

my $bv = '...';

my $ua = Mojo::UserAgent->new;

my @emails = ( ... );

for my $email ( @emails ) {

  my $tx = $ua->get( $bv . $email );

  my $res = $tx->res->json;

  say Dumper $res;

}

Mojo::IOLoop::Delay->new->steps(

  sub {

    my $delay = shift;

    $ua->get( $bv . $_ => $delay->begin ) for @emails;

  },

  sub {

    my $delay = shift;

    for my $tx ( @_ ) {

      my $res = $tx->res->json;

      say Dumper $res;

    }

  }

)->wait;

I do have EV installed.

I have several sites using the non-blocking ua code and until a few days ago it was fine. I use the mojo ua to send emails through another http api and still use mango (only until end of January and with a freeze on the MongoDB updates) on one site in non-blocking fashion with no problems.

[Alex Efros]

Hi!

Please add error-checking code and see is it change anything.
Also, I recommend you do add something like "warn 1;", "warn 2;", etc.
between most lines to find out on which line perl crash.

On Tue, Dec 09, 2014 at 03:14:28PM -0800, Scott wrote:
> for my $email ( @emails ) {
> my $tx = $ua->get( $bv . $email );

warn($tx->error->{message}), next if $tx->error;

> my $res = $tx->res->json;
> say Dumper $res;
> }
>
> Mojo::IOLoop::Delay->new->steps(
> sub {
> my $delay = shift;
> $ua->get( $bv . $_ => $delay->begin ) for @emails;
> },
> sub {
> my $delay = shift;
> for my $tx ( @_ ) {

warn($tx->error->{message}), next if $tx->error;

> my $res = $tx->res->json;
> say Dumper $res;
> }
> }
> )->wait;
>
> I do have EV installed.

Cool, but then please add 'use EV;' at beginning of your script, just to
make sure it works.

--
WBR, Alex.

[Scott]

I have tracked down my issue to Net::DNS::Native. If removed, then the non-blocking script works. I played around with installing older Mojolicious versions and the script started intermittently segment faulting at version 5.59 with Net::DNS::Native installed. Per the docs, the module needs threads and will not install if the environment cannot support the module, but it installs without message with cpanm and pass tests when installed manually on up-to-date Ubuntu 14.04. I have removed it from my workstations and server and all is OK.

Scott K.

[Scott]

Should also mention that my perl (5.18.2, 5.20.0) is not compiled with threads. Might of been assumed, but should of mentioned.

Scott K.

[sri]

I have tracked down my issue to Net::DNS::Native. If removed, then the non-blocking script works.

Please open an issue for Net::DNS::Native, this needs to be tracked down.

--

sebastian 

[sri]

Should also mention that my perl (5.18.2, 5.20.0) is not compiled with threads. Might of been assumed, but should of mentioned.

Just to make sure, the Net::DNS::Native version is 0.13, right?

--

sebastian 

[Alex Efros]

Hi!

To be honest, when I noticed this module in Mojo changelog it sounds like
very bad idea… perl, XS, threads… really? May I ask you why do you choose
it over, for example, pure-perl, fast and reliable AnyEvent::DNS?
Mojo is usually used with EV (at least by everyone who bothers about async
DNS), so AnyEvent will work just fine.

--
WBR, Alex.

[Scott]

Yes, Net::DNS::Native 0.13.

[sri]

Yes, Net::DNS::Native 0.13.

Too bad, i would hate having to remove support for non-blocking name resolution. Please keep us updated on the issue.

--

sebastian 

[Scott]

I have opened up an issue on rt.cpan.org. Very much a novice at it, hope it is done correctly.

Scott K.

[sri]

Please keep us updated on the issue.

In the meantime, it would be really nice if you could try again without extra threads. (by applying this patch)

    https://gist.github.com/anonymous/5a8b49b2749d77cd716b

--

sebastian 

[Scott]

Sorry, did not seem to help.

[Oleg]

Can you connect to #mojo channel at irc.perl.org and ping me to resolve this faster?

[sri]

Sorry, did not seem to help.

Could you please help Oleg with tracking this down?

    https://rt.cpan.org/Public/Bug/Display.html?id=100834#txn-1441004

It's a little unfortunate that every Net::DNS::Native bug has such severe consequences.

If this can't get resolved within the next few days, i think we only have two options.

a) We remove support for non-blocking name resolution, until Net::DNS::Native had more time to iron out the kinks.

b) We add proper unit tests for non-blocking name resolution, and make MOJO_NO_NDN default to 1. (This would depend on volunteers stepping up to design portable tests)

--

sebastian 

[sri]

a) We remove support for non-blocking name resolution, until Net::DNS::Native had more time to iron out the kinks.

I guess we could also add a public API for pluggable resolvers, but this would once again depend on volunteers coming up with a nice design.

--

sebastian 

[sri]

Oleg has found the problem, and i'm afraid it doesn't look good.

    https://github.com/kraih/mojo/issues/716

--

sebastian

[sri]

    https://github.com/kraih/mojo/issues/716

I'm afraid this has now resulted in the removal of non-blocking name resolution support. :(

--

sebastian 

[sri]

I'm afraid this has now resulted in the removal of non-blocking name resolution support. :(

We do have a prototype for a pluggable resolver API though, but currently no volunteers to clean it up.

    https://github.com/kraih/mojo/compare/pluggable_resolver

--

sebastian 

[Alex Efros]

Hi!

On Thu, Dec 11, 2014 at 11:54:08AM -0800, sri wrote:
> We do have a prototype for a pluggable resolver API though, but currently
> no volunteers to clean it up.
>
> https://github.com/kraih/mojo/compare/pluggable_resolver

It makes sense to implement second plugin using AnyEvent::DNS to see how
it fit into resolver API.

Also, I think it probably doesn't make sense to use resolver in case user
already provided IP (v4/v6) in $address.

And I'm not sure about special handling of 'localhost':
- IP for localhost is usually defined in /etc/hosts, and it might be IPv6
one or be like 127.0.0.2
- in my experience it's very annoying when library see 'localhost' and
make some "smart" decision based on this - like when DBD::MySQL (or
libmysqlclient.so, not sure) switch to using UNIX sockets instead of TCP
- Net::DNS::Native is better than AnyEvent::DNS because it also support
host names defined in /etc/hosts, append search domain defined in
/etc/resolv.conf, etc. - but we can partially compensate for this by
switching to using (blocking) system resolver for host names which
doesn't contain dot/domain… this will also solve issue with non-standard
address for 'localhost', and it won't be actually blocking in most
cases, but it probably should be implemented in plugin for
AnyEvent::DNS, not in resolver API

--
WBR, Alex.

[sri]

We do have a prototype for a pluggable resolver API though, but currently no volunteers to clean it up.

    https://github.com/kraih/mojo/compare/pluggable_resolver

Sadly i've made a mistake, and this version is off the table again, since it only allows to change the resolver globally. And we need to be able to allow event loop specific resolvers like UV::getaddrinfo or IO::Async::Resolver.

--

sebastian

[sri]

We are going to try again with Net::DNS::Native 0.14, if you encounter any new segfaults with Mojolicious 5.69 that disappear with MOJO_NO_NDN=1, please let us know.

    https://github.com/kraih/mojo/commit/ce0b9082ac0bad4a907d938f47e65813cc053623

--

sebastian