qpsmtpd-async weird ParaDNS lookup problem

[David Favor]

I've been running qpsmtpd-async for years on all sorts of servers.

Likely I have something in DNS setup slightly wrong on a new server
I'm setting up + what's wrong escapes me.

The symptom is qpsmtpd-async hanging forever in the HELO sequence.

Both forkserver + prefork work fine + async works so much better,
I'd like to resolve this problem.

Here's an example of the problem...

Listen child making a Qpsmtpd::PollServer for 7.
11869 in config(plugins)
11869 config(plugins) returning (resolve_sender_host dont_require_anglebrackets rcpt_simple te...@newswire.net ch...@newswire.net do...@newswire.net sup...@newswire.net accou...@newswire.net
sa...@newswire.net deb...@newswire.net maildir /cluster/clients/ivan-budimir %d/users/%l/Maildir) from cache
DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0)
11869 (connect) running plugin: resolve_sender_host
11869 (connect) resolve_sender_host: DEBUG: ip=127.0.0.1 host=localhost domain=localhost
11869 Plugin resolve_sender_host, hook connect returned DECLINED,
11869 in config(smtpgreeting)
11869 config(smtpgreeting) returning (Ready!) from cache

Notice this line emitted from ParaDNS...

DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0)

Bind looks good...

biz-net2# netstat -pluten | grep named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 105 6068 4715/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 105 6073 4715/named
tcp6 0 0 ::1:53 :::* LISTEN 105 6070 4715/named
tcp6 0 0 ::1:953 :::* LISTEN 105 6074 4715/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 105 6067 4715/named
udp6 0 0 ::1:53 :::* 105 6069 4715/named

And host seems to work too...

biz-net2# host localhost
localhost has address 127.0.0.1
localhost has IPv6 address ::1

biz-net2# host 127.0.0.1
1.0.0.127.in-addr.arpa domain name pointer localhost.

If I turn on ParaDNS debugging (export PARADNS_DEBUG=100) I just see the lookup fail again...

100/2 [17636] dns lookup: Trying to resolve A: 127.0.0.1
100/2 [17636] dns lookup: NS Query: 127.0.0.1 (60806)
DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0)

biz-net2# perl -MParaDNS -e 'print "$ParaDNS::VERSION\n"'
2.0

Anyone have any suggestions?

[Ed McLain]

For an IP you need to lookup the PTR in reverse notation
(1.0.0.127.in-addr.arpa) vs an A record which is name to IP. PTR is IP to
Name.

--
Edward McLain
Director of Cloud Architecture
Solutions Architect, VCP, CDCDP
emc...@teklinks.com
d:205.314.6646
<http://www.teklinks.com/>

Please direct all support questions to our support team at
hos...@teklinks.com <http://mailto:hos...@teklinks.com/> or 205.314.6634.

[David Favor]

Ed McLain wrote:
> For an IP you need to lookup the PTR in reverse notation
> (1.0.0.127.in-addr.arpa) vs an A record which is name to IP. PTR is IP to
> Name.

Oops... Hit reply rather than reply-all...

Resending via reply-all...
_______

If I understand correctly, you're asking about this...

dig 1.0.0.127.in-addr.arpa

which returns the same on machines where qpsmtpd-async is working
and the one machine where it's broken...

;; AUTHORITY SECTION:
127.in-addr.arpa. 604800 IN SOA localhost. root.localhost. 1 604800 86400 2419200 604800

If you're referring to something else, give me an exact command + what
I should be looking for.

Thanks.

[Ed McLain]

If using dig you can either do:

dig -x 127.0.0.1

OR

dig 1.0.0.127.in-addr.arpa PTR

That is going to query the primary name server listed in /etc/resolv.conf
- so unless it has a reverse zone setup for the l27.x block you will not
get an answer back or you will get an answer like the one you have below.

What¹s the actual problem you are trying to solve though?

--
Edward McLain
Director of Cloud Architecture
Solutions Architect, VCP, CDCDP
emc...@teklinks.com
d:205.314.6646
<http://www.teklinks.com/>

Please direct all support questions to our support team at
hos...@teklinks.com <http://mailto:hos...@teklinks.com/> or 205.314.6634.




-----Original Message-----
From: David Favor <da...@davidfavor.com>

[David Favor]

Ed McLain wrote:
> If using dig you can either do:
>
> dig -x 127.0.0.1
>
> OR
>
> dig 1.0.0.127.in-addr.arpa PTR
>
> That is going to query the primary name server listed in /etc/resolv.conf
> - so unless it has a reverse zone setup for the l27.x block you will not
> get an answer back or you will get an answer like the one you have below.

Same data on for both commands on both servers...

biz-net2# dig -x 127.0.0.1 | egrep '^[a-z0-9]'
1.0.0.127.in-addr.arpa. 604800 IN PTR localhost.
127.in-addr.arpa. 604800 IN NS localhost.
localhost. 604800 IN A 127.0.0.1
localhost. 604800 IN AAAA ::1

biz-net2# dig 1.0.0.127.in-addr.arpa PTR | egrep '^[a-z0-9]'
1.0.0.127.in-addr.arpa. 604800 IN PTR localhost.
127.in-addr.arpa. 604800 IN NS localhost.
localhost. 604800 IN A 127.0.0.1
localhost. 604800 IN AAAA ::1

> What�s the actual problem you are trying to solve though?

Problem is ParaDNS is failing on all lookups for this server.

export PARADNS_DEBUG=100

perl ./qpsmtpd-async -j 1 -v --user root --port 3000 --config-port=23000
16024 Loaded Qpsmtpd::Plugin::logging::warn=HASH(0x1401988)
16024 in config(plugins)
16024 config(plugins): hook returned (0, )
16024 trying to get config for plugins
16024 setting _config_cache for plugins to [resolve_sender_host dont_require_anglebrackets rcpt_simple te...@newswire.net ch...@newswire.net do...@newswire.net sup...@newswire.net
accou...@newswire.net sa...@newswire.net deb...@newswire.net maildir /cluster/clients/ivan-budimir %d/users/%l/Maildir] from get_qmail_config and returning it
16024 in config(plugin_dirs)
16024 config(plugin_dirs) returning () from cache
16024 Loading resolve_sender_host from ./plugins/resolve_sender_host
16024 resolve_sender_host hooking connect
16024 in config(plugin_dirs)
16024 config(plugin_dirs) returning () from cache
16024 Loading dont_require_anglebrackets from ./plugins/dont_require_anglebrackets
16024 dont_require_anglebrackets hooking rcpt_pre
16024 dont_require_anglebrackets hooking mail_pre
16024 in config(plugin_dirs)
16024 config(plugin_dirs) returning () from cache
16024 Loading rcpt_simple te...@newswire.net ch...@newswire.net do...@newswire.net sup...@newswire.net accou...@newswire.net sa...@newswire.net deb...@newswire.net from ./plugins/rcpt_simple
16024 rcpt_simple hooking rcpt
16024 in config(plugin_dirs)
16024 config(plugin_dirs) returning () from cache
16024 Loading maildir /cluster/clients/ivan-budimir %d/users/%l/Maildir from ./plugins/maildir
16024 maildir hooking queue
16024 Running as user root, group root
16024 Listening on 3000 with 1 children with poll()
Use of inherited AUTOLOAD for non-method Net::DNS::Resolver::MAX_ID() is deprecated at ./qpsmtpd-async line 157.

... waiting for connection ...

Generate a connection...

swaks -s localhost -p 3000 -t te...@example.com

Where -s $host can be anything, as nothing resolves.

... then when connection occurs...

Listen child making a Qpsmtpd::PollServer for 7.

16025 in config(plugins)
16025 config(plugins) returning (resolve_sender_host dont_require_anglebrackets rcpt_simple te...@newswire.net ch...@newswire.net do...@newswire.net sup...@newswire.net accou...@newswire.net

sa...@newswire.net deb...@newswire.net maildir /cluster/clients/ivan-budimir %d/users/%l/Maildir) from cache

>>> 100/2 [16025] dns lookup: Trying to resolve A: 127.0.0.1
>>> 100/2 [16025] dns lookup: NS Query: 127.0.0.1 (60240)

>>> DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0)

16025 (connect) running plugin: resolve_sender_host
16025 (connect) resolve_sender_host: DEBUG: ip=127.0.0.1 host=localhost domain=localhost
16025 Plugin resolve_sender_host, hook connect returned DECLINED,
16025 in config(smtpgreeting)
16025 config(smtpgreeting): hook returned (0, )
16025 trying to get config for smtpgreeting
16025 setting _config_cache for smtpgreeting to [Ready!] from get_qmail_config and returning it

Notice the there lines with prepended '>>>' which show ParaDNS failing.

[Ed McLain]

David,
I’ve been trying to find the code for the plugin resolve_sender_host but
haven’t had any luck - is that a custom plugin or can you send me the link
to it?

Just looking at the below:

>>> 100/2 [16025] dns lookup: Trying to resolve A: 127.0.0.1
>>> 100/2 [16025] dns lookup: NS Query: 127.0.0.1 (60240)
>>> DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0)
16025 (connect) running plugin: resolve_sender_host
16025 (connect) resolve_sender_host: DEBUG: ip=127.0.0.1 host=localhost
domain=localhost
16025 Plugin resolve_sender_host, hook connect returned DECLINED,

It appears that resolve_sender_host is trying to do an A query for the IP
127.0.0.1 vs a PTR which is what it should be doing.

―

Edward McLain
Director of Cloud Architecture
Solutions Architect, VCP, CDCDP
emc...@teklinks.com
d:205.314.6646
<http://www.teklinks.com/>

Please direct all support questions to our support team at
hos...@teklinks.com <http://mailto:hos...@teklinks.com/> or 205.314.6634.




-----Original Message-----
From: David Favor <da...@davidfavor.com>
Date: Friday, July 18, 2014 at 9:07 AM
To: Edward McLain <emc...@teklinks.com>
Cc: qpsmtpd <qps...@perl.org>
Subject: Re: qpsmtpd-async weird ParaDNS lookup problem

[Charlie Brady]

On Fri, 18 Jul 2014, David Favor wrote:

> > What's the actual problem you are trying to solve though?
>
> Problem is ParaDNS is failing on all lookups for this server.

...

> Notice the there lines with prepended '>>>' which show ParaDNS failing.

Perhaps you should be looking for assistance on a paraDNS support
channel...

[David Favor]

Ed McLain wrote:
> David,
> I占쏙옙ve been trying to find the code for the plugin resolve_sender_host but
> haven占쏙옙t had any luck - is that a custom plugin or can you send me the link

> to it?
>
> Just looking at the below:
>
>>>> 100/2 [16025] dns lookup: Trying to resolve A: 127.0.0.1
>>>> 100/2 [16025] dns lookup: NS Query: 127.0.0.1 (60240)
>>>> DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0)
> 16025 (connect) running plugin: resolve_sender_host
> 16025 (connect) resolve_sender_host: DEBUG: ip=127.0.0.1 host=localhost
> domain=localhost
> 16025 Plugin resolve_sender_host, hook connect returned DECLINED,
>
>
>
> It appears that resolve_sender_host is trying to do an A query for the IP
> 127.0.0.1 vs a PTR which is what it should be doing.
>

> 占쏙옙

>>> What占쏙옙s the actual problem you are trying to solve though?

I've removed this this plugin, as it has no relation to problem.

As mentioned, the exact same qpsmtpd setup works with prefork, which is what I'm
running till this is resolved. The problem looks to be how ParaDNS is interacting
with this setup.

Listen child making a Qpsmtpd::PollServer for 7.

25500 in config(plugins)
25500 config(plugins) returning (dont_require_anglebrackets rcpt_simple te...@newswire.net ch...@newswire.net do...@newswire.net sup...@newswire.net accou...@newswire.net sa...@newswire.net

deb...@newswire.net maildir /cluster/clients/ivan-budimir %d/users/%l/Maildir) from cache

100/2 [25500] dns lookup: Trying to resolve A: 127.0.0.1
100/2 [25500] dns lookup: NS Query: 127.0.0.1 (49469)

DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0)

25500 in config(smtpgreeting)
25500 config(smtpgreeting): hook returned (0, )
25500 trying to get config for smtpgreeting
25500 setting _config_cache for smtpgreeting to [Ready!] from get_qmail_config and returning it

And I do see what you mean... The line above...

100/2 [25500] dns lookup: Trying to resolve A: 127.0.0.1

does seem rather odd... I'll run the same test on a working qpsmtpd-async
system + see what ParaDNS debug spews...

100/2 [15241] dns lookup: Using nameserver 127.0.0.1:53
100/2 [15241] dns lookup: Using nameserver 68.233.248.187:53
100/2 [15241] dns lookup: Using nameserver 68.233.248.204:53
100/2 [15241] dns lookup: Trying to resolve A: 127.0.0.1
100/2 [15241] dns lookup: NS Query: 127.0.0.1 (29444)
100/1 [15241] dns lookup: Query mismatch for id: 29444. 127.0.0.1 ne 1.0.0.127.in-addr.arpa

DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0)

Well... records still look very odd... even on working server... How
ParaDNS + Danga::Socket is being called seems wrong.

The only difference I noticed between the two servers was an entry
for 127.0.0.1 in /etc/resolv.conf so I added this in the broken server.

Same behavior.

Also check ParaDNS + Danga::Socket versions. Same on both servers.

A head scratcher. Guess I'll add some debug code to Danga::Socket to
see if that helps.

[David Favor]

The symptom of this problem is qpsmtpd-async just hangs
forever in the HELO part of SMTP transaction.

After putting reams of debug code in ParaDNS.pm + qpsmtpd,
the fix I came up with is a minor change to...

lib/Qpsmtpd/PollServer.pm

In start_conversation() no nameserver list specified. I have
servers where qpsmtpd-async has been running for years + have
no clue how this code could ever work.

My is to add a list of nameservers to query...

my @nameservers = qw/127.0.0.1 8.8.8.8 8.8.4.4/;

Then add to the ParaDNS->new() constructor...

nameservers => \@nameservers

Now everything works like a charm.

Debug shows...

00/2 [1394] dns lookup: Nameservers set to: 127.0.0.1 8.8.8.8 8.8.4.4
100/2 [1394] dns lookup: Using override nameserver 127.0.0.1:53
100/2 [1394] dns lookup: Using override nameserver 8.8.8.8:53
100/2 [1394] dns lookup: Using override nameserver 8.8.4.4:53
100/2 [1394] dns lookup: Trying to resolve A: 127.0.0.1
100/2 [1394] dns lookup: NS Query: 127.0.0.1 (39546)
100/1 [1394] dns lookup: Query mismatch for id: 39546. 127.0.0.1 ne 1.0.0.127.in-addr.arpa

DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0)

Last line still seems bogus to me + at least qpsmtpd-async delivers
mail now.

If anyone has an idea of how ParaDNS comes up with a nameserver list,
when none is specified... let me know.

Thanks.

[Matt Sergeant]

Do you have ParaDNS::XS installed? If not, try installing it. It's
generally likely to be better than plain ParaDNS (and will be used
automatically if it's installed).

[David Favor]

Matt Sergeant wrote:
> Do you have ParaDNS::XS installed? If not, try installing it. It's
> generally likely to be better than plain ParaDNS (and will be used
> automatically if it's installed).

There is no ParaDNS::XS on cpan... At least nothing returned for...

http://search.cpan.org/search?query=paradns&mode=all

Unsure what this has to do with setting a list of servers for ParaDNS
to query. Whether pure perl or XS, likely there still has to be a set
of servers to query.

The fix I'm using right now is a slight patch to Qpsmtpd/PollServer.pm
which simply gives ParaDNS a list of servers to check...

my @nameservers = qw/127.0.0.1 8.8.8.8 8.8.4.4/;

my $obj = ParaDNS->new(
nameservers => \@nameservers,
finished => sub { $self->continue_read(); $self->run_hooks("connect"); },
# NB: Setting remote_info to the same as remote_host
callback => sub { $conn->remote_info($conn->remote_host($_[0])); },
host => $ip,
);

This works like a charm.

Reading through the ParaDNS code, it's unclear to me how qpsmtpd can work
at all without a list of servers for ParaDNS to query.

[Matt Sergeant]

To be fair I don't do perl coding any more, so I'm happy for someone
to take over ParaDNS maintainence.

ParaDNS-XS is in my SVN server. Happy to share that with anyone who wants
it. It's basically ParaDNS using adns backend rather than Net::DNS. It's a
bit hacky and probably doesn't work 100%

Matt.

[Kaare Rasmussen]

Hi Matt

How about putting it on Github? Then anybody can look and decide if they
want to continue work on it and perhaps make it cpan -ready.

[David Favor]

This was meant as a qpsmtpd bug to be fixed. I can't see how this
ever worked, without a list of DNS servers to pull from.

ParaDNS is working fine.