Strange: My Bind (8.4.6) freezes randomly
Santiago Romero
I have a strange problem with bind 8.4.6 in a Compaq Proliant ML350
running Debian Sarge Linux as a primary DNS Server. The server has been
running with no problems until a couple of days ago, when it started to
"freeze" (Only the DNS service) and stop answering queries.
# named -v
named 8.4.6-REL-NOESW Wed Aug 2 14:56:29 CEST 2006
English is not my native tongue, so I'm going to try to explain it
and give all the info or facts I could.
I hope any of you find my problem challenging enough to give me a
hand and help me to find where's the problem...
The machine:
============
I have a machine (brand new hardware) installed 5 months ago and
running as a DNS+DHCP+RADIUS server. I haven't had any incidence since
we installed it. We keep it updated (kernel, security updates) and
the machine didn't need to be rebooted since we installed it.
The problem:
============
Suddenly, since 4th January 2007 we have strange problems with the
named/bind daemon. It's working perfectly until randomly it freezes.
DNS querys continue arriving to the machine but bind doesn't resolve
them. I can't even solve queries launched from the own machine:
(into the DNS server machine):
# dig www.google.com @My_IP
; <<>> DiG 9.2.4 <<>> www.google.com @My_IP
;; global options: printcmd
;; connection timed out; no servers could be reached
I only can "kill -9" the named process and start it again, and it
works again correctly. I've have about 7 DNS machines and I've been
maintaining them since 1999 and I never found such a "freezing".
Bind of this machine never had any problem UNTIL 4th January, so
I'm starting to think that maybe it's any kind of new (or old)
DoS bug. Bind doesn't log anything strange to daemon.log or
named.run (started with -d 4), it just stops logging (and working).
As the server has configured it's own IP address as primary ns in
resolv.conf, when this happens I can't log into the console or
execute any su or sudo commando, so I have a permanent root console
session oppened to solve it (kill -9, /etc/init.d/bind start).
How can I reproduce it
======================
2 ways:
1.- While trying to detect the problem I found also the following:
The SIGWINCH signal is suppossed to switch on the "query logging" to
syslog. If I do a kill -s SIGWINCH `pidof named`, a few seconds/minutes
later I suffer the same "named freezing" that I suffer randomly.
2.- I wrote an script that does about 200 DNS querys to DNS blacklists
using the rblcheck command. Something like:
for i in `seq 1 $j`;
do
timeout -s TERM 10 rblcheck -t -c -s sbl-xbl.spamhaus.org \
-s bl.spamcop.net -s list.dsbl.org -s dnsbl.ahbl.org \
-s combined.njabl.org -s dnsbl.sorbs.net My_IP
(...)
done
Sometimes, when executing the script, bind freezes and "timeout"
command kills the queries because they are not working:
rbl.sh: line 19: 20709 Terminated timeout -s TERM 10 rblcheck -t \
-c -s sbl-xbl.spamhaus.org -s bl.spamcop.net -s list.dsbl.org -s \
dnsbl.ahbl.org -s combined.njabl.org -s dnsbl.sorbs.net My_IP
Things I did previously to contact this list
============================================
- I compiled latest bind version (8.4.7) and it happens also.
- I copied "named" binary from another machine (8.4.6 compiled in
a Woody machine running in my Sarge machine) and it happens a lot
less times. ***But the SIGWINCH signal also freezes the DNS***.
- The Sigwinch signal works perfectly in all my other DNS servers,
it just start logging queries until I disable it.
- People from our Networking Staff put an sniffer in the switch
port the machine is connected to, and it just found that when the
freeze happens, the machine continues receiving DNS querys but
it doesn't answer any of them.
- I can't find any kind of error messages on system or kernel
logs, and all other services are working, it's just the DNS
service which fails.
Any idea?
--
Santiago Romero
David Nolan
--On Wednesday, January 10, 2007 17:21:38 +0100 Santiago Romero
<sro...@servicom2000.com> wrote:
> Suddenly, since 4th January 2007 we have strange problems with the
> named/bind daemon. It's working perfectly until randomly it freezes.
> DNS querys continue arriving to the machine but bind doesn't resolve
> them. I can't even solve queries launched from the own machine:
Were any changes made to the system on or around that date?
This sounds a lot like a problem I had a while back which turned out to be
a kernel bug in Linux 2.6.14 where a zero length UDP packet would generate
an error on the socket, resulting in bind no longer processing that socket.
Two symptoms I noticed in my case were:
- queries to other IP addresses on the same server would work, including
localhost. What happens if you dig @127.0.0.1?
- strace'ing the named process I saw it iterating over the sockets calling
select on each one, until the failure point, at which point the primary udp
socket was no longer being selected. What do you see if you strace the
process?
(In my case, upgrading to 2.6.16 solved the problem.)
-David Nolan
Network Software Designer
Computing Services
Carnegie Mellon University
Santiago Romero
>> Suddenly, since 4th January 2007 we have strange problems with the
>> named/bind daemon. It's working perfectly until randomly it freezes.
>> DNS querys continue arriving to the machine but bind doesn't resolve
>> them. I can't even solve queries launched from the own machine:
>
> Were any changes made to the system on or around that date?
No changes.
Machine had a 4 month's uptime with no incidences (we monitorize all
in the machine, disk, mem, cpu, ports...).
> - queries to other IP addresses on the same server would work, including
> localhost. What happens if you dig @127.0.0.1?
No, it doesn't work:
/var/named# dig www.google.com @localhost
; <<>> DiG 9.2.4 <<>> www.google.com @localhost
;; global options: printcmd
;; connection timed out; no servers could be reached
I'm going to send a new message including some tcpdump
data in a minute.
Thanks for your answer, anyway :)
--
Santiago Romero
Santiago Romero
More info... I'm running bind in debug mode, and I found:
1.- Strange lines I've never seen before:
==========================================
datagram from [212.101.67.30].2169, fd 20, len 45
XX+/212.101.67.30/ono.es.dsn.rfc-ignorant.org/A/IN
req: nlookup(ono.es.dsn.rfc-ignorant.org) id 55678 type=1 class=1
req: found 'ono.es.dsn.rfc-ignorant.org' as 'dsn.rfc-ignorant.org' (cname=0)
evSetTimer(ctx 0x80e7768, func 0x806126c, uap (nil), due
1168449414.000000000, inter 0.000000000)
forw: forw -> [208.201.249.252].53 ds=20 nsid=8658 id=55678 8ms retry 4sec
datagram from [212.101.67.30].2170, fd 20, len 55
XX+/212.101.67.30/11.227.42.62.ipwhois.rfc-ignorant.org/TXT/IN
req: nlookup(11.227.42.62.ipwhois.rfc-ignorant.org) id 55679 type=16 class=1
req: found '11.227.42.62.ipwhois.rfc-ignorant.org' as
'ipwhois.rfc-ignorant.org' (cname=0)
NS 'localhost.rfc-ignorant.org' Bogus LOOPBACK A RR
forw: nslookup reports danger
(Bogus LOOPBACK A RR? nslookup reports danger?)
I can't see any other kind of errors, only IPV6 queries (I don't know
if they can be the problem). I just "tail -f" the named.run file, and
when it stops, I know that bind is freeze.
The last time, the logging ended with:
datagram from [212.101.64.35].53, fd 22, len 38
qfindid(41653) -> 0x80ea440
Response (SYSTEM NORMAL -) nsid=41653 id=0
++ ns_resp(rcvd) ++ (38)
;; ->>HEADER<<- opcode: NOTIFY, status: NOERROR, id: 41653
;; flags: qr aa ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; anticcolonial.com.es, type = SOA, class = IN
-- ns_resp(rcvd) --
stime 1168452153/915606 now 1168452153/958910 rtt 43
Received NOTIFY answer (AA) from 212.101.64.35 for "dominio.com.es IN SOA"
qremove(0x80ea440)
unsched(0x80ea440, 0)
evSetTimer(ctx 0x80e7768, func 0x806126c, uap (nil), due
1168452156.000000000, inter 0.000000000)
ns_freeqry(0x80ea440)
datagram from [212.101.64.4].53, fd 22, len 24
ns_req(from [212.101.64.4].53)
(and bind freezes)
If I wait, then some queries and answers are logged in named.run, maybe
queued queries, because it doesn't answer to new queries...
I'm totally lost, this seems a kernel/bind/DoS problem, but I don't know
how to detect it.
--
Santiago Romero
Departamento de Sistemas
sro...@servicom2000.com
Av. Primado Reig 189, entlo
46020 Valencia - Spain
Telf. (+34) 96 332 12 00
Fax. (+34) 96 332 12 01
http://www.servicom2000.com
Santiago Romero
I've noticed that AXFR transfers work when bind is frozen! :?
Jan 10 19:21:29 beauty named[28084]: approved AXFR from
[212.101.64.35].52744 for "xxxx.com"
Jan 10 19:21:34 beauty named[28084]: zone transfer (AXFR) of "lladro.com"
(IN) to [212.101.64.35].52744 serial 2006101903
Jan 10 19:21:39 beauty named[28084]: approved AXFR from
[212.101.64.35].52745 for "yyyy.com"
Jan 10 19:21:44 beauty named[28084]: zone transfer (AXFR) of
"maxaminit.biz" (IN) to [212.101.64.35].52745 serial 2006051501
Jan 10 19:21:49 beauty named[28084]: approved AXFR from
[212.101.64.35].52746 for "ppppp.pt"
Messages about AXFR transfer appear slowly (slower than when it's running
OK) but appear.
It seems that bind continues working with TCP traffic, but UDP traffic
stops working...
I'm running latest Debian Stable Kernel (2.4.27-3) ... I think I'm going
to download the latest vanilla kernel from kernel.org and try .
Thanks.
Florian Weimer
> I'm running latest Debian Stable Kernel (2.4.27-3) ... I think I'm going
> to download the latest vanilla kernel from kernel.org and try .
By the way, it is strongly recommended to run Linux 2.6 on Debian
stable, together with BIND 9.
We haven't been clear on the BIND 9 part, I suppose. I'll see if we
can add something to the release notes for etch.
Santiago Romero
Latest news.
Maybe it's a bug between **** bind and sysklog **** ?
I noticed that:
- If I stop sysklogd, bind doesnt (*) freezes.
I can also send SIGWITH to named, and it doesn't freeze.
- If I configure bind to log as "default { null; }", bind doesnt' freezes.
- If I set bind to log as "file /var/log/named.log", bind doesn't freezes.
I can also send SIGWITH to named, it logs all the queries,
and it doesn't freeze.
- If I configure bind to log using syslog, I start named, send the
SIGWICH signal, and it freezes 5-10-30-60 seconds later. ALWAYS.
(*) doesnt freezes = I tested it for 3 hours, with no problems.
sysklogd version (Debian Sarge -stable- up to date):
ii sysklogd 1.4.1-17 System Logging Daemon
Any idea?
Mark Andrews
syslogd want's to resolve a address. It blocks until the
resolver returns a answer / timesout. Named wants to log
to syslogd. syslog() blocks until syslogd reads the socket.
When will OS developers
a) restore syslog()'s non blocking behaviour
and / or
b) not translate IP addresses to names
Syslog was originally written on top of UDP. UDP *does*
*not* block.
Mark
--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: Mark_A...@isc.org
Santiago Romero
>> I noticed that:
>>
>> - If I stop sysklogd, bind doesnt (*) freezes.
>> I can also send SIGWITH to named, and it doesn't freeze.
>> - If I configure bind to log as "default { null; }", bind doesnt' freezes.
>> - If I set bind to log as "file /var/log/named.log", bind doesn't freezes.
>> I can also send SIGWITH to named, it logs all the queries,
>> and it doesn't freeze.
>> - If I configure bind to log using syslog, I start named, send the
>> SIGWICH signal, and it freezes 5-10-30-60 seconds later. ALWAYS.
>
> resolver returns a answer / timesout. Named wants to log
> to syslogd. syslog() blocks until syslogd reads the socket.
>
> When will OS developers
>
> a) restore syslog()'s non blocking behaviour
> and / or
> b) not translate IP addresses to names
>
> Syslog was originally written on top of UDP. UDP *does*
> *not* block.
What do you mean? (Sorry, I don't understand what are you trying
to tell me).
Do you mean it's an OS or sysklog bug?
The strange is that this problem has never happened until it
started last 4th January 2007. Since that day, our bind freezed
10-15 times in a day.
Do you think it's a problem related with having heavy traffic
load? Malformed queries? OS bug?
Cihan Subasi (Garanti Teknoloji)
I have a similar problem with 9.2.6-P2....
* Santiago Romero:
This message and attachments are confidential and intended solely for the individual(s) stated in this
message. If you received this message although you are not the addressee, you are responsible to keep the
message confidential. The sender has no responsibility for the accuracy or correctness of the
information in the message and its attachments. Our company shall have no liability for any changes
or late receiving, loss of integrity and confidentiality, viruses and any damages caused in
anyway to your computer system.
Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve gizlidir. Bu mesajin muhatabi
olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune
uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde yer alan bilgilerin dogrulugu ve
guncelligi konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz
mesajin ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin
korunamamasindan, virus icermesinden ve bilgisayar sisteminize verebilecegi herhangi bir zarardan
sorumlu tutulamaz.
Stefan Puiu
I don't think I can explain things more clearly than Mark did - it's
basically a deadlock between two applications (BIND and sysklogd)
which can be broken in the two ways that Mark detailed. However, his
rant was aimed at OS/sysklogd programmers. You are probably an admin
which can't do much about the sorry state of syslog()/sysklogd.
However, one option you have is to go with syslog-ng, which provides
b) through a configuration file option:
http://www.balabit.com/products/syslog_ng/
I haven't used it in production (I'm not an admin), so you might want
to do a bit of research first. I've been told that it's quite good.
Disclaimer: I'm not involved in any way with syslog-ng. Somebody has
recommended it to me, too.
Stefan.
Santiago Romero
> http://www.balabit.com/products/syslog_ng/
>
> I haven't used it in production (I'm not an admin), so you might want
> to do a bit of research first. I've been told that it's quite good.
I changed from syslog to syslog-ng and named stopped freezing :-?
Maybe the bind package in Debian should have syslog-ng as
a dependency...
Thanks :)
Stefan Puiu
> I changed from syslog to syslog-ng and named stopped freezing :-?
If you have this set to no (options section), it should be ok:
---
# enable or disable DNS usage
# syslog-ng blocks on DNS queries, so enabling DNS may lead to
# a Denial of Service attack
# (default is yes)
use_dns(no);
--
That seems to be the case on Ubuntu edgy, and they often use the
Debian defaults.
>
> Maybe the bind package in Debian should have syslog-ng as
> a dependency...
If it wasn't reported as a bug in Debian, it should be. However, not
sure the Debian packagers would understand the issue... I'll look for
it in their BTS, they should at least know about it...
Stefan.