Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

dig +trace = Bad Referral orBad Horizontal referral

2,467 views
Skip to first unread message

project722

unread,
Sep 16, 2016, 11:12:20 AM9/16/16
to bind-...@lists.isc.org
I have an interesting problem. I started noticing that when I do a dig +trace against one of the domains we are authoritative for, we get errors from our nameservers for "Bad Referral" and you can see where it forwarded the request back up the namespace tree instead of giving the answer. Unfortunately I am unable to reproduce this problem at the moment. However I can reproduce the Bad (HORIZONTAL) referral. Basically once the query is referred to our name server I see this:

;; BAD (HORIZONTAL) REFERRAL
;; Received 187 bytes from x.x.x.x#53(ns.example.com in 2 ms

Sometimes this loops until I reach the "too many lookups" error and it just gives up, or it will stop and try our other server which then gets an answer. Here's the interesting part. If I was to perform a simple dig A lookup against the server that continusouly failed the trace, I get an answer and a correct one. If I then turn around and do the dig +trace against that same query again it never fails again. Its almost like records are being cached somewhere and after it expires, then you will start to see the message again. Can anyone help me figure out what id going on here? Here is info about our environment. 

1)RHEL 6 running BIND 9.8.2
2)Public facing auhtoritative Master and Slave Server."
3)Servers are setup in a "view" configuration. (dual-view - one for "internal" one for external)
4)It seems like we just started to see these oddities after implementing views. 
5)Queries that hit the internal view are forwarded to the external view for zones not contained in the internal view. We use the loopback IP to acheive this.
6) These authoritative servers are also setup to answer non-authoritative queries with recusrion through the use of the recusion statement and root "hint" zone. 

Here is our setup simplified. 

Master server:

acl ext_trusted {
x.x.x.x; // trusted net external
}; // end of "ext_trusted" ACL definition.

ccl int_trusted {
x.x.x.x; // trusted internal
}; // end of ACL for "int_trusted"  


options {
        directory "/var/named";
        recursive-clients 30000;
        tcp-clients 2000;
        check-names master ignore;
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named.stats.txt";
        memstatistics-file "/var/named/data/named_mem/stats.txt";
        zone-statistics yes;
        cleaning-interval 30;
   

// Listen on ipv4: // Adding localhost for view forwarding
listen-on port 53 { x.x.x.x; 127.0.0.1; };

// And, also listen on ipv6:
// loopback is required for view forwarding do not remove
listen-on-v6 { ::1; x.x.x.x; };

// Enforce the Customer DNS Query ACL Defined Above:
allow-query { ext_trusted; int_trusted; };

// Enforce the Zone-Transfer ACL Defined Above:
// This will now be defined on a "per view" basis
//allow-transfer { other_ns; };

};


key "internal" {
 algorithm HMAC-SHA512;
secret "xxxxxxxxx";
};

key "external" {
algorithm HMAC-SHA512;
secret "xxxxxxxx";
};

 view "internal" {
        match-clients { !key external; int_trusted; key internal; };

        //IP of slave server
        server x.x.x.x {
        keys { internal; };
};

        server x.x.x.x {
        keys { internal; };
};

        also-notify { //slave servers go here
        x.x.x.x; x.x.x.x; 
};

        allow-transfer { key internal; local_ns; int_ns; };
        empty-zones-enable no;
        server fe80::/16 { bogus yes; };
        server 0.0.0.0/8 {bogus yes; };

        zone "example.org" {
        type master;
        file "db.eample.org";
        allow-query { int_trusted; };
};

        forwarders {
        // forward to external view //
        127.0.0.1; ::1; 
};

        forward only;

};

view "external" {
      match-clients { any; };

//IP of slave server
      server x.x.x.x {
      keys { external; };
};

      server x.x.x.x {
      keys { external; };
};

also-notify { //IP address of slave server
       x.x.x.x; x.x.x.x;
};

allow-transfer { key external; ext_ns; local_ns; };
server fe80::/16 { bogus yes; };
server 0.0.0.0/8 {bogus yes; };
empty-zones-enable yes;
recursion yes;
allow-recursion { any; };

zone "." {
         type hint;
         file "/var/named/named.ca";
};


zone "example.org" {
        type master;
        file "db.eampleout.org";
        allow-query { any; };
};

zone "example.com" {
        type master;
        file "db.eample.com";
        allow-query { any; };
};
      
};

  
Slave server config:
     

acl ext_trusted {
x.x.x.x; // trusted net external
}; // end of "ext_trusted" ACL definition.

ccl int_trusted {
x.x.x.x; // trusted internal
}; // end of ACL for "int_trusted"  

options {
        directory "/var/named/slaves";
        recursive-clients 30000;
        tcp-clients 2000;
        check-names master ignore;
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt"; 
        cleaning-interval 30;
       
// Listen on ipv4:
// Change this to the proper IP address if you ever switch back!
// loopback is required for view forwarding do not remove
listen-on port 53 { 127.0.0.1; x.,x.x.x;; };

// And, also listen on ipv6:

// Configure ipv6 before uncommenting this:
// loopback is required for view forwarding do not remove
listen-on-v6 port 53 { ::1; x,x.x.x; ;

// Enforce the "trusted" ACL defined at the begining of this config file: 
allow-query { ext_trusted; int_trusted; };

};


key "internal" {
 algorithm HMAC-SHA512;
secret "xxxxxxxxx";
};

key "external" {
algorithm HMAC-SHA512;
secret "xxxxxxxx";
};

view "internal" {
        match-clients { !key external; int_trusted; key internal; };

        //IP of master server
        server x.x.x.x {
        keys { internal; };
};

        server x.x.x.x. {
        keys { internal; };
};
        allow-transfer { key internal; local_ns; int_ns; };
        transfer-source x.x.x.x.; 
        empty-zones-enable no;
        server fe80::/16 { bogus yes; };
        server 0.0.0.0/8 {bogus yes; };

        zone "example.org" {
        type slave;
        file "db.example.org";
        masters { x.x.x.x; x.x.x.x; };
        allow-query { int_trusted; };
};

        forwarders {
        // forward to external view // 
        127.0.0.1; ::1; 
};

        forward only;
};

view "external" {
      match-clients { any; };

     //IP of master server
     server x.x.x.x {
     keys { external; };
};
     //IPv6
     server x.x.x.x. {
     keys { external; };
};

allow-transfer { key external; ext_ns; local_ns; };
transfer-source x.x.x.x; 
server fe80::/16 { bogus yes; };
server 0.0.0.0/8 {bogus yes; };
empty-zones-enable yes;

recursion yes;
allow-recursion { any; };

zone "." {
        type hint;
//      file "cache";
        file "/var/named/named.ca";
};

 zone "example.org" {
        type slave;
        file "db.exampleout.org";
        masters { x.x.x.x; x.x.x.x; };
        allow-query { any; };
};

zone "example.com" {
        type master;
        file "db.eample.com";
        allow-query { any; };
};
      
};



Matthew Pounsett

unread,
Sep 20, 2016, 9:59:05 AM9/20/16
to project722, bind-...@lists.isc.org
On 16 September 2016 at 11:12, project722 <proje...@gmail.com> wrote:
I have an interesting problem. I started noticing that when I do a dig +trace against one of the domains we are authoritative for, we get errors from our nameservers for "Bad Referral" and you can see where it forwarded the request back up the namespace tree instead of giving the answer. Unfortunately I am unable to reproduce this problem at the moment. However I can reproduce the Bad (HORIZONTAL) referral. Basically once the query is referred to our name server I see this:

;; BAD (HORIZONTAL) REFERRAL
;; Received 187 bytes from x.x.x.x#53(ns.example.com in 2 ms

A horizontal referral is when one authoritative zone (the parent) delegates a subdomain to a server that responds out of the same parent zone, rather than a subzone.  The DNS is an inverted tree structure, and delegations are always supposed to be "down" the tree toward the leaves.  If a delegation ends up being across, then you get a horizontal referral error.

Since you obfuscated your configuration nobody is going to be able to provide you with specific advice on where the problem is.  If you can find the error in your authoritative data (or share which zone is giving you problems so that someone here can point it out) that should clear up your issue.

project722

unread,
Sep 20, 2016, 12:51:50 PM9/20/16
to Matthew Pounsett, bind-...@lists.isc.org
I've reverted my configuration back to before we started using views. But, if this was a delegation issue, wouldn't we expect to see it regardless of using views or not? Works fine without views. 

Matthew Pounsett

unread,
Sep 20, 2016, 2:07:01 PM9/20/16
to project722, bind-...@lists.isc.org
On 20 September 2016 at 12:50, project722 <proje...@gmail.com> wrote:
I've reverted my configuration back to before we started using views. But, if this was a delegation issue, wouldn't we expect to see it regardless of using views or not? Works fine without views. 

It's entirely possible you introduced some error in your zone structure when you changed configurations for views.  Again, it's impossible to tell since you've (poorly) substituted all of the zone names and file name in your configuration.   If you want help from someone in the list, you're going to have to share details of your configuration.
0 new messages