DNS anycast node monitor

[Hillary Nelson]

Currently we have about 20 DNS servers sit behind two pairs of F5 LTM on 
campus, the two pairs of F5s using router injection for DNS virtual
addresses. This setup is costly and we are trying to use direct anycast between router and server instead, with quagga and bgp.

The decision of advertise/withdraw route seems to be most critical one. I
guess we'll need two monitors, one on the server, another like nagios 
monitor from a remote system.  I know there are people doing this for many year, wonder if

there are working script that would like to share.

Thanks!

Hillary

[Eli Heady]

We're considering doing something similar, but have nothing in the way
of scripts to offer.

I'm curious what you mean by 'router injection for DNS virtual
addresses'. Off topic for this list, sorry, but are you meaning that
you're currently using dynamic routing (BGP/OSPF/RIP) in the F5 TMM?

This is another option we are considering and I'm curious whether any
issues apart from cost have caused you to consider alternatives.

Off list replies welcome.

Thanks!
Eli

> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to
> unsubscribe from this list
>
> bind-users mailing list
> bind-...@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users

[Anand Buddhdev]

On 09/04/15 16:50, Hillary Nelson wrote:

Hi Hillary,

> Currently we have about 20 DNS servers sit behind two pairs of F5 LTM on
> campus, the two pairs of F5s using router injection for DNS virtual
> addresses. This setup is costly and we are trying to use direct anycast
> between router and server instead, with quagga and bgp.

If you merely want to announce a route from the DNS server to the
router, and don't need to receive routes and insert them into the
server's routing table, then consider ExaBGP. It's great for simply
announcing prefixes via BGP.

We use this setup at the RIPE NCC and it works rather well.

> The decision of advertise/withdraw route seems to be most critical one. I
> guess we'll need two monitors, one on the server, another like nagios
> monitor from a remote system. I know there are people doing this for many
> year, wonder if
> there are working script that would like to share.

We run ExaBGP under the CentOS 6 upstart supervisor. ExaBGP's upstart
script is configured to stop it on certain events. We use another
upstart script to monitor the DNS server (with queries using dig), and
if the DNS server doesn't respond, then an event is emitted causing
exabgp to die, and withdraw the route to that server. The client almost
doesn't notice this.

You can also run all this under systemd if you wish, with its
dependencies. Or use monit, which can also let you define dependencies,
and stop or start services under certain condition. I don't have all the
URLs handy, but I'm sure you can search for all these things.

Regards,

Anand Buddhdev
RIPE NCC

[Frank Even]

Quagga works fine as well.

[Phil Mayers]

On 09/04/15 15:50, Hillary Nelson wrote:
> Currently we have about 20 DNS servers sit behind two pairs of F5 LTM on
> campus, the two pairs of F5s using router injection for DNS virtual
> addresses. This setup is costly and we are trying to use direct anycast
> between router and server instead, with quagga and bgp.

Consider using exabgp. It can run scripts as a sub-process that echo
"advertise" and "withdraw" commands or toggle the status of a "watchdog"
which can be trivially used to do this kind of service check-based route
advertisement, and you don't have to actually change the OS routing
table (and incur an rndc reconfig) which is win IMO.

Example would be something like:

neighbour x.x.x.x {
peer-as ...
...

static {
route x.x.x.x/32 next-hop ... watchdog blah;
}
process foo {
run /usr/local/bin/...
}
}

...and your script would be:

#!/bin/sh

VIP="<the IP bind is anycasted on>

while true; do
# ensure we're talking to the local virtual IP
dig -b 127.0.0.1 @$VIP example.com soa >/dev/null 2>&1
if [ $? -eq 0 ]
then
echo announce watchdog blah
else
echo withdraw watchdog blah
fi
sleep 1
done

[Bob Harold]

We use ExaBGP to query the DNS process locally and advertise/withdraw routes.

We use BFD between the server and the router to detect link failure (or server crash):
The BFD (Bidirectional Forwarding Detection) protocol is designed to provide
rapid detection of communication failures between two devices.  Routing
protocols can use BFD state to rapidly (< 1 second) withdraw routes in the
event of a failure, without having to wait for a routing protocol timeout
(3 minutes by default for BGP).

Seems to work well.

--
Bob Harold