Invalidate upstream DNS cache

[Jon Keys]

We're using an AWS elastic loadbalancer to balance upstream traffic. It is useful for dynamically adding / removing servers in a central location and supports some nice features (e.g. connection draining, auto scaling, etc...).

The problem we're having is that nginx only resolves the IP of our AWS loadbalancer at startup and then caches the IP forever (until nginx reboot). Since the AWS loadbalancer IP can change anytime this is obviously a big problem.

I found this issue / feature request on the ngx_lua_upstream module which I think could help solve our problem. However, it isn't currently implemented.

Does anyone know of any other method to invalidate an upstream DNS cache / force nginx to honor DNS TTL responses?

I'm aware of the upstream "resolve" option in nginx+ but I'd rather not pay nginx+ license fees just for this option. Also, I'd much rather use the exact nginx version that has been thoroughly tested by agentzh as part of an OpenResty release.

[Yichun Zhang (agentzh)]

Hello!

On Mon, Mar 2, 2015 at 9:52 AM, Jon Keys wrote:
> I found this issue / feature request on the ngx_lua_upstream module which I
> think could help solve our problem. However, it isn't currently implemented.
>

Yes, adding an API to ngx_lua_upstream so as to dynamically change the
peer address in the upstream configuration data structures is the
right way to go.

I'm going to review that pull request at some point.

Regards,
-agentzh

[Brooks Cunningham]

Hello!

I am facing a similar issue when using the upstream feature and proxying to ELBs. Is there an update on this topic?

Cheers,

Brooks

[Jon Keys]

We ended up writing our own upstream resolver using `balancer_by_lua`. It uses a master-client model to lookup IP's only once in a master context and then cache the IP's according to the DNS TTL. Then clients in a worker context can then quickly fetch the cached IP's from the master (again, always obeying the DNS TTL values).

It actually has some nice features and it is well tested. I might have open-sourced it before I just thought we were the only ones with this issue :-)

If others are interested we could publish to GitHub or somewhere public (we currently use a private GitLab server with GitLab CI for automated testing).

If we do that then we could probably publish to OPM as well.

One thing worth noting: it currently only supports IPv4 (i.e. A records)

[Brooks Cunningham]

That would be amazing Jon! Using the upstream module is pretty great for performance since NGINX doesn't have to go through the full TCP or TLS build up and tear down for nearly every new connection. However, finding out that my site is 100% down because the ELB A records have changed makes it tough to utilize the upstream module.

I am very interested in the fix even with the IPv4 only support.

Thanks Jon!

Brooks

[Jon Keys]

I just published to GitHub: https://github.com/jkeys089/lua-resty-resolver

If I get a chance this week I'll publish to OPM as well.

I hope it helps!

[Jon Keys]

I just went ahead and published to OPM -- I forgot how easy it is :-D

It is available at: jkeys089/lua-resty-resolver

[Brooks Cunningham]

I really do appreciate the contribution. I'll take a look.

Thanks!

Brooks

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bilal Ahmad]

Hi Guys,

Can you share any link to a working example of the resolver for ELBs with lua-resty-resolver or lua-upstream-nginx-module? It would be really helpful. Thanks,
Bilal