how does grpc handle DNS address changes?

1,826 views
Skip to first unread message

Yaz Saito

unread,
Aug 17, 2016, 4:49:21 PM8/17/16
to grpc-io
Assume we have foo.example.com, whose value changes unpredictably.

1. Initially, the A record of foo.example.com points to, say 1.2.3.4.
2. Start a grpc client, with the server pointed at foo.example.com:10000. The client fails to
  connect since there's nothing running at 1.2.3.4:10000.
3. Start a grpc server at an IP address 1.2.3.5:10000
4. Update the DNS server so that foo.example.com points to 1.2.3.5.

In this situation, the grpc client never seems to discover the server at 1.2.3.5, even when I set the DNS TTL very short.

Am I diagnosing the problem correct? And if so, is there a way to work around?

--
yaz

raut...@gmail.com

unread,
Aug 18, 2016, 7:15:08 PM8/18/16
to grpc.io
Steps 4 and 5 should be interchanged I think.

What does host foo.example.com results in, on your computer?

Are you running your code on a Linux box or some other environment?

Thanks,

Abhishek Parmar

unread,
Aug 18, 2016, 7:20:34 PM8/18/16
to raut...@gmail.com, grpc.io
(Replying for Yaz, he is on vacation)

On Thu, Aug 18, 2016 at 4:15 PM, <raut...@gmail.com> wrote:
Steps 4 and 5 should be interchanged I think.

You mean 3 and 4 should be swapped? I don't think it matters.
 

What does host foo.example.com results in, on your computer?

Are you running your code on a Linux box or some other environment?

It is running on linux.
 

Thanks,

On Wednesday, 17 August 2016 13:49:21 UTC-7, Yaz Saito wrote:
Assume we have foo.example.com, whose value changes unpredictably.

1. Initially, the A record of foo.example.com points to, say 1.2.3.4.
2. Start a grpc client, with the server pointed at foo.example.com:10000. The client fails to
  connect since there's nothing running at 1.2.3.4:10000.
3. Start a grpc server at an IP address 1.2.3.5:10000
4. Update the DNS server so that foo.example.com points to 1.2.3.5.

In this situation, the grpc client never seems to discover the server at 1.2.3.5, even when I set the DNS TTL very short.

Am I diagnosing the problem correct? And if so, is there a way to work around?

--
yaz

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/f20ea25a-0487-4907-adb4-59f58791aff5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
-Abhishek

Craig Tiller

unread,
Aug 18, 2016, 7:38:40 PM8/18/16
to Abhishek Parmar, raut...@gmail.com, grpc.io
The behavior is supposed to be that we periodically (with exponential back-off) retry resolution and then connection until we get a successful connection.

We currently rely on getaddrinfo to do the resolution, so we don't get any insight into the TTL on the dns entry - that's something we'll pick up soon though (with an optional embedding of c-ares).

Could you add a log line to blocking_resolve_address_impl() in resolve_address_posix.c and verify that we are in fact repeatedly calling this?

On Thu, Aug 18, 2016 at 4:20 PM Abhishek Parmar <abhi...@carbon3d.com> wrote:
(Replying for Yaz, he is on vacation)

On Thu, Aug 18, 2016 at 4:15 PM, <raut...@gmail.com> wrote:
Steps 4 and 5 should be interchanged I think.

You mean 3 and 4 should be swapped? I don't think it matters.
 

What does host foo.example.com results in, on your computer?

Are you running your code on a Linux box or some other environment?

It is running on linux.

Thanks,

On Wednesday, 17 August 2016 13:49:21 UTC-7, Yaz Saito wrote:
Assume we have foo.example.com, whose value changes unpredictably.

1. Initially, the A record of foo.example.com points to, say 1.2.3.4.
2. Start a grpc client, with the server pointed at foo.example.com:10000. The client fails to
  connect since there's nothing running at 1.2.3.4:10000.
3. Start a grpc server at an IP address 1.2.3.5:10000
4. Update the DNS server so that foo.example.com points to 1.2.3.5.

In this situation, the grpc client never seems to discover the server at 1.2.3.5, even when I set the DNS TTL very short.

Am I diagnosing the problem correct? And if so, is there a way to work around?

--
yaz

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.



--
-Abhishek

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.

raut...@gmail.com

unread,
Aug 18, 2016, 8:06:22 PM8/18/16
to grpc.io, raut...@gmail.com
What language is your client written in?
Java, C++ or Go?

For GO, there are two ways to get list of IP addresses for a FQDN Name.

IIRC, one of the implementations for resolver is via a syscall and other one is via a Go's stdlib.

You might want to double check on behavior if that is GO.


On Thursday, 18 August 2016 16:20:34 UTC-7, Abhishek Parmar wrote:
(Replying for Yaz, he is on vacation)
On Thu, Aug 18, 2016 at 4:15 PM, <raut...@gmail.com> wrote:
Steps 4 and 5 should be interchanged I think.

You mean 3 and 4 should be swapped? I don't think it matters.
 

What does host foo.example.com results in, on your computer?

Are you running your code on a Linux box or some other environment?

It is running on linux.
 

Thanks,

On Wednesday, 17 August 2016 13:49:21 UTC-7, Yaz Saito wrote:
Assume we have foo.example.com, whose value changes unpredictably.

1. Initially, the A record of foo.example.com points to, say 1.2.3.4.
2. Start a grpc client, with the server pointed at foo.example.com:10000. The client fails to
  connect since there's nothing running at 1.2.3.4:10000.
3. Start a grpc server at an IP address 1.2.3.5:10000
4. Update the DNS server so that foo.example.com points to 1.2.3.5.

In this situation, the grpc client never seems to discover the server at 1.2.3.5, even when I set the DNS TTL very short.

Am I diagnosing the problem correct? And if so, is there a way to work around?

--
yaz

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.



--
-Abhishek

Craig Tiller

unread,
Aug 18, 2016, 8:16:48 PM8/18/16
to raut...@gmail.com, grpc.io
(I'm assuming Node or C++ based on previous conversations with Yaz/Abhishek)

Abhishek Parmar

unread,
Aug 18, 2016, 8:19:28 PM8/18/16
to Craig Tiller, raut...@gmail.com, grpc.io
This one is c++.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/b7a72b7a-1d4c-4e7c-b016-508e0ac76841%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
-Abhishek

Yaz Saito

unread,
Aug 26, 2016, 1:54:39 PM8/26/16
to Abhishek Parmar, Craig Tiller, raut...@gmail.com, grpc.io
Sorry for delay. I just checked the behavior with grpc-1.0.0, and the things are working as expected. 




For more options, visit https://groups.google.com/d/optout.



--
yaz
Reply all
Reply to author
Forward
0 new messages