gRFC A5: Encoding grpclb info in DNS

[Mark D. Roth]

I've put together the following gRFC for encoding grpclb data in DNS:

https://github.com/grpc/proposal/pull/10

As per the gRFC process, please keep discussion in this thread.  All comments welcome.

Thanks!

--

Mark D. Roth <ro...@google.com>
Software Engineer
Google, Inc.

[David Garcia Quintas]

The document should be updated to mention https://github.com/grpc/grpc/pull/11237, perhaps once it's been merged.

[谭锦彪]

Hi,Roth.

I got "dns_resolver.c:252]         authority based dns uri's not supported", when use dns like below:

  ChannelArguments args;

  args.SetLoadBalancingPolicyName("round_robin");

  GreeterClient greeter(grpc::CreateCustomChannel(

      "dns://localhost:8600/sag_test_grpc.service.consul-dev.",                                                                                                        

      grpc::InsecureChannelCredentials(), args));

So What's your plan to supports this?

在 2017年2月1日星期三 UTC+8上午2:30:04，Mark D. Roth写道：

[Mark D. Roth]

The feature you're asking about actually has nothing to do with supporting grpclb in DNS; it's actually about how to use an alternative DNS server for testing.

I originally added support for specifying an alternative DNS server by IP address (not hostname) to the c-ares DNS resolver implementation in https://github.com/grpc/grpc/pull/10706, although there was a bug that prevented it from working, which I fixed in https://github.com/grpc/grpc/pull/11954.  If your build includes those PRs, then you should be able to use the c-ares resolver with an alternative DNS server specified as an IP address.

Note that the c-ares resolver is not currently the default DNS resolver; to enable it, you need to run with the environment variable GRPC_DNS_RESOLVER=ares.  There are still a few features missing from the c-ares DNS resolver (such as CNAME resolution and address sorting), which is why it's not yet the default, but we're working to fix that.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/0c3f5bcc-d5a4-48b2-9813-c31d884c94fb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[谭锦彪]

Thank you so much for your response.

Follow you suggest, I got this:

D0811 14:07:29.205003940   20324 dns_resolver_ares.c:209]    dns_ares_next is called.

D0811 14:07:29.205322437   20324 grpc_ares_ev_driver_posix.c:127] grpc_ares_ev_driver_create

I0811 14:07:29.205340232   20324 grpc_ares_wrapper.c:249]    Using DNS server 127.0.0.1:8600

and a coredump too, here is the call stack:

#0  0x0000000000000000 in ?? ()

#1  0x0000000000461e3b in FileLinePrinter (fd=11, pc=0x3717207faf <ares_set_servers_ports+79>, out=0x2b41a3dfe3b0 "\b", out_size=1024, relocation=236611174400)

    at base/stacktrace_utils.cpp:42

#2  0x00002b41a2470485 in google::SymbolizeAndDemangle (pc=0x3717207faf <ares_set_servers_ports+79>, out=0x2b41a3dfe3b0 "\b", out_size=1024) at src/symbolize.cc:611

#3  0x00002b41a2470835 in google::(anonymous namespace)::DumpStackFrameInfo (prefix=0x2b41a24725f6 "PC: ", pc=0x3717207fb0 <ares_set_servers_ports+80>)

    at src/signalhandler.cc:220

#4  0x00002b41a2470dbf in google::(anonymous namespace)::FailureSignalHandler (signal_number=11, signal_info=0x2b41a3dfeb70, ucontext=<optimized out>)

    at src/signalhandler.cc:300

#5  <signal handler called>

#6  0x0000003717207fb0 in ares_set_servers_ports () from /usr/lib64/libcares.so.2

#7  0x00002b41a297d0cf in grpc_dns_lookup_ares () from /usr/lib64/libgrpc.so.4

#8  0x00002b41a297bd66 in ?? () from /usr/lib64/libgrpc.so.4

#9  0x00002b41a296ca4c in ?? () from /usr/lib64/libgrpc.so.4

#10 0x00002b41a296cd3d in ?? () from /usr/lib64/libgrpc.so.4

#11 0x00002b41a296d091 in ?? () from /usr/lib64/libgrpc.so.4

#12 0x00002b41a29123ec in grpc_combiner_continue_exec_ctx () from /usr/lib64/libgrpc.so.4

#13 0x00002b41a2920f48 in grpc_exec_ctx_flush () from /usr/lib64/libgrpc.so.4

#14 0x00002b41a29358cf in grpc_call_start_batch () from /usr/lib64/libgrpc.so.4

#15 0x00002b41a26b17ec in grpc::Channel::PerformOpsOnCall(grpc::CallOpSetInterface*, grpc::Call*) () from /usr/lib64/libgrpc++.so.1

#16 0x00000000004551eb in grpc::Call::PerformOps (this=0x2b41a405fda8, ops=0x2b41a405fdc8) at /usr/local/include/grpc++/impl/codegen/call.h:655

#17 0x00000000004f1414 in grpc::ClientAsyncResponseReader<helloworld::HelloReply>::ClientAsyncResponseReader<helloworld::HelloRequest> (this=0x2b41a405fd98, 

    call=..., context=0x2b41a40481e0, request=...) at /usr/local/include/grpc++/impl/codegen/async_unary_call.h:152

#18 0x00000000004f01f3 in grpc::ClientAsyncResponseReader<helloworld::HelloReply>::Create<helloworld::HelloRequest> (channel=0x2b41a4010d00, cq=0x2b41a3dffa88, 

    method=..., context=0x2b41a40481e0, request=...) at /usr/local/include/grpc++/impl/codegen/async_unary_call.h:102

#19 0x00000000004ec390 in helloworld::Greeter::Stub::AsyncSayHelloRaw (this=0x2b41a400f100, context=0x2b41a40481e0, request=..., cq=0x2b41a3dffa88)

    at build64_debug/ads/serving/grpc_lb_demo/helloworld.grpc.pb.cc:36

#20 0x0000000000455684 in helloworld::Greeter::Stub::AsyncSayHello (this=0x2b41a400f100, context=0x2b41a40481e0, request=..., cq=0x2b41a3dffa88)

    at build64_debug/ads/serving/grpc_lb_demo/helloworld.grpc.pb.h:51

#21 0x000000000045332f in (anonymous namespace)::GreeterClient::SayHello (this=0x2b41a3dffa80, user=...) at ads/serving/grpc_lb_demo/greeter_async_client2.cc:58

#22 0x000000000045373f in (anonymous namespace)::SendRequest (loop=0x2b41a38c3000) at ads/serving/grpc_lb_demo/greeter_async_client2.cc:163

#23 0x000000000045798e in std::_Bind_simple<void (*(ev_loop*))(ev_loop*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x2b41a38c1eb0)

    at /usr/local/gcc4.8.2/include/c++/4.8.2/functional:1732

#24 0x000000000045788f in std::_Bind_simple<void (*(ev_loop*))(ev_loop*)>::operator()() (this=0x2b41a38c1eb0) at /usr/local/gcc4.8.2/include/c++/4.8.2/functional:1720

#25 0x0000000000457828 in std::thread::_Impl<std::_Bind_simple<void (*(ev_loop*))(ev_loop*)> >::_M_run() (this=0x2b41a38c1e98)

    at /usr/local/gcc4.8.2/include/c++/4.8.2/thread:115

#26 0x0000000000515010 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../.././libstdc++-v3/src/c++11/thread.cc:84

#27 0x0000003254007aa1 in start_thread () from /lib64/libpthread.so.0

---Type <return> to continue, or q <return> to quit---

#28 0x0000003253ce893d in clone () from /lib64/libc.so.6

It works good when not use authority,  so have you tested this function?

My enviroment is grpc 1.4.1 and c-ares 1.13.0

在 2017年8月8日星期二 UTC+8下午10:39:42，Mark D. Roth写道：

The feature you're asking about actually has nothing to do with supporting grpclb in DNS; it's actually about how to use an alternative DNS server for testing.

I originally added support for specifying an alternative DNS server by IP address (not hostname) to the c-ares DNS resolver implementation in https://github.com/grpc/grpc/pull/10706, although there was a bug that prevented it from working, which I fixed in https://github.com/grpc/grpc/pull/11954.  If your build includes those PRs, then you should be able to use the c-ares resolver with an alternative DNS server specified as an IP address.

Note that the c-ares resolver is not currently the default DNS resolver; to enable it, you need to run with the environment variable GRPC_DNS_RESOLVER=ares.  There are still a few features missing from the c-ares DNS resolver (such as CNAME resolution and address sorting), which is why it's not yet the default, but we're working to fix that.

On Tue, Aug 8, 2017 at 2:02 AM, 谭锦彪 <miren...@gmail.com> wrote:

Hi,Roth.

I got "dns_resolver.c:252]         authority based dns uri's not supported", when use dns like below:

  ChannelArguments args;

  args.SetLoadBalancingPolicyName("round_robin");

  GreeterClient greeter(grpc::CreateCustomChannel(

      "dns://localhost:8600/sag_test_grpc.service.consul-dev.",                                                                                                        

      grpc::InsecureChannelCredentials(), args));

So What's your plan to supports this?

在 2017年2月1日星期三 UTC+8上午2:30:04，Mark D. Roth写道：

I've put together the following gRFC for encoding grpclb data in DNS:

https://github.com/grpc/proposal/pull/10

As per the gRFC process, please keep discussion in this thread.  All comments welcome.

Thanks!

--

Mark D. Roth <ro...@google.com>
Software Engineer
Google, Inc.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/0c3f5bcc-d5a4-48b2-9813-c31d884c94fb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[谭锦彪]

A suggestion: why not parse the port use c-ares? as the dns srv support port. Thus will helpful to multiprocess server!

在 2017年2月1日星期三 UTC+8上午2:30:04，Mark D. Roth写道：

I've put together the following gRFC for encoding grpclb data in DNS:

[Mark D. Roth]

I'm sorry, I don't follow your suggestion.  What exactly do you mean by "parse the dns srv support port", and what use-case do you mean by "multiprocess server"?

--

You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/1b0a00e9-d9fd-4c85-87ea-46d35e69a4c6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Mark D. Roth]

It works fine for me when building from master:

D0811 07:57:48.380480115   16947 simple_request.c:116]       client_peer_before_call=dns://127.0.0.1:1234/foo.com.:23164

D0811 07:57:48.380498101   16947 dns_resolver_ares.c:304]    dns_ares_next is called.

D0811 07:57:48.380896092   16947 grpc_ares_ev_driver_posix.c:116] grpc_ares_ev_driver_create

I0811 07:57:48.380908935   16947 grpc_ares_wrapper.c:375]    Using DNS server 127.0.0.1:1234

D0811 07:57:48.381079580   16947 grpc_ares_ev_driver_posix.c:255] new fd: 8

D0811 07:57:48.381092125   16947 grpc_ares_ev_driver_posix.c:82] Ref ev_driver 35859312

D0811 07:57:48.381097921   16947 grpc_ares_ev_driver_posix.c:277] notify read on: 8

D0811 07:57:48.381121499   16947 grpc_ares_ev_driver_posix.c:187] readable on 8

D0811 07:57:48.381155057   16947 grpc_ares_wrapper.c:202]    c-ares resolver gets a AF_INET result: 

  addr: 127.0.0.1

  port: 23164

D0811 07:57:48.381171851   16947 grpc_ares_wrapper.c:230]    on_query_srv_done_cb

D0811 07:57:48.381183455   16947 grpc_ares_wrapper.c:275]    on_txt_done_cb

I0811 07:57:48.381193330   16947 grpc_ares_wrapper.c:308]    found service config: [{"serviceConfig":{"loadBalancingPolicy":"round_robin"}}]

I0811 07:57:48.381209488   16947 dns_resolver_ares.c:248]    selected service config choice: {"loadBalancingPolicy":"round_robin"}

D0811 07:57:48.381220558   16947 dns_resolver_ares.c:338]    dns_ares_maybe_finish_next_locked

D0811 07:57:48.381254825   16947 dns_resolver_ares.c:304]    dns_ares_next is called.

D0811 07:57:48.381329568   16947 grpc_ares_ev_driver_posix.c:88] Unref ev_driver 35859312

D0811 07:57:48.381345965   16947 grpc_ares_ev_driver_posix.c:99] delete fd: 8

D0811 07:57:48.381357034   16947 grpc_ares_ev_driver_posix.c:308] ev driver stop working

D0811 07:57:48.381363065   16947 grpc_ares_ev_driver_posix.c:88] Unref ev_driver 35859312

D0811 07:57:48.381367935   16947 grpc_ares_ev_driver_posix.c:90] destroy ev_driver 35859312

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/28254d01-aed2-46ad-90cb-10f93d080de7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[谭锦彪]

Sorry for my pool english. When I say "multiprocess server", I mean multi process in a server with the same ip, but bind different port, for example:

ps -ef | grep greeter_server

root     15494 12990  0 20:17 pts/4    00:00:00 ./greeter_server 5051

root     16034 15945  0 20:19 pts/8    00:00:00 ./greeter_server 5052

I want to combine grpc with consul, cause grpc do not support registration and discovery. so when I start multi process in one server, the client can not distingusih it.

and there are port in the DNS SRV records, for example:

dig @localhost -p 8600 sag-test-grpc.service.consul-dev. SRV

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6 <<>> @localhost -p 8600 sag-test-grpc.service.consul-dev. SRV

; (2 servers found)

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23645

;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; WARNING: recursion requested but not available

;; QUESTION SECTION:

;sag-test-grpc.service.consul-dev. IN SRV

;; ANSWER SECTION:

sag-test-grpc.service.consul-dev. 0 IN SRV 1 1 5052 sag-adsbackend-RTB-bjdx-10.110.28.11-test.node.bjdx.consul-dev.

sag-test-grpc.service.consul-dev. 0 IN SRV 1 1 5051 sag-adsbackend-RTB-bjdx-10.110.28.11-test.node.bjdx.consul-dev.

;; ADDITIONAL SECTION:

sag-adsbackend-RTB-bjdx-10.110.28.11-test.node.bjdx.consul-dev. 0 IN A 10.110.28.11

sag-adsbackend-RTB-bjdx-10.110.28.11-test.node.bjdx.consul-dev. 0 IN A 10.110.28.11

;; Query time: 1 msec

;; SERVER: 127.0.0.1#8600(127.0.0.1)

;; WHEN: Sat Aug 12 20:19:49 2017

;; MSG SIZE  rcvd: 174

And I find grpc parse the DNS SRV records for some other informations in this proposal:

https://github.com/grpc/proposal/blob/master/A5-grpclb-in-dns.md

So, why not parse the port too?

在 2017年8月11日星期五 UTC+8下午10:48:11，Mark D. Roth写道：

I'm sorry, I don't follow your suggestion.  What exactly do you mean by "parse the dns srv support port", and what use-case do you mean by "multiprocess server"?

On Fri, Aug 11, 2017 at 2:29 AM, 谭锦彪 <miren...@gmail.com> wrote:

A suggestion: why not parse the port use c-ares? as the dns srv support port. Thus will helpful to multiprocess server!

在 2017年2月1日星期三 UTC+8上午2:30:04，Mark D. Roth写道：

I've put together the following gRFC for encoding grpclb data in DNS:

https://github.com/grpc/proposal/pull/10

As per the gRFC process, please keep discussion in this thread.  All comments welcome.

Thanks!

--

Mark D. Roth <ro...@google.com>
Software Engineer
Google, Inc.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/1b0a00e9-d9fd-4c85-87ea-46d35e69a4c6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[谭锦彪]

Thanks for your response, maybe my environment has some problem, I will study it .

在 2017年8月11日星期五 UTC+8下午11:00:10，Mark D. Roth写道：

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/28254d01-aed2-46ad-90cb-10f93d080de7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Mark D. Roth]

gRPC only uses SRV records for grpclb, as described in this gRFC.  It does not use them for backend addresses, and we have no plans to change that.  If you want to implement your own system for service discovery on top of DNS, you can write your own resolver plugin for gRPC.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/8ebc89ce-fcf1-4599-96b7-3c134f2107e9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.