gRPC tools creating library from ETCD V3 that throws errors

[KaterchenSama]

Hello!

I am desperately trying to get a Python3.5.2 native interface to the ETCD V3 Api (since there is no new library currently in place or supported).

I nstalled the newest Protoc Compiler as well as the current GRPC tools straight from the source, gathered up all the proto-files which are imported in the etcd/etcdserver/etcdserverpb/rpc.proto from various sources and let it generate a grpc python file for each of them. I put them all into one folder, fixed up the imports to match and then fired up Ipython to try some things in the console.

At the very least the KV store seemed to work great. Put and Range were working fine and I was sure that everything else would work out as well.

But when I tried to set up a Watch, the Thread spawned in the background immediately dies. Here is my console output:

In [1]: import etcd_grpc.rpc_pb2 as rpc

In [2]: import grpc

In [3]: chan= grpc.insecure_channel('localhost:2379')

In [4]: kv = rpc.KVStub(chan)

In [5]: put = kv.Put(rpc.PutRequest(key=b'lala', value=b'hoho'))

In [6]: put
Out[6]:
header {
  cluster_id: 14841639068965178418
  member_id: 10276657743932975437
  revision: 15
  raft_term: 3
}

In [7]: wcr = rpc.WatchCreateRequest(key=b'bam')

In [8]: wr = rpc.WatchRequest(create_request=wcr)

In [9]: wi = rpc.WatchStub(grpc.insecure_channel('localhost:2379'))

In [10]: watch = wi.Watch(wr)

In [11]: Exception in thread Thread-849:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 184, in consume_request_iterator
    for request in request_iterator:
TypeError: 'WatchRequest' object is not iterable

The thread supposed to feed the watch-Rendevouz died leaving the console in an eternal loop (or maybe there is a long timeout) if I try to use watch.next()

I tried debugging it, but the automatedly generated code is stacked so deep and on top there is the threading complexity.... i have no idea what the final function does and why it would try to treat the WatchRequest as an iterator. I haven't modified the output of the grpc tools at all.

If someone has an Idea, could help me with this, or even has a working ETCD V3 Interface for python laying around. I would be grateful for either.

Thank you,
Martin Znamirowski

[Nathaniel Manista]

This smells like a cardinality problem - in particular I suspect that in your .proto file you've specified the Watch RPC method as request-streaming but in the code you've pasted above you are attempting to invoke the method as though it were request-unary. If you intend the Watch RPC method to only ever be given exactly one WatchRequest message, do you need to remove the first "stream " from "rpc Watch(stream WatchRequest) returns (stream WatchResponse) {}" in your .proto file? If you intend the Watch RPC method to be able to be given zero or more WatchRequest messages and wish in the context of this one call to give it just one WatchRequest message, do you need to change your invocation to "watch = wi.Watch(iter((wr,)))"?

The thread supposed to feed the watch-Rendevouz died leaving the console in an eternal loop (or maybe there is a long timeout) if I try to use watch.next()

I tried debugging it, but the automatedly generated code is stacked so deep and on top there is the threading complexity.... i have no idea what the final function does and why it would try to treat the WatchRequest as an iterator. I haven't modified the output of the grpc tools at all.

If someone has an Idea, could help me with this, or even has a working ETCD V3 Interface for python laying around. I would be grateful for either.

Thank you for including your Python code - it would also help to look at the associated .proto code.

-Nathaniel

[KaterchenSama]

This smells like a cardinality problem - in particular I suspect that in your .proto file you've specified the Watch RPC method as request-streaming but in the code you've pasted above you are attempting to invoke the method as though it were request-unary. If you intend the Watch RPC method to only ever be given exactly one WatchRequest message, do you need to remove the

first "stream " from "rpc Watch(stream WatchRequest) returns (stream WatchResponse) {}" in your .proto file? If you intend the Watch RPC method to be able to be given zero or more WatchRequest messages and wish in the context of this one call to give it just one WatchRequest message, do you need to change your invocation to "watch = wi.Watch(iter((wr,)))"?

The thread supposed to feed the watch-Rendevouz died leaving the console in an eternal loop (or maybe there is a long timeout) if I try to use watch.next()

I tried debugging it, but the automatedly generated code is stacked so deep and on top there is the threading complexity.... i have no idea what the final function does and why it would try to treat the WatchRequest as an iterator. I haven't modified the output of the grpc tools at all.

If someone has an Idea, could help me with this, or even has a working ETCD V3 Interface for python laying around. I would be grateful for either.

Thank you for including your Python code - it would also help to look at the associated .proto code.

-Nathaniel

Of course, my bad. The protoflies are part of the official etcd3 project, I will post the link below:
https://github.com/coreos/etcd/blob/master/etcdserver/etcdserverpb/rpc.proto

It seems that it actually expects a stream of watch requests (which doesn't appear to make sense to me at first glance, but I am not very experienced with either etcd nor grpc yet). I will try to go for either of your solutions. Thank you so much for the response. I really was stuck. I hope providing the proto-file gives more clarification.

Greets,
Martin

[KaterchenSama]

Hello once more.

I just realized that the whole thing could have been avoided if I had read the description of the watch function in the protofile correctly. The whole thing happened, because I was playing around with the generated grpc code for etcd3 and at every turn if I gave one of the functions a wrong request type, some sanity check would tell my that I made an invalid request. When I gave a watchRequest to the watch function (instead of an iterator/stream of watchRequests) the very same sanity check didn't trigger. Assuming that it would be supposed to, should I open up a bug ticket for the grpc python module ?

[Masood Malekghassemi]

I'm not sure I understand; could you please clarify what it was that you did to the code you provided above that ended up silencing the sanity check? While it's possible that we're indeed missing an error case, I think we ought to make sure that we understand what it is that we might be missing.

[Nathaniel Manista]

On Mon, Sep 26, 2016 at 1:10 AM, KaterchenSama <znamirows...@gmail.com> wrote:

When I gave a watchRequest to the watch function (instead of an iterator/stream of watchRequests) the very same sanity check didn't trigger. Assuming that it would be supposed to, should I open up a bug ticket for the grpc python module ?

I think you're right that we're not currently doing enough to be helpful to applications that are behaving incorrectly in this respect. I've filed issue 8231 to improve this.

-N