GRPC error 14 in C++ API CheckAndMutateRow : Socket closed

[Bastien Duclaux]

Hello

Some long-running jobs using the C++ API are getting the following GRPC errors after a few hours :

terminate called after throwing an instance of 'google::cloud::bigtable::v0::GRpcError’

what():  Table::CheckAndMutateRow(table_name=xxx) Socket closed: Table::CheckAndMutateRow(table_name=xxx) Socket closed [14=UNAVAILABLE] - 

Looking at the C++ API code, it seems that the library only catches the most frequent errors, but not this one (seems that it is a PermanentMutationFailure ).

What is the right way to handle this ? How can I restart the connection and retry the mutation ?

I had at look at the GetState method inside the grpc::Channel class, but it is not clear is such code below would work :

try {

        do_mutation();

}

catch (const google::cloud::bigtable::v0::GRpcError &e)

{       

        cout<<"Trying to restart connection"<<endl;

        auto chan=client->Channel();

        chan->GetState(true);

        do_mutation();

}

Thanks for your help.

Bastien

[Solomon Duskis]

Check And Mutate us not idempotent, so we cannot retry it.  Even a "socket closed" exception might start after the service initiates the transaction, but before it's finished.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Bigtable Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-bigtabl...@googlegroups.com.
To post to this group, send email to google-cloud-b...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-bigtable-discuss/5EF39EDF-46EC-4D41-880E-BBD90CA440D7%40nuwa.be.
For more options, visit https://groups.google.com/d/optout.

[Bastien Duclaux]

Thanks !

Sure it is not idempotent. My question was more about the type of logic I should implement when receiving this error.

More specifically, is the underlying grpc::Channel « dead » , hence also the DataClient ? 

Or is there a chance to re-enable the underlying channel without destroying and recreating the full set of C++ objects ( DataClient and Table ) ?

[Carlos ORyan]

Hi,

There is an open bug to be able to override the idempotency policy for these operations:

https://github.com/GoogleCloudPlatform/google-cloud-cpp/issues/1390

Once we fix the bug you won't need to do an explicit retry loop.  I will bump it up in the priority queue, since it is affecting you.  While we fix the bug, you do not need to explicitly close the channel, simply retrying the operation should work, because the client library round-robins over the connections and gRPC reconnects a channel when you start a new request (but does not retry for the same reasons we do not).

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-bigtable-discuss+unsub...@googlegroups.com.