Table.exists hangs followed by DEADLINE_EXCEEDED

[talonx]

I've a Bigtable client that connects to a local emulator. Lately it has started hanging on table.exists - the only thing that was updated recently was the emulator itself via apt. A thread dump reveals

"main" #1 prio=5 os_prio=0 tid=0x00007ff678010000 nid=0x67c2 waiting on condition [0x00007ff67f143000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000007441b7ad0> (a com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation$GrpcFuture)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:469)
	at com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getBlockingResult(AbstractRetryingOperation.java:293)
	at com.google.cloud.bigtable.grpc.BigtableDataGrpcClient.readFlatRowsList(BigtableDataGrpcClient.java:327)
	at com.google.cloud.bigtable.hbase.BigtableTable.getResults(BigtableTable.java:257)
	at com.google.cloud.bigtable.hbase.BigtableTable.exists(BigtableTable.java:166)

----snipped----

Subsequent thread dumps taken at 30 second intervals reveals the the thread is stuck at the same place. After about 12 minutes, I get the following error -

java.io.IOException: Failed to perform operation. Operation='checkAndMutate', projectId='emulator', tableName='mquery', rowKey='149de7d2-272b-4524-ace3-8d977da6faf6#27c893ee-b50c-4b47-b08f-f3824291a62a'
	at com.google.cloud.bigtable.hbase.BigtableTable.logAndCreateIOException(BigtableTable.java:522)
	at com.google.cloud.bigtable.hbase.BigtableTable.checkAndMutate(BigtableTable.java:470)
----snipped----
Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 359999872003ns
	at io.grpc.Status.asRuntimeException(Status.java:517)
	at com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getBlockingResult(AbstractRetryingOperation.java:300)
	at com.google.cloud.bigtable.grpc.BigtableDataGrpcClient.checkAndMutateRow(BigtableDataGrpcClient.java:267)
	at com.google.cloud.bigtable.hbase.BigtableTable.checkAndMutate(BigtableTable.java:467)
	... 32 common frames omitted

This happens for each call. The 12-minute interval occurs in each case. It's brought my development to a grinding halt since I cannot use HBase/Bigtable at all (although I have not tried it with Bigtable itself, just the emulator). Since this has started in the last 1 week or so, I suspect this might be due to some recent update in the emulator.

Library versions:
google-cloud-core 1.8.0,
google-cloud-core-grpc 1.8.0,
google-cloud-core 1.8.0,
bigtable-hbase-1.x 1.0.0-pre3

gcloud version output:
Google Cloud SDK 177.0.0
alpha 2017.10.20
beta 2017.10.20
bigtable
bq 2.0.27
cbt
core 2017.10.20
gsutil 4.28
pubsub-emulator 2017.10.20

OS: Ubuntu 17.10

java -version output:
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

[talonx]

So I tried rolling back the gcloud version to google-cloud-sdk-175, and it started working. But this is a temporary solution.

[Gary Elliott]

-customer

There have been no changes to the emulator between the two builds he tried - 175 and 177. The last change was 3 months ago in version 158. It's possible that other cloud sdk changes are impacting us.

On Mon, Nov 6, 2017 at 10:28 AM talonx <tal...@gmail.com> wrote:

So I tried rolling back the gcloud version to google-cloud-sdk-175, and it started working. But this is a temporary solution.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Bigtable Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-bigtabl...@googlegroups.com.
To post to this group, send email to google-cloud-b...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-bigtable-discuss/3c2e186a-83ae-4286-baf0-1729b4e1c6b0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.