io.grpc.StatusRuntimeException: CANCELLED: call already cancelled

2,000 views
Skip to first unread message

Ashwin Madavan

unread,
May 1, 2018, 3:17:36 AM5/1/18
to ScalaPB
I have a cluster of nodes running my service BeakerGrpc. My nodes are sending requests to each other, but sometimes this error gets thrown.

io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
       at io.grpc.Status.asRuntimeException(Status.java:517)
       at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:335)
       at com.trueaccord.scalapb.grpc.Grpc$.completeObserver(Grpc.scala:21)
       at beaker.server.protobuf.BeakerGrpc$$anon$8.$anonfun$invoke$7(BeakerGrpc.scala:183)
       at beaker.server.protobuf.BeakerGrpc$$anon$8.$anonfun$invoke$7$adapted(BeakerGrpc.scala:183)
       at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
       at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:140)
       at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
       at java.util.concurrent.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1021)
       at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1046)
       at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
       at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
       at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)


Here's where the client makes the request.

private[beaker] def accept(proposal: Proposal): Future[Unit] = {
  val fork = Context.current().fork()
  val prev = fork.attach()
  val stub = BeakerGrpc.stub(this.channel).withDeadlineAfter(1, TimeUnit.SECONDS)

  try {
      stub.accept(proposal).filter(_.successful)
  } finally {
    fork.detach(prev)
  }
}

But the exception is thrown from many of the services in my program, even ones without forked contexts.

Nadav Samet

unread,
May 1, 2018, 10:26:11 AM5/1/18
to Ashwin Madavan, ScalaPB
I am not sure if this is a ScalaPB problem or related to grpc. Try to see if you can reproduce this problem without having ScalaPB involved to see on which side it falls.

--
You received this message because you are subscribed to the Google Groups "ScalaPB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalapb+unsubscribe@googlegroups.com.
To post to this group, send email to sca...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scalapb/beae5012-17e9-41dc-bb86-9559a1910a22%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
-Nadav

Ashwin Madavan

unread,
May 1, 2018, 1:46:23 PM5/1/18
to ScalaPB
The error seems to be coming here.

def completeObserver[T](observer: StreamObserver[T])(t: Try[T]): Unit = t match {
  case scala.util.Success(value) =>
    observer.onNext(value)
    observer.onCompleted()
  case scala.util.Failure(s: StatusException) =>
    observer.onError(s)
  case scala.util.Failure(s: StatusRuntimeException) =>
    observer.onError(s)
  case scala.util.Failure(e) =>
    observer.onError(
      Status.INTERNAL.withDescription(e.getMessage).withCause(e).asException()
    )
}

The observer.onNext call is failing. What should happen is that observer.onError should be called if observer.onNext failed. I think it should look something like this.

def completeObserver[T](observer: StreamObserver[T])(t: Try[T]): Unit = t.flatMap(observer.onNext) match {
  case scala.util.Success(value) =>
    observer.onCompleted()
  case scala.util.Failure(s: StatusException) =>
    observer.onError(s)
  case scala.util.Failure(s: StatusRuntimeException) =>
    observer.onError(s)
  case scala.util.Failure(e) =>
    observer.onError(
      Status.INTERNAL.withDescription(e.getMessage).withCause(e).asException()
    )
}
To unsubscribe from this group and stop receiving emails from it, send an email to scalapb+u...@googlegroups.com.

To post to this group, send email to sca...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scalapb/beae5012-17e9-41dc-bb86-9559a1910a22%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
-Nadav

Nadav Samet

unread,
May 1, 2018, 1:53:32 PM5/1/18
to Ashwin Madavan, ScalaPB
Thanks for looking into this. I think that you are right. We are not handling the case where onNext() raises an exception. Based on the last sentence of https://grpc.io/grpc-java/javadoc/io/grpc/stub/StreamObserver.html#onNext-V- :

If an exception is thrown by an implementation the caller is expected to terminate the stream by calling onError(Throwable) with the caught exception prior to propagating it.

Would you mind sending a PR? Can you add a test that ensures that onError gets called?


To unsubscribe from this group and stop receiving emails from it, send an email to scalapb+unsubscribe@googlegroups.com.

To post to this group, send email to sca...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scalapb/1dd9c95e-25d8-4920-8419-cf0e021375f8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
-Nadav

Ashwin Madavan

unread,
May 1, 2018, 2:00:18 PM5/1/18
to ScalaPB
Sounds good.



--
-Nadav

Ashwin Madavan

unread,
May 1, 2018, 3:07:03 PM5/1/18
to ScalaPB

Ashwin Madavan

unread,
May 1, 2018, 3:45:10 PM5/1/18
to ScalaPB
What is the release schedule? I have a presentation next week, and I would really like for the fix for this bug to be merged by then.

Nadav Samet

unread,
May 1, 2018, 3:54:50 PM5/1/18
to Ashwin Madavan, ScalaPB
I'll be able to make a release later today. In the meantime, is there a way for you to verify that this PR fixes the original issue you reported (for example, you could use `sbt publishLocal` and rely on the SNAPSHOT version)

To unsubscribe from this group and stop receiving emails from it, send an email to scalapb+unsubscribe@googlegroups.com.

To post to this group, send email to sca...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scalapb/d40749e1-50b0-478e-b0f7-8e6355157f88%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
-Nadav

Ashwin Madavan

unread,
May 1, 2018, 5:06:57 PM5/1/18
to ScalaPB
I'm not using SBT. I'll be able to verify once a release is cut.



--
-Nadav

Nadav Samet

unread,
May 1, 2018, 7:25:50 PM5/1/18
to Ashwin Madavan, ScalaPB
I've just cut 0.7.4.

To unsubscribe from this group and stop receiving emails from it, send an email to scalapb+unsubscribe@googlegroups.com.

To post to this group, send email to sca...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scalapb/933c60c1-0f8c-41b3-8ee1-e67b5be4ef52%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
-Nadav

Ashwin Madavan

unread,
May 1, 2018, 7:58:54 PM5/1/18
to ScalaPB
Looks like the fix works! Thanks for the quick turnaround. Love what you did with this project.



--
-Nadav
Reply all
Reply to author
Forward
0 new messages