Error while forwarding register schema request to the master

[cgue...@myfitnesspal.com]

Hello,

We are using the Confluent schema registry as a master-slave set up, but sometimes when we try to register a new schema and the request is received by a slave instance, the request fails - the log reports an error with the schema I am trying to register.

However, the exact same request, with no changes to the schema at all, if it is sent directly to the master, succeeds immediately. It doesn't seem like there are really issues with the schema itself; it just fails when forwarded. Moreover, if we restart the slave instances, all write requests succeed (even using the same schemas that have had trouble before) even when forwarded from the slaves to the master. We cannot really tell when write requests start failing.

We have already discounted networking issues among the hosts. Are there any known issues with write requests forwarded from slaves to the master instance?

Confluent version: 1.0

Confluent tools: 1.1

Kafka version: 0.8.2

Zookeeper: 3.4.5--1

I appreciate any help.

Claudia Guerrero

Error log:

io.confluent.kafka.schemaregistry.rest.exceptions.RestRequestForwardingException: Error while forwarding register schema request to the master

io.confluent.kafka.schemaregistry.rest.exceptions.RestRequestForwardingException: Error while forwarding register schema request to the master

at io.confluent.kafka.schemaregistry.rest.exceptions.Errors.requestForwardingFailedException(Errors.java:88)

at io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource.register(SubjectVersionsResource.java:163)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)

at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:136)

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)

at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:406)

at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:350)

at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:106)

at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:259)

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)

at org.glassfish.jersey.internal.Errors.process(Errors.java:315)

at org.glassfish.jersey.internal.Errors.process(Errors.java:297)

at org.glassfish.jersey.internal.Errors.process(Errors.java:267)

at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:319)

at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:236)

at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)

at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:219)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

at org.eclipse.jetty.server.Server.handle(Server.java:370)

at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)

at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)

at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)

at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)

at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)

at java.lang.Thread.run(Thread.java:745)

Caused by: io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryRequestForwardingException: Unexpected error while forwarding the registering schema request {schema=<SCHEMA_REDACTED>} to http://schema-1-confluent-mbus-prod.mfpaws.com:3060

at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.forwardRegisterRequestToMaster(KafkaSchemaRegistry.java:486)

at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.registerOrForward(KafkaSchemaRegistry.java:342)

at io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource.register(SubjectVersionsResource.java:154)

... 47 more

Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

 at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@401698e1; line: 1, column: 2]

at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1419)

at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:508)

at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:437)

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2363)

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:794)

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:690)

at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3090)

at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3036)

at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2192)

at io.confluent.kafka.schemaregistry.client.rest.utils.RestUtils.httpRequest(RestUtils.java:144)

at io.confluent.kafka.schemaregistry.client.rest.utils.RestUtils.registerSchema(RestUtils.java:174)

at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.forwardRegisterRequestToMaster(KafkaSchemaRegistry.java:482)

... 49 more

at uacf.platform.gg.schemaregistry.client.rest.RestClientActor$$anonfun$handleResponse$1.applyOrElse(RestClientActor.scala:103) ~[schema-registry-service_2.11-1.0.0.jar:1.0.0]

at akka.actor.Actor$class.aroundReceive(Actor.scala:467) ~[akka-actor_2.11-2.3.11.jar:na]

at uacf.platform.gg.schemaregistry.client.rest.RestClientActor.akka$actor$DiagnosticActorLogging$$super$aroundReceive(RestClientActor.scala:22) ~[schema-registry-service_2.11-1.0.0.jar:1.0.0

]

at akka.actor.DiagnosticActorLogging$class.aroundReceive(Actor.scala:319) ~[akka-actor_2.11-2.3.11.jar:na]

at uacf.platform.gg.schemaregistry.client.rest.RestClientActor.aroundReceive(RestClientActor.scala:22) ~[schema-registry-service_2.11-1.0.0.jar:1.0.0]

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [akka-actor_2.11-2.3.11.jar:na]

at akka.actor.ActorCell.invoke(ActorCell.scala:487) [akka-actor_2.11-2.3.11.jar:na]

at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [akka-actor_2.11-2.3.11.jar:na]

at akka.dispatch.Mailbox.run(Mailbox.scala:220) [akka-actor_2.11-2.3.11.jar:na]

at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [akka-actor_2.11-2.3.11.jar:na]

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.11.7.jar:na]

at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.11.7.jar:na]

at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.11.7.jar:na]

at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.11.7.jar:na]

[Ewen Cheslack-Postava]

Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

That error looks like something that has been reported elsewhere. The way I've seen it before is if Jetty (or something else if you have a proxy in front of your instances) returns an HTML error page instead. This could be due to the master triggering some unexpected error that is not being correctly caught and transformed into a JSON error response. If you have any of the request logs from the master that show the error codes returned, that might help track down the problem.

It's worth pointing out that this stacktrace appears to be from an older version of CP (pre 3.0.0). Newer versions of the schema registry are catching JsonProcessingException (of which JsonParseException is a subclass) and should correctly convert the error messages. So a newer version may, at a minimum, help you understand the root of the problem.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f8c9b208-55a3-49d0-beea-1aedd7d65a1f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen