kafka-rest Too many schema objects created

[Tyler Hawkes]

I'm getting errors on our health checks for kafka-rest. We send it 10 messages in a single post and read them back, but I'm getting an error while producing avro messages after a while.

It looks like this has been run into before here with other schema clients. I'm running confluent 2.0.0. Just wondering if upgrading to 2.0.1 will fix it or if this is still a bug and I need to periodically restart my kafka-rest servers until its fixed?

[2016-02-23T23:31:42.341] ERROR Unhandled exception resulting in internal server error response (io.confluent.rest.exceptions.GenericExceptionMapper:37)

java.lang.IllegalStateException: Too many schema objects created for monitor-avro-value!

        at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:87)

        at io.confluent.kafka.serializers.AbstractKafkaAvroSerDe.register(AbstractKafkaAvroSerDe.java:106)

        at io.confluent.kafkarest.AvroRestProducer.produce(AvroRestProducer.java:72)

        at io.confluent.kafkarest.ProducerPool.produce(ProducerPool.java:160)

        at io.confluent.kafkarest.resources.TopicsResource.produce(TopicsResource.java:127)

        at io.confluent.kafkarest.resources.TopicsResource.produceAvro(TopicsResource.java:119)

        at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)

        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)

        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)

        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:143)

        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)

        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)

        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)

        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)

        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:308)

        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)

        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)

        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)

        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)

        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)

        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)

        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:291)

        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1140)

        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:403)

        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:386)

        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:334)

        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

        at org.eclipse.jetty.server.Server.handle(Server.java:499)

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

        at java.lang.Thread.run(Thread.java:745)

[2016-02-23T23:31:42.343] INFO <clientIP> - - [23/Feb/2016:23:31:42 -0700] "POST /topics/monitor-avro HTTP/1.1" 500 52  3 (io.confluent.rest-utils.requests:77)

[Tyler Hawkes]

So I just confirmed this is a major issue. 5,000 minutes after the checks failed, every single kafka-rest agent started returning failures again (limit of 1000 schemas per topic w/ check every 5 minutes). 

Is there no one out there producing avro with kafka-rest? This appears to be a major issue with kafka-rest to me.

[Alex]

Having the same issue here and can't figure out exactly how to fix.  Anybody knows how to fix this?

This makes very little sense to me; looks like a poor architectural design decision.

[George @paytm.com]

In our production, we use node.js to produce to kafka-rest. Current message rate is around 500/sec on average, but keep climbing as we integreate more and more subsystems.

I ran into the similar problem. My solution is

1. We implemented a buffering mechanism so that each request sends many messages

2. In some cases, we force the request to include schema id instead of concrete avro schema json.

So far this solution, especially 1), serves us pretty well.