In Memory Schema Registry

2,094 views
Skip to first unread message

Tony Zhao

unread,
Jun 24, 2015, 7:04:11 PM6/24/15
to confluent...@googlegroups.com
I can have test zookeeper and kafka servers in memory for testing our data pipeline, is there a way to test with a schema-registry in memory so that I can test our setup with the confluent platform? I see that there is a MockSchemaRegistryClient, but not sure how I would use that in our code to achieve an in memory schema registry. Any help is appreciated!

Thanks,
Tony

Ewen Cheslack-Postava

unread,
Jun 24, 2015, 7:33:39 PM6/24/15
to confluent...@googlegroups.com
Are you instantiating ZK and Kafka yourself or are you using the KafkaServerTestHarness from Kafka (https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala)? Schema registry has something similar called ClusterTestHarness (https://github.com/confluentinc/schema-registry/blob/master/core/src/test/java/io/confluent/kafka/schemaregistry/ClusterTestHarness.java) that you might be able to use if you pull in the test artifacts for schema-registry.

Alternatively, you should just be able to instantiate it yourself just like that class does -- just instantiate and call the appropriate lifecycle methods of the io.confluent.kafka.schemaregistry.RestApp class (https://github.com/confluentinc/schema-registry/blob/master/core/src/test/java/io/confluent/kafka/schemaregistry/RestApp.java). The Main class is pretty straightforward and should offer some guidance for how to use the RestApp class: https://github.com/confluentinc/schema-registry/blob/master/core/src/main/java/io/confluent/kafka/schemaregistry/rest/Main.java

The MockSchemaRegistryClient is intended for in-process unit testing, but it was originally just for internal testing (and called LocalSchemaRegistryClient). Recent changes have renamed it and we might want to make it more useful for testing like you're talking about since it's much lighter weight than running the full service, but currently not all the serializer/deserializer classes support passing in the schema registry client. At the moment, you might need to run the actual schema registry as described above.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/b46ba099-7c83-4836-999b-267b02e39990%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen

Tony Zhao

unread,
Jun 25, 2015, 2:33:48 AM6/25/15
to confluent...@googlegroups.com
I was following the example code here https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIwriteunittestsusingKafka?

I tried downloading the test jar you mentioned but it doesn't look like it is published or perhaps I have the name incorrect?

[warn] ==== Confluent repository: tried
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] ::              FAILED DOWNLOADS            ::
[warn] :: ^ see resolution messages for details  ^ ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: io.confluent#kafka-schema-registry;1.0!kafka-schema-registry.jar
[warn] ::::::::::::::::::::::::::::::::::::::::::::::


On Wednesday, June 24, 2015 at 4:33:39 PM UTC-7, Ewen Cheslack-Postava wrote:
Are you instantiating ZK and Kafka yourself or are you using the KafkaServerTestHarness from Kafka (https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala)? Schema registry has something similar called ClusterTestHarness (https://github.com/confluentinc/schema-registry/blob/master/core/src/test/java/io/confluent/kafka/schemaregistry/ClusterTestHarness.java) that you might be able to use if you pull in the test artifacts for schema-registry.

Alternatively, you should just be able to instantiate it yourself just like that class does -- just instantiate and call the appropriate lifecycle methods of the io.confluent.kafka.schemaregistry.RestApp class (https://github.com/confluentinc/schema-registry/blob/master/core/src/test/java/io/confluent/kafka/schemaregistry/RestApp.java). The Main class is pretty straightforward and should offer some guidance for how to use the RestApp class: https://github.com/confluentinc/schema-registry/blob/master/core/src/main/java/io/confluent/kafka/schemaregistry/rest/Main.java

The MockSchemaRegistryClient is intended for in-process unit testing, but it was originally just for internal testing (and called LocalSchemaRegistryClient). Recent changes have renamed it and we might want to make it more useful for testing like you're talking about since it's much lighter weight than running the full service, but currently not all the serializer/deserializer classes support passing in the schema registry client. At the moment, you might need to run the actual schema registry as described above.

-Ewen
On Wed, Jun 24, 2015 at 4:04 PM, Tony Zhao <yut...@gmail.com> wrote:
I can have test zookeeper and kafka servers in memory for testing our data pipeline, is there a way to test with a schema-registry in memory so that I can test our setup with the confluent platform? I see that there is a MockSchemaRegistryClient, but not sure how I would use that in our code to achieve an in memory schema registry. Any help is appreciated!

Thanks,
Tony

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.



--
Thanks,
Ewen

Ewen Cheslack-Postava

unread,
Jun 25, 2015, 12:38:55 PM6/25/15
to confluent...@googlegroups.com
Sorry, we're haven't actually published the test jars, I was just suggesting you could potentially reuse those libraries. For now you'd have to build those yourself or grab a copy of the class. We really should be publishing the test jars though, so I filed https://github.com/confluentinc/packaging/issues/13 to track it for our next release (and make sure it gets done for all our projects.

However, do keep in mind we don't currently treat these classes as public APIs so they are subject to change!

-Ewen

On Wed, Jun 24, 2015 at 11:33 PM, Tony Zhao <yut...@gmail.com> wrote:
I tried downloading the test jar you mentioned but it doesn't look like it is published or perhaps I have the name incorrect?

[warn] ==== Confluent repository: tried
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] ::              FAILED DOWNLOADS            ::
[warn] :: ^ see resolution messages for details  ^ ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: io.confluent#kafka-schema-registry;1.0!kafka-schema-registry.jar
[warn] ::::::::::::::::::::::::::::::::::::::::::::::


On Wednesday, June 24, 2015 at 4:33:39 PM UTC-7, Ewen Cheslack-Postava wrote:
Are you instantiating ZK and Kafka yourself or are you using the KafkaServerTestHarness from Kafka (https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala)? Schema registry has something similar called ClusterTestHarness (https://github.com/confluentinc/schema-registry/blob/master/core/src/test/java/io/confluent/kafka/schemaregistry/ClusterTestHarness.java) that you might be able to use if you pull in the test artifacts for schema-registry.

Alternatively, you should just be able to instantiate it yourself just like that class does -- just instantiate and call the appropriate lifecycle methods of the io.confluent.kafka.schemaregistry.RestApp class (https://github.com/confluentinc/schema-registry/blob/master/core/src/test/java/io/confluent/kafka/schemaregistry/RestApp.java). The Main class is pretty straightforward and should offer some guidance for how to use the RestApp class: https://github.com/confluentinc/schema-registry/blob/master/core/src/main/java/io/confluent/kafka/schemaregistry/rest/Main.java

The MockSchemaRegistryClient is intended for in-process unit testing, but it was originally just for internal testing (and called LocalSchemaRegistryClient). Recent changes have renamed it and we might want to make it more useful for testing like you're talking about since it's much lighter weight than running the full service, but currently not all the serializer/deserializer classes support passing in the schema registry client. At the moment, you might need to run the actual schema registry as described above.

-Ewen
On Wed, Jun 24, 2015 at 4:04 PM, Tony Zhao <yut...@gmail.com> wrote:
I can have test zookeeper and kafka servers in memory for testing our data pipeline, is there a way to test with a schema-registry in memory so that I can test our setup with the confluent platform? I see that there is a MockSchemaRegistryClient, but not sure how I would use that in our code to achieve an in memory schema registry. Any help is appreciated!

Thanks,
Tony

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/578cff11-d1bd-4229-a745-f144ea91c9ab%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen

Robert Duffy

unread,
Jan 17, 2017, 4:59:11 PM1/17/17
to Confluent Platform
Hello.  was this ever published?  There are ways to run a Kafka / Zookeeper inside a test but having to run Schema Registry external + Kafka seems to defeat the purpose.  Having a RestApp for SchemaRegistry be spun up in a test would be useful
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen
Message has been deleted

Michael Noll

unread,
Apr 3, 2017, 5:33:26 AM4/3/17
to confluent...@googlegroups.com
Robert,

take a look at:

This includes an "embedded" schema registry instance.

The following pom.xml snippet explains how to get the artifact that includes RestApp:

Caveat:  As you can see, this is part of a test-scoped artifact, which means there's no contract that guarantees things will work across newer versions of Confluent Schema Registry.

-Michael



To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

chinch...@gmail.com

unread,
Nov 9, 2018, 11:55:13 AM11/9/18
to Confluent Platform
Hi folks,
 The below links  return a 404 . I understand this may  have  been moved around . Have the test artifacts been published ? .What I am looking to do  is  unit  test a pipe that has producer > avro record > topic > consumer.I guess the easiest way to do this would be to start a embedded kafka and schema registry.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Sacha Barber

unread,
Nov 9, 2018, 12:13:54 PM11/9/18
to Confluent Platform
Shame you cant use Scala, you could use this awesome library : https://github.com/manub/scalatest-embedded-kafka which does Kafka/Zookeeper/SchemaRegistry

chinchu chinchu

unread,
Nov 9, 2018, 4:19:20 PM11/9/18
to confluent...@googlegroups.com
I  tried  using a  stub of spring clouds schema registry but running into the  below error .Any  thoughts?.

org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {"type":"record","name":"Log","namespace":"com.test.avro","fields":
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Cannot construct instance of `io.confluent.kafka.schemaregistry.client.rest.entities.ErrorMessage` (although at least one Creator exists): no int/Int-argument constructor/factory method to deserialize from Number value (404)
 at [Source: (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column: 1]; error code: 50005
at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:191)
at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:218)
at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:307)
at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:299)
at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:294)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:61)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:100)
at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:79)
at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:53)
at org.apache.kafka.common.serialization.ExtendedSerializer$Wrapper.serialize(ExtendedSerializer.java:65)
at org.apache.kafka.common.serialization.ExtendedSerializer$Wrapper.serialize(ExtendedSerializer.java:55)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:791)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:768)
at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer.send(DefaultKafkaProducerFactory.java:285)
at org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:348)
at org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:181)
at com.test.KafkaUnitTest.dummy(KafkaUnitTest.java:91)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:73)
at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:83)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/fdf11074-08eb-4aee-a1fa-49b1869733fc%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages