Re: One question regarding Schema Registry cache

[Ewen Cheslack-Postava]

The REST proxy does include a cache because it uses the Avro serializer/deserializer internally and those have their own cache. However, the API also provides support to avoid this in the first place because you probably don't want to include your full schema in every request you make (which can have quite a bit of overhead depending on how much data you batch into each request).

If possible, you should generally use the key_schema_id and value_schema_id parameters after the first request. These are returned by your first request that includes the full key_schema and value_schema. You can see in the docs for producing to a topic how the IDs are returned and are also permitted in the request: http://docs.confluent.io/2.0.0/kafka-rest/docs/api.html#post--topics-%28string-topic_name%29

-Ewen

On Mon, Jan 25, 2016 at 8:31 AM, Jimmy Zheng <zhimin...@nytimes.com> wrote:

Hi all,

We are quite happy to find out the Schema Registry(SR) feature provided by Confluent, and now we are trying to test the performance. But just want to figure out if it have a caching system on the client side.

My concern is, if SR and REST Proxy are installed on different machines(and I believe they should), my understanding is every RESTful call to the REST Proxy will call SR service once to get the schema info. So with large amount of REST call, the connection between Rest Proxy and SR could be a bottleneck. My guess is PROXY might have a local schema cache so when there is a REST Proxy call, so it does not need to call the SR service in another machine.

Base on my reading of the architect(http://docs.confluent.io/2.0.0/schema-registry/docs/design.html#kafka-backend), I didn’t see any caching system except the master/slave mode, but as I check in the source code, it seems it have a server-client mode. 

io.confluent.kafka.schemaregistry.client.rest

io.confluent.kafka.schemaregistry.rest

So I guess there will be a SR client installed(on the REST Proxy side?) to act as the SR cache? In that case both the server side and client side have a restful service for SR? I also tried to find info regarding this from the installation part but with no luck. 

Could you help with my questions? Appreciate if you guys could provide some additional info regarding the mechanism related to SR performance.

Thanks,

Jimmy

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/518b46f3-f888-41ca-bbbe-5f140c18d15e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Ewen Cheslack-Postava]

That code is just the serializer, but yes, you can see that it is using CachedSchemaRegistryClient, which internally has a cache.

-Ewen

On Tue, Jan 26, 2016 at 12:57 PM, Zheng, Jimmy <zhimin...@nytimes.com> wrote:

Hi Ewen,

Thanks much for the help! Now I think I understand how it works.

So following code is the cache for Avro serializer/deserializer, which sits on the REST Proxy server side, and it should have a mapping of schema and schema_id(generated by SchemaRepository), may I know if I get it right?

https://github.com/confluentinc/schema-registry/blob/f34e9161391533060355a71f3bb129d95fabe496/avro-serializer/src/main/java/io/confluent/kafka/serializers/KafkaAvroSerializer.java

Thanks again!

Regards,

Zhiming(Jimmy) Zheng

Data Engineer, DS&E

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/qh5PGEOIPkQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAE1jLMNj6sDvPHMkxyy5wB6c-ZibbR96W83XKoPzL-pjpOYORw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CA%2BDCjejHKV4M-eWd%2BWgbuD7tFvADHvYxvc8noLYiHL2gjZbbxQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen