What is the correct key schema?

[Vladimir Lebedev]

Hey,

I got stuck with finding proper schema for the message key. When I produce messages to Kafka via rest-proxy without keys, everything is fine, but when I add the key into records objects, I'm getting either status 422 if I omit key_schema, or status 500 when I try to define some.

Here is my typical POST body contents:

{ 

  "value_schema_id":321,

  "key_schema": "string",

  "records": [

                    {

                     "partition": 0,

                     "key": "52da54f0-28c7-4362-7f54-c1c8b4e2c61e",

                     "value": { "id":"52da54f0-28c7-4362-7f54-c1c8b4e2c61e", "url":"http://someurl/", "ip":"1.2.3.4", "ts":1427455310804}

                    }

                 ]

}

I also tried "{\"type\": \"string\"}" and {\"name\": \"string\", \"type\": \"string\"} as values for key_schema, but result is always the same: status 500.

Could you please advise me what schema should be used for simple string message keys?

Vladimir

[Ewen Cheslack-Postava]

Vladimir,

Schemas are currently passed serialized as strings. If you use "key_schema": "string", it fails to parse the schema because it isn't valid JSON. Using "key_schema": "\"string\"" is supposed to work since that is a valid JSON-serialized Avro schema. (Having to use the serialized version isn't ideal, we have this issue filed to just be able to embed the JSON directly: https://github.com/confluentinc/kafka-rest/issues/45).

The cause of the 500 looks like it's actually a bug in the serializer. The serializer is checking the type of the object passed in and has a case for strings, but the REST proxy is using Avro to convert the JSON-encoded Avro it receives to an in-memory representation to pass the Kafka producer, and for strings it uses its own Utf8 class. The serializer isn't currently handling that properly because it checks for String rather than CharSequence. This existing bug was tracking some other types we may not be supporting fully: https://github.com/confluentinc/schema-registry/issues/110

This should have been caught in the REST proxy tests, so I've also filed this issue to add tests for more schema types in the REST proxy: https://github.com/confluentinc/kafka-rest/issues/72

By the way, if you're encountering errors like this, it can help to run the REST proxy with the debug=true setting. That will cause it to include stack traces in the error responses which can often make the issue pretty clear. This is especially useful for 500 errors since any 500 that it's generating without a more specific error code + message indicates a bug that we probably want to file an issue for.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4f1df951-4338-4083-9120-8eb101dd3575%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen