Can the Schema Registry handle a dynamic list of schemas?

[Baden]

Being new to Confluent Kafka Connect and the whole ecosystem, I have some questions about how this would fit/work with my proposed architecture in a multi-tenant data ingestion platform.

Users will have the ability to use various data endpoint types (files, SQL, etc.) and define endpoints through a dashboard.  Thus, the endpoint data metadata will differ from one endpoint to another, and will change through the course of a Schema Registry server being activated.  

When defining an endpoint, the user may not yet know the schema, so maybe retrieving sample data and implementing automation or a manual process to define the schema would be necessary.  Thoughts/advice on this?

From https://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you-really-need-one/:

This is why Confluent’s stream data platform includes a Schema Registry. We don’t want to force anyone to use them, and if you decide that Schemas are not applicable for your use-case, you can still use Kafka and the rest of the stream processing platform. But we believe that in many cases, a schema registry is a best practice and want to encourage its use.

It appears that Schemas are not required.  Given this use case, what would the best course of action be?

Is there a workflow/process for allowing the Schema Registry configuration to be dynamic? (hopefully I am asking the question correctly here).

[Baden]

To add to this, we would need to be able to dynamically add Topics, etc.

[ha...@confluent.io]

You can dynamically add topics to Kafka (if you enable auto topic creation) and you can dynamically add schemas to the Confluent Schema Registry (if they are backward compatible). You can also programmatically change the configuration of a connector running in Kafka Connect (via the connect workers REST endpoint).

[Baden]

That's good news.  Can you please direct me to some good information sources?

[ha...@confluent.io]

1) dynamically creating topics - see auto.create.topics.enable in http://kafka.apache.org/documentation.html#configuration

2) dynamically creating schemas in Confluent Schema Registry can be done two ways 

• directly from the publishing application using the KafkaAvroSerializer avro serdes library - see Java example code at http://docs.confluent.io/3.1.1/schema-registry/docs/serializer-formatter.html#serializer
  
• using the REST API of the Schema Registry - see quickstart examples using curl at http://docs.confluent.io/3.1.1/schema-registry/docs/intro.html#quickstart

3) programmatically changing the configuration can be done using the REST API of the Kafka Connect Worker process - see the example "Updating connector configuration" in the Using the REST Interface chapter of http://docs.confluent.io/3.1.1/connect/managing.html#managing-running-connectors