does it help with streaming cases only?

[Boris Key]

Hi all,

I just discovered the project and have a couple questions. It seems to me that schema registry does not address a broader problem which is central place to maintain schemas for entire ecosystem. If one wants to create HIVE tables, or HBase tables or create ElasticSearch schema - it still needs to be done outside of schema registry. Someone will need to craft create statement DDL for HIVE and if you do INSERT statements, write them as well. Is there any plans to handle data at rest and handle those other systems, which require creation of tables or schemas?

This was my problem with Confluent schema registry as well and I ended up creating a few tables that would describe my schemas (really tables and columns with names and data types).

Second question, who do you handle schema conversion in the pipeline when you need to convert data types or rename fields - basically do typical ETL type of processing. How do you relate one schema to another? This was my other issue with Confluent schema registry as their schema describes data at point A or point B but cannot describe how schema B was derived from schema A (not to confuse with versioning).

Hope my questions make sense

thanks

Boris

[Harsha Ch]

Hi Boris,

            Thanks for your interest in the project.  

Question 1: Yes. currently, schema registry only addresses the use cases related to streaming. But we are making progress on merging Hive Metastore and registry APIs so that user can have a single metadata repository to maintain schemas for data at rest and in motion. Here is the uber Jira https://issues.apache.org/jira/browse/HIVE-17989 where you can follow the progress. We are going to retain backward compatibility of all the registry APIs when the metastore merge happens.

Question 2: If I understood correctly what you are asking is a way to track the lineage of schemas that are getting changed in ETL process be it streaming or not. This is one of the use cases we are going to address in near future.

Thanks,

Harsha

--
You received this message because you are subscribed to the Google Groups "Registry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to registry+unsubscribe@googlegroups.com.
To post to this group, send email to regi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/registry/58db58de-4742-4583-8385-601122eae8eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Boris]

Hi Harsha,

thanks for your comments! I am glad to hear you guys are thinking about that. You got my second question exactly right - thanks!

This is a much needed project, thanks for making this a reality! I am happy to learn that this is not just another version of Confluent Schema Registry but something bigger and something more universal.

Hopefully you guys are also thinking about extending this project beyond Hortonworks distro. 

Boris