Schema-registry-> get avro schema

1,913 views
Skip to first unread message

Sagar Rao

unread,
Aug 4, 2016, 12:12:31 PM8/4/16
to Confluent Platform
Hi,

1) Is there a way using Schema-registry to get the avro schema files? I know we have this REST api

GET subjects/topic-value/versions/version-number but this returns a json which would need parsing. I was looking for something whereby I can hit and download all the schemas as avsc files which is what the SpecificRecord uses. I found this:


which seems to  have a mojo to download the schema files as avsc file but I wasn't able to download the plugin  from maven. So, if it exists, what would be the correct maven co-ordinates? OR any other way can be suggested?

2) Another generic question is regarding kafka version. In terms of APIs how much of a difference is there if I use Kafka 0.9 version v/s the 0.10 ? 

Thanks!
Sagar.

Ewen Cheslack-Postava

unread,
Aug 4, 2016, 1:29:28 PM8/4/16
to Confluent Platform
Sagar,

That plugin was just merged recently, so it hasn't been released yet. But it would be a good way to accomplish this.

With respect to Kafka versions, http://www.confluent.io/blog/announcing-apache-kafka-0.10-and-confluent-platform-3.0 explains all the new stuff in 0.10.0.0. For clients APIs specifically, Kafka Streams is entirely new, and "traditional" clients haven't changed much API-wise, but there are substantial improvements (and some additional flags) for the new consumer. You'll be fine using 0.9, but if at all possible we'd highly recommend using 0.10.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4447fa61-eb88-41bd-934f-c7746dd97871%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen

Sagar Rao

unread,
Aug 5, 2016, 8:27:40 AM8/5/16
to Confluent Platform
Thanks Ewen.

I went ahead and built that project locally and was able to get the avsc file downloaded from Schema registry.

Just as a rough estimate, by when can this be available?

Another thing I wanted to ask is that is there a way to set namespaces from the kafka-connect side? Right now no namespaces are set and when those avsc files are built using maven-avro-plugin, it throws errors like:

 error: `package' expected but `public' found.
[INFO] public class eng_match extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
[INFO] ^
[ERROR] /Users/raosa/gitprojects/examples/kafka-streams/target/generated-sources/eng_match.java:3619: error: identifier expected but eof found.
[INFO] }
[INFO]  ^
[ERROR] two errors found

I manually added the namespace and it started to compile.

Thanks!
Sagar.

On Thursday, 4 August 2016 22:59:28 UTC+5:30, Ewen Cheslack-Postava wrote:
Sagar,

That plugin was just merged recently, so it hasn't been released yet. But it would be a good way to accomplish this.

With respect to Kafka versions, http://www.confluent.io/blog/announcing-apache-kafka-0.10-and-confluent-platform-3.0 explains all the new stuff in 0.10.0.0. For clients APIs specifically, Kafka Streams is entirely new, and "traditional" clients haven't changed much API-wise, but there are substantial improvements (and some additional flags) for the new consumer. You'll be fine using 0.9, but if at all possible we'd highly recommend using 0.10.

-Ewen
On Thu, Aug 4, 2016 at 9:12 AM, Sagar Rao <sagarme...@gmail.com> wrote:
Hi,

1) Is there a way using Schema-registry to get the avro schema files? I know we have this REST api

GET subjects/topic-value/versions/version-number but this returns a json which would need parsing. I was looking for something whereby I can hit and download all the schemas as avsc files which is what the SpecificRecord uses. I found this:


which seems to  have a mojo to download the schema files as avsc file but I wasn't able to download the plugin  from maven. So, if it exists, what would be the correct maven co-ordinates? OR any other way can be suggested?

2) Another generic question is regarding kafka version. In terms of APIs how much of a difference is there if I use Kafka 0.9 version v/s the 0.10 ? 

Thanks!
Sagar.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.



--
Thanks,
Ewen

Ewen Cheslack-Postava

unread,
Aug 5, 2016, 7:35:17 PM8/5/16
to Confluent Platform
On Fri, Aug 5, 2016 at 5:27 AM, Sagar Rao <sagarme...@gmail.com> wrote:
Thanks Ewen.

I went ahead and built that project locally and was able to get the avsc file downloaded from Schema registry.

Just as a rough estimate, by when can this be available?

It'll be in the 3.1 release, but I don't have a date to give you for that yet.
 

Another thing I wanted to ask is that is there a way to set namespaces from the kafka-connect side? Right now no namespaces are set and when those avsc files are built using maven-avro-plugin, it throws errors like:

 error: `package' expected but `public' found.
[INFO] public class eng_match extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
[INFO] ^
[ERROR] /Users/raosa/gitprojects/examples/kafka-streams/target/generated-sources/eng_match.java:3619: error: identifier expected but eof found.
[INFO] }
[INFO]  ^
[ERROR] two errors found

I manually added the namespace and it started to compile.

Are you asking how to use a Connect schema to generate data and you want the Avro schema to have a namespace? Connect schemas intentionally only have a name, but if you use a dotted name (e.g. io.confluent.schemas.MySchema) the Avro converter will split on the last '.' to determine the Avro namespace and name, i.e. you'll get namespace=io.confluent.schemas and name=MySchema.

-Ewen
 

Thanks!
Sagar.

On Thursday, 4 August 2016 22:59:28 UTC+5:30, Ewen Cheslack-Postava wrote:
Sagar,

That plugin was just merged recently, so it hasn't been released yet. But it would be a good way to accomplish this.

With respect to Kafka versions, http://www.confluent.io/blog/announcing-apache-kafka-0.10-and-confluent-platform-3.0 explains all the new stuff in 0.10.0.0. For clients APIs specifically, Kafka Streams is entirely new, and "traditional" clients haven't changed much API-wise, but there are substantial improvements (and some additional flags) for the new consumer. You'll be fine using 0.9, but if at all possible we'd highly recommend using 0.10.

-Ewen
On Thu, Aug 4, 2016 at 9:12 AM, Sagar Rao <sagarme...@gmail.com> wrote:
Hi,

1) Is there a way using Schema-registry to get the avro schema files? I know we have this REST api

GET subjects/topic-value/versions/version-number but this returns a json which would need parsing. I was looking for something whereby I can hit and download all the schemas as avsc files which is what the SpecificRecord uses. I found this:


which seems to  have a mojo to download the schema files as avsc file but I wasn't able to download the plugin  from maven. So, if it exists, what would be the correct maven co-ordinates? OR any other way can be suggested?

2) Another generic question is regarding kafka version. In terms of APIs how much of a difference is there if I use Kafka 0.9 version v/s the 0.10 ? 

Thanks!
Sagar.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen

Sagar Rao

unread,
Aug 8, 2016, 3:39:47 AM8/8/16
to Confluent Platform
Hi Ewen,

Thanks for the replies. 

Regarding this:

Are you asking how to use a Connect schema to generate data and you want the Avro schema to have a namespace? Connect schemas intentionally only have a name, but if you use a dotted name (e.g. io.confluent.schemas.MySchema) the Avro converter will split on the last '.' to determine the Avro namespace and name, i.e. you'll get namespace=io.confluent.schemas and name=MySchema.

Yes, what I want is that when I run the kafka-connect JDBC connector, the avro messages sent to the table specific topics should have a namespace set. I tried to set the name field in the jdbc-connect.properties file to include names, but still I didn't get any namespaces set. 

So, when you say set name in the Connect schemas, does it not refer to the kafka-connect properties file?

Sagar. 
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

Ewen Cheslack-Postava

unread,
Aug 9, 2016, 11:04:47 PM8/9/16
to Confluent Platform
Sagar,

Ah, I understand the issue now. I was referring to how the Java code for a connector creates schemas, not something you control as a user. The connector would need to expose the ability for you to add a prefix (or we might be able to use single messages transforms to do this, once they are available) because they generate schemas dynamically. The JDBC connector doesn't include this today, although it seems it'd be an easy feature to support. (Today it only passes in the table name, so it'd just be a matter of adding a config and prepending it to the table name.)

-Ewen

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/dbdd8953-2a0c-495f-8d56-91200516ed8f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen

Sagar Rao

unread,
Aug 10, 2016, 7:29:18 AM8/10/16
to Confluent Platform
Thanks Ewen,

Do we need to raise an issue on github for this?

Thanks!
Sagar.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

Roel R

unread,
Aug 11, 2016, 3:07:56 AM8/11/16
to Confluent Platform
To plug my own thing: https://github.com/datamountaineer/schema-registry gives you a CLI that allows you to do e.g.:

schema-registry-cli get <your-subject>

Which outputs the .avsc for <your-subject>. It allows you to add, list, inspect the registry as well.

Op donderdag 4 augustus 2016 18:12:31 UTC+2 schreef Sagar Rao:

Ewen Cheslack-Postava

unread,
Aug 11, 2016, 9:50:30 PM8/11/16
to Confluent Platform
Sagar,

Sure having a record that this was requested would be great. As I said, this could be implemented by the connector itself or via single message transforms when it's available. It might make sense to implement in connectors until the latter becomes available.

-Ewen

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/aa1f0c64-f4dc-4cc3-8e8c-6992c70c097b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen

aneesha...@reflektion.com

unread,
Aug 1, 2017, 11:51:16 AM8/1/17
to Confluent Platform
Hello Even, 

I was trying to download all the schemas in schema registry as avsc files. I added the following to the pom.xml file. 

<repositories>
    <repository>
        <id>confluent</id>
        <url>http://packages.confluent.io/maven/</url>
    </repository>
</repositories>

<plugin>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-schema-registry-maven-plugin</artifactId>
    <version>3.3.0</version>
    <executions>
        <execution>
            <phase>generate-sources</phase>
            <goals>
                <goal>download</goal>
            </goals>
            <configuration>
                <schemaRegistryUrls>
                    <param>http://localhost:8081</param>
                </schemaRegistryUrls>
                <outputDirectory>${project.basedir}/src/schema/</outputDirectory>
                <subjectPatterns>^test</subjectPatterns>
            </configuration>
        </execution>
    </executions>
</plugin>

<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-schema-registry-maven-plugin</artifactId>
    <version>3.3.0</version>
</dependency>

But its not working I am getting the following error : 
"ERROR] Plugin io.confluent:kafka-schema-registry-maven-plugin:3.3.2 or one of its dependencies could not be resolved: Could not find artifact io.confluent:kafka-schema-registry-maven-plugin:jar:3.3.2 in central (https://repo.maven.apache.org/maven2) "

From the Schema registry documentation it seems it should work with version 3.3.0. 

Thanks in Advance!

Aneesha


On Thursday, August 4, 2016 at 10:29:28 AM UTC-7, Ewen Cheslack-Postava wrote:
Sagar,

That plugin was just merged recently, so it hasn't been released yet. But it would be a good way to accomplish this.

With respect to Kafka versions, http://www.confluent.io/blog/announcing-apache-kafka-0.10-and-confluent-platform-3.0 explains all the new stuff in 0.10.0.0. For clients APIs specifically, Kafka Streams is entirely new, and "traditional" clients haven't changed much API-wise, but there are substantial improvements (and some additional flags) for the new consumer. You'll be fine using 0.9, but if at all possible we'd highly recommend using 0.10.

-Ewen
On Thu, Aug 4, 2016 at 9:12 AM, Sagar Rao <sagarme...@gmail.com> wrote:
Hi,

1) Is there a way using Schema-registry to get the avro schema files? I know we have this REST api

GET subjects/topic-value/versions/version-number but this returns a json which would need parsing. I was looking for something whereby I can hit and download all the schemas as avsc files which is what the SpecificRecord uses. I found this:


which seems to  have a mojo to download the schema files as avsc file but I wasn't able to download the plugin  from maven. So, if it exists, what would be the correct maven co-ordinates? OR any other way can be suggested?

2) Another generic question is regarding kafka version. In terms of APIs how much of a difference is there if I use Kafka 0.9 version v/s the 0.10 ? 

Thanks!
Sagar.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.



--
Thanks,
Ewen

Ewen Cheslack-Postava

unread,
Aug 2, 2017, 12:47:29 PM8/2/17
to Confluent Platform
The schema registry plugin is published in Confluent's Maven repository. You'll need to add

<pluginRepositories>
  <pluginRepository>
    <id>confluent</id>
  </pluginRepository>
</pluginRepositories>

to your pom file as well.

-Ewen

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/336c472d-fecb-4720-b987-9c303eee7416%40googlegroups.com.

Nitin Agarwal

unread,
Sep 13, 2017, 9:06:05 AM9/13/17
to Confluent Platform
I'm also unable to download the avsc from the schema registry server by the maven plugin below:

<plugin>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-schema-registry-maven
-plugin</artifactId>
    <version>3.3.0</version>
    <configuration>
        <schemaRegistryUrls>
            <param>http://192.168.99.100:8081</param>
        </schemaRegistryUrls>
        <outputDirectory>src/main/avro</outputDirectory>
        <subjectPatterns>
            <param>^TestSubject000-(Key|Value)$</param>
        </subjectPatterns>
    </configuration>
</plugin>

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.



-- 
Thanks,
Ewen









-- 

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
pramod narayana
unread,
Jan 23, 2019, 3:33:33 AM1/23/19
to Confluent Platform
Hi Ewen,

I am using Kafka Schema Registry Maven Plugin.
It is downloading the latest Schema. But I want to download a specific version of the schema. How can I specify the schema version number in my pom.xml

<plugin>
                <groupId>io.confluent</groupId>
                <artifactId>kafka-schema-registry-maven-plugin</artifactId>
                <version>5.1.0</version>
                <configuration>
                    <schemaRegistryUrls>
                        <param>http://192.168.0.100:8081</param>
                    </schemaRegistryUrls>
                    <outputDirectory>src/main/resources/avro</outputDirectory>
                    <subjectPatterns>
                        <param>^pageview-avro-topic-(key|value)$</param>
                    </subjectPatterns>
                    <subjects>
                    </subjects>
                </configuration>
            </plugin>


-Pramod Narayana

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.



-- 
Thanks,
Ewen









-- 

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



-- 
Thanks,
Ewen









-- 

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



-- 
Thanks,
Ewen









-- 

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



-- 
Thanks,
Ewen




Reply all
Reply to author
Forward
0 new messages