Kafka CRUD operations using REST API's

[Surbhi Agarwal]

Hi,

I am new to Kafka. There are few rest api's available on website of kafka.

https://kafka.apache.org/documentation/#connect_rest

I couldn't find any api for topics. Like perform CRUD operations on topic but using api only.

An example would be like to run on postman..

GET: http://localhost:8083/topics/{topic name}

GET: http://localhost:8083/topics/ => gets all topic lists available on kafka server

Is it possible?

Thanks,

Surbhi

[Robin Moffatt]

Hi Surbhi, 

The REST API you refer to there is the Kafka Connect REST API (tutorial), and is to do with creating and managing connectors. 

To manage topics, and read and write messages from them, you can use the Kafka REST Proxy: https://docs.confluent.io/platform/current/kafka-rest/index.html

-- 

Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/cba7cfe9-692f-4cb3-a6bc-845a4cccedb9n%40googlegroups.com.

[sunil chaudhari]

Hi Surbhi,

Just wondering why do you need rest api to manage topics?

Not sure about delete but other stuffs you can perform with confluent control centre.

--

[Surbhi Agarwal]

Hi Sunil,

I an new to kafka, so I am getting confused as I am proceeding in my learning. Another question arises in my mind, if we want to query on producer and consumer as well. Can this be done or not. Is this even valid question? Why can't we manage topics through REST API's?

Thanks Robins for replying.

Thanks,

Surbhi

[BadwolF ForeveR]

Hi Surbhi,

A couple of questions before I start diving in with details

- What do you mean by "Query on Producer and Consumer"?

- When you say "manage topics through REST APIs" .. do you mean their creation/deletion/config changes (i.e. admin work related to topics) or producing/consuming messages from topics?

Good to see you are learning new technologies and that too Kafka! Keep learning.

Thanks,

Vinay

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/a78aea66-f822-4534-a25d-9dbe17ba830cn%40googlegroups.com.

--

Cheers,

Vinay Vyas

[Surbhi Agarwal]

Hi Vinay,

1. Querying producer and consumer would means that I would like to get the info about those producer and consumer who have published or subscribed to topics. 
2. Yes, along with this I would also like to do something like get all the producers and consumers to particular topic.

Also, I would like to know about transformational messages in Kafka. How are the messages transformed into different schema? The whole process.

Thanks,

Surbhi

[BadwolF ForeveR]

Please see answers inline below. Hope it helps.

Here is the link to Confluent documentation for more reading: https://docs.confluent.io/home/overview.html

Sent from my iPhone

On Jan 21, 2021, at 06:27, Surbhi Agarwal <surb...@gmail.com> wrote:

Hi Vinay,

1. Querying producer and consumer would means that I would like to get the info about those producer and consumer who have published or subscribed to topics. (V): both producers and consumers are not part of Kafka and they are always some external systems sending messages to Kafka or reading messages from Kafka. So typically Kafka does not care who is producing messages to a topic or topics (until ACLs are not in picture), so keeping it simple as of now. Similarly Kafka does not care about consumers but cares about CONSUMER GROUPS in which there can be many consumers. Using Kafka admin APIs/commands or rest proxy you can enquire about consumer groups.

2. Yes, along with this I would also like to do something like get all the producers and consumers to particular topic. (V): See above answer. You can get consumer names only when you query for consumer group. But that would not mean anything. Those consumers can be any kind of application which has that name and which is part of that consumer group

Also, I would like to know about transformational messages in Kafka. (V): there is no such thing in Kafka. Not sure if you are talking about SMT. If yes, then that is part of Kafka connect, not Kafka. Kafka does not know about any message schema as such. Kafka stores all messages as “bytes”

How are the messages transformed into different schema? The whole process. (V): I am not sure if I understood the question. Messages stored in Kafka do not get transformed in different schema. If you are referring to converters then it’s done within Producer and consumer libraries and still is outside of Kafka. Kafka stores messages as “bytes”.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/44e5dd06-5ec0-4910-815c-c3625b44d946n%40googlegroups.com.