How to publish and subscribe to Kafka topic(Consumer and Producer)

[sanoo singh]

I want to put the message on the Kafka queue using RIDE. Is there any library keyword available?

[Hélio Guilherme]

What is Kafka?

I know that RIDE is an Robot Framework IDE, which allows to edit files and call pybot.

Please give more info. If you interact with a Consumer and a Producer, show us an example (I'm sure we could translate into Robot Framework script).

On Sat, Dec 26, 2015 at 10:02 PM, sanoo singh <sanook...@gmail.com> wrote:

I want to put the message on the Kafka queue using RIDE. Is there any library keyword available?

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[David]

I doubt any kafka library exists for RF, however, you can wrap your own around the Python (or Java/.NET via Jython/IronPython) clients for kafka or any other kafka language client for use with the remote servers as remote libraries for RF. A list of kafka clients can be found here:

https://cwiki.apache.org/confluence/display/KAFKA/Clients

or search online for kafka & your language of choice

One would probably just need to wrap variants of the following actions as keywords in a kafka library:

* consume - high level consumer version that uses zookeeper & no explicit offsets

* consume - low level simple consumer version using kafka & offsets, no zookeeper

* produce

* get offset (for low level simple consumer) - arguments would allow getting offset value for current/latest offset, or earliest offset

* get list of topics/queues on kafka

* get details about a given topic/queue ok kafka

* create a topic/queue

* delete a topic/queue

produce & consume would likely deal with an array of messages (typically JSON strings)

[sanoo singh]

Kafka is a asynchronous message broker.

So I have a topic created by our developer, which consume the message from the Kafka message queue.

I as a tester need to connect to the Kafka port and have to publish my message on the topic. That should be consumed.

On Saturday, December 26, 2015 at 4:27:49 PM UTC-6, Hélio Guilherme wrote:

What is Kafka?

I know that RIDE is an Robot Framework IDE, which allows to edit files and call pybot.

Please give more info. If you interact with a Consumer and a Producer, show us an example (I'm sure we could translate into Robot Framework script).

On Sat, Dec 26, 2015 at 10:02 PM, sanoo singh <sanook...@gmail.com> wrote:

I want to put the message on the Kafka queue using RIDE. Is there any library keyword available?

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsub...@googlegroups.com.

[sanoo singh]

I am new working with RF and I have used all the existing libraries written in Python for testing Rest Client. Have also tried installing Kafka Library from https://github.com/dpkp/kafka-python but I am unable to import those producer and consumer libraries to RIDE.

Basically I have the message(JSON) that is a variable for me, and I have to put it on the topic.

[David]

I've never worked with RIDE, so I'm not sure if you can directly import non-RF code/modules/libraries into RIDE. The library/module may have to follow RF library format/conventions in order to be imported. If that is the case, you would first have to wrap kafka-python into a custom RF library that RF understands. Then you can import this wrapper library into RIDE. Perhaps other RF/RIDE users/developers of this group can clarify expectations here.

As for wrapping kafka-python, you would as I mentioned in previous email create & implement keyword methods/functions I listed such as this pseudo code sample:

def produce(kafka_broker_list, topic, json_message)

  # put in the necessary kafka_python code to actually produce the message to topic, given the arguments provided to this method/keyword

full details on how to write/define a RF library here: http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#creating-test-libraries

Also, regarding the asynchronous messaging of kafka, depending on how you set up the test environment, this could be a potential issue for you. If you have an isolated test environment (1 producer, 1 consumer), then the asynchronous behavior will behave quite synchronously, and things will just work. As you produce message in your test, the consumer under test should have no problems consuming it. That and the fact that there are no other producers & consumers interfering by producing other messages or consuming your test messages unexpectedly.

But in a real live-like environment (staging or production), this asynchronous behavior will be hard to test deterministically as it will be hard to guarantee/correlate that a given message produced will be consumed right after because of other producers pushing in their non-test messages, and other consumers competing to consume the messages, whether they are test messages or not. So that needs to be accounted for and set up (or routed/filtered) properly or else you find you've sent a message but it doesn't appear to be consumed (or consumed too fast before you could validate). It's like testing multi-threading without explicit synchronization in your test.

[David]

As a follow up, if I were you, I'd focus on getting kafka_python to work in a simple test/demo standalone script (not used with RF/RIDE) and make sure you can produce such that your consumer can consume it. When that works, then work on adapting it to work for RF/RIDE.

Also, have you used/created RF user keywords (e.g. RF resource files, not code-based RF libraries)? See http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#creating-user-keywords

Because with that, you can kind of wrap kafka with the kafka command line scripts and tools without dealing with much coding by making OperatingSystem library or shell calls to those scripts/tools, for example these:

https://cwiki.apache.org/confluence/display/KAFKA/System+Tools

https://github.com/edenhill/kafkacat

along with kafka-console-consumer.sh and kafka-console-producer.sh

In this case, I would also make sure you can produce successfully for consumer to consume with these shell scripts/tools first then integrate with RF/RIDE

[nadim]

@Sanoo any luck around kafka? I have similar requirements 

[Haythem HMILA]

I think this may help you

robotframework-ConfluentKafkaLibrary