Kafa REST API - do the consumers have to keep polling for data?

[pubu...@gmail.com]

Hi All,

I've been looking into the confluent Kafka REST proxy features. From what I understand the Kafka Java API for consumers is a blocking API that blocks until messages are available. But how to have that kind of behavior with the REST API? Or does the REST API consumers have to periodically poll the API for data?

Thanks,
Pubudu

[Ewen Cheslack-Postava]

Hi Pubudu,

You should poll for data, but each request will block waiting for data if enough isn't available yet when the request is made. The configuration option consumer.request.timeout.ms controls how long a REST proxy request will wait before returning. This works in conjunction with the consumer.request.max.bytes config option (and parameter to the GET requests) to determine exactly when the response will be triggered: a response will be sent either when enough bytes are available or the timeout is reached, whichever occurs first.

There's also an issue filed for adding a consumer.request.min.bytes option, which would allow for something more like long-polling: https://github.com/confluentinc/kafka-rest/issues/88

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/c536d3e3-ee0a-471b-8551-4f623810d552%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[pubudu gunawardena]

Thanks you for the response Ewen.

> You received this message because you are subscribed to a topic in the

> Google Groups "Confluent Platform" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/confluent-platform/GQHWAml_xI0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> confluent-platf...@googlegroups.com.
> To post to this group, send email to confluent...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/CAE1jLMMdecg2wNkJMgTUgmJG0AoS%2B2r_cD6bp1cqOLwQMX5-dA%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.



--
Thanks,

Pubudu

[Alan Gawthorpe]

I can't seem to get this functionality to work (testing on a Mac).

I've got a clean install of 1.0.1

In my kafka-rest.properties I have :

consumer.request.timeout.ms=10000

consumer.request.max.bytes=1

The consumer waits for the correct time but returns nothing []

If I increase the max.bytes to something like 10,000 then data is returned but only after my 10s timeout.

What am I doing wrong?!

Ta,

Alan

On Tuesday, 8 September 2015 01:41:44 UTC+1, Ewen Cheslack-Postava wrote:

Hi Pubudu,

You should poll for data, but each request will block waiting for data if enough isn't available yet when the request is made. The configuration option consumer.request.timeout.ms controls how long a REST proxy request will wait before returning. This works in conjunction with the consumer.request.max.bytes config option (and parameter to the GET requests) to determine exactly when the response will be triggered: a response will be sent either when enough bytes are available or the timeout is reached, whichever occurs first.

There's also an issue filed for adding a consumer.request.min.bytes option, which would allow for something more like long-polling: https://github.com/confluentinc/kafka-rest/issues/88

-Ewen

On Mon, Sep 7, 2015 at 3:21 AM, <pubu...@gmail.com> wrote:

Hi All,

I've been looking into the confluent Kafka REST proxy features. From what I understand the Kafka Java API for consumers is a blocking API that blocks until messages are available. But how to have that kind of behavior with the REST API? Or does the REST API consumers have to periodically poll the API for data?

Thanks,
Pubudu

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/c536d3e3-ee0a-471b-8551-4f623810d552%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[sneha george]

Hi Alan,

How did you get around this problem? Even I am running into the same issue.
If i set max_bytes=10000 in GET request, then it returns the data, else if I set it to 1, it does'nt.
Please let me know. Thanks.