Amazon SQS Kafka Connect plugin

[saket...@hivehome.com]

Guys,

We at CentricaConnectedHomes have built a Amazon SQS source connector for Kafka Connect. https://github.com/ConnectedHomes/sqs-kafka-connect

Please have a look at the connector and feel free to provide any feedback!

It would be great if we could get it added to the list at https://www.confluent.io/product/connectors/

Regards,

Saket

[Ewen Cheslack-Postava]

Saket,

This looks great, thanks for contributing it!

We have a guide for partners that walks through a bunch of details you'll want to make sure you've addressed to get a feature complete connector that takes advantage of all of Kafka Connect's features. You can find it on the sidebar of https://www.confluent.io/product/connectors/ titled "Partner Development Guide for Kafka Connect".

One example is that you expose the configuration via a ConfigDef (great!) but don't use all the features of ConfigDef (validators, grouping, etc), which will result in a pretty basic UI for any frontend interface for Connect and means users can't be confident that their connector config is correct before actually creating the connector (you can use the connector plugin validation API to do this http://docs.confluent.io/3.1.1/connect/restapi.html#put--connector-plugins-(string-name)-config-validate).

Also, from the README I'm curious, when you say

> Pause and Resume operations are not supported by the connector

what do you mean? In Connect, the framework handles pause/resume just by stopping/starting the connector, so normally these would come for free.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/12302c06-2d7b-4acc-b8e1-70fe0f59df85%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[saket...@hivehome.com]

Ewen,

I'll look at the Partner Development Guide. Thanks for pointing that out. I'll look at the validation bit as well.

The reason Pause and Resume does not work properly (I think) is because the thread on the task is polling for a new record and I pause the connector, the thread of each task is still waiting for a new record. At this point when a message arrives on SQS it is still processed even though the connector is paused. The connector is going to process n messages after it has been paused where n is the number of tasks for the connector. I am not sure if this is the expected behaviour of the connector hence I added the statement that pause/resume was unsupported. 

Regards,

Saket

On Friday, 25 November 2016 22:21:18 UTC, Ewen Cheslack-Postava wrote:

Saket,

This looks great, thanks for contributing it!

We have a guide for partners that walks through a bunch of details you'll want to make sure you've addressed to get a feature complete connector that takes advantage of all of Kafka Connect's features. You can find it on the sidebar of https://www.confluent.io/product/connectors/ titled "Partner Development Guide for Kafka Connect".

One example is that you expose the configuration via a ConfigDef (great!) but don't use all the features of ConfigDef (validators, grouping, etc), which will result in a pretty basic UI for any frontend interface for Connect and means users can't be confident that their connector config is correct before actually creating the connector (you can use the connector plugin validation API to do this http://docs.confluent.io/3.1.1/connect/restapi.html#put--connector-plugins-(string-name)-config-validate).

Also, from the README I'm curious, when you say

> Pause and Resume operations are not supported by the connector

what do you mean? In Connect, the framework handles pause/resume just by stopping/starting the connector, so normally these would come for free.

-Ewen

On Fri, Nov 25, 2016 at 8:59 AM, <saket...@hivehome.com> wrote:

Guys,

We at CentricaConnectedHomes have built a Amazon SQS source connector for Kafka Connect. https://github.com/ConnectedHomes/sqs-kafka-connect

Please have a look at the connector and feel free to provide any feedback!

It would be great if we could get it added to the list at https://www.confluent.io/product/connectors/

Regards,

Saket

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/12302c06-2d7b-4acc-b8e1-70fe0f59df85%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Ewen Cheslack-Postava]

Saket,

I see. Normally for source connectors you don't need a separate background thread because you can just fetch data in the poll() call and return it. However, whether you use the background thread or not, the expectation is that you'll try to interrupt any outstanding processing when SourceTask.stop() is invoked (see http://docs.confluent.io/3.1.1/connect/javadocs/org/apache/kafka/connect/source/SourceTask.html#stop() for details). It's best if you can immediately interrupt, but if you can't, the framework will wait as long as it reasonably can for any poll() calls to return before proceeding with any shutdown/rebalancing.

If the library you're working with doesn't make it easy to interrupt ongoing work, your best bet is to use reasonably small timeouts on your requests as long as it won't incur excessive costs otherwise. Note that some libraries might be interruptible but not really document it (e.g. via a simple Thread.interrupt() call).

-Ewen

On Mon, Nov 28, 2016 at 5:16 AM, <saket...@hivehome.com> wrote:

Ewen,

I'll look at the Partner Development Guide. Thanks for pointing that out. I'll look at the validation bit as well.

The reason Pause and Resume does not work properly (I think) is because the thread on the task is polling for a new record and I pause the connector, the thread of each task is still waiting for a new record. At this point when a message arrives on SQS it is still processed even though the connector is paused. The connector is going to process n messages after it has been paused where n is the number of tasks for the connector. I am not sure if this is the expected behaviour of the connector hence I added the statement that pause/resume was unsupported. 

Regards,

Saket

On Friday, 25 November 2016 22:21:18 UTC, Ewen Cheslack-Postava wrote:

Saket,

This looks great, thanks for contributing it!

We have a guide for partners that walks through a bunch of details you'll want to make sure you've addressed to get a feature complete connector that takes advantage of all of Kafka Connect's features. You can find it on the sidebar of https://www.confluent.io/product/connectors/ titled "Partner Development Guide for Kafka Connect".

One example is that you expose the configuration via a ConfigDef (great!) but don't use all the features of ConfigDef (validators, grouping, etc), which will result in a pretty basic UI for any frontend interface for Connect and means users can't be confident that their connector config is correct before actually creating the connector (you can use the connector plugin validation API to do this http://docs.confluent.io/3.1.1/connect/restapi.html#put--connector-plugins-(string-name)-config-validate).

Also, from the README I'm curious, when you say

> Pause and Resume operations are not supported by the connector

what do you mean? In Connect, the framework handles pause/resume just by stopping/starting the connector, so normally these would come for free.

-Ewen

On Fri, Nov 25, 2016 at 8:59 AM, <saket...@hivehome.com> wrote:

Guys,

We at CentricaConnectedHomes have built a Amazon SQS source connector for Kafka Connect. https://github.com/ConnectedHomes/sqs-kafka-connect

Please have a look at the connector and feel free to provide any feedback!

It would be great if we could get it added to the list at https://www.confluent.io/product/connectors/

Regards,

Saket

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/12302c06-2d7b-4acc-b8e1-70fe0f59df85%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/6e474afa-0118-4875-b808-5e50b8e2857c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen