MongoDB Sink Connector

[Francesco Nobilia]

Hi there,

with my team, we are working on a MongoDB Sink Connector.

This is the link for the GitHub project https://github.com/RADAR-CNS/RADAR-MongoDbConnector

We would appreciate any comments or suggestions that will help us to enhance our implementation.

Happy to share our work with the community!

Kind Regards,

Francesco Nobilia

[Ewen Cheslack-Postava]

Francesco,

Thanks for the contribution, this looks like a great start! I've included a link to your connector on connectors.confluent.io so others can find it easily.

Have you seen our guide for connector developers? You can find it here: https://www.confluent.io/wp-content/uploads/Partner-Dev-Guide-for-Kafka-Connect.pdf?x77929 It outlines a lot of best practices and details that you may not have thought about when just trying to get the connector working for yourself. For example, one thing I checked quickly and noticed was missing is that the connector doesn't return data via the config() method on the Connector. Without this, the framework can't provide users real validation of their configs before they submit them. Small details like this can make a big difference to users.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/a227f471-b0fc-4cb1-baff-baff4b444845%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jo...@thehyve.nl]

Dear Ewen,

Op dinsdag 13 december 2016 08:17:32 UTC+1 schreef Ewen Cheslack-Postava:

Thanks for the contribution, this looks like a great start! I've included a link to your connector on connectors.confluent.io so others can find it easily.

Thanks for that!

 

Have you seen our guide for connector developers? You can find it here: https://www.confluent.io/wp-content/uploads/Partner-Dev-Guide-for-Kafka-Connect.pdf?x77929 It outlines a lot of best practices and details that you may not have thought about when just trying to get the connector working for yourself. For example, one thing I checked quickly and noticed was missing is that the connector doesn't return data via the config() method on the Connector. Without this, the framework can't provide users real validation of their configs before they submit them. Small details like this can make a big difference to users.

Thanks for the comprehensive document. We've gone ahead and implemented most of the suggestions (https://github.com/RADAR-CNS/RADAR-MongoDbConnector) but we're still lacking System Tests as described in the document. Would it be acceptable to do the system tests with docker-compose on Travis?

Best wishes,

Joris Borgdorff

[Ewen Cheslack-Postava]

Joris,

Awesome, cleaning up all those details really makes a difference to end users.

For system tests, Docker-based tests are definitely a lightweight way to write them! Kafka and Confluent use a small framework called ducktape (https://github.com/confluentinc/ducktape) for writing system tests. It's a simple, python based tool that abstracts out the concept of a cluster and formalizes the idea of services & tests in a way that makes it pretty straightforward to write new tests (albeit with some of the unavoidable challenges you'll always get with system tests, e.g. longer test turn around times).

Our system tests for connectors focus on a reasonable end-to-end exercise of the connector -- once you've got all your services allocated, just make sure data is actually delivered as expected and try to exercise lots of functionality in one test since the overhead of a system test is pretty high. We tend to focus on things like validation of delivery of data, schema migration (and its impact on the external system), and robustness to failures (i.e. exactly once delivery or at least once delivery) since these are things that aren't easily validated in unit tests.

 

-Ewen

Best wishes,

Joris Borgdorff

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/092e0231-b331-42d1-b937-72049eb8a26f%40googlegroups.com.

[nive...@thehyve.nl]

Dear Ewen,

We have renamed our MongoDb Sink Connector, to fit our naming conventions of repositories.
The updated Github repository is now https://github.com/RADAR-CNS/MongoDb-Sink-Connector .

Could you please update the link on connectors.confluent.io , so that our repository remains reachable and up-to-date.

Thanks in advance.

Best regards,
Nivethika

On Sunday, 18 December 2016 04:07:50 UTC+1, Ewen Cheslack-Postava wrote:

Joris,

On Wed, Dec 14, 2016 at 5:52 AM, <jo...@thehyve.nl> wrote:

Dear Ewen,

Op dinsdag 13 december 2016 08:17:32 UTC+1 schreef Ewen Cheslack-Postava:

Thanks for the contribution, this looks like a great start! I've included a link to your connector on connectors.confluent.io so others can find it easily.

Thanks for that!

 

Have you seen our guide for connector developers? You can find it here: https://www.confluent.io/wp-content/uploads/Partner-Dev-Guide-for-Kafka-Connect.pdf?x77929 It outlines a lot of best practices and details that you may not have thought about when just trying to get the connector working for yourself. For example, one thing I checked quickly and noticed was missing is that the connector doesn't return data via the config() method on the Connector. Without this, the framework can't provide users real validation of their configs before they submit them. Small details like this can make a big difference to users.

Thanks for the comprehensive document. We've gone ahead and implemented most of the suggestions (https://github.com/RADAR-CNS/RADAR-MongoDbConnector) but we're still lacking System Tests as described in the document. Would it be acceptable to do the system tests with docker-compose on Travis?

Awesome, cleaning up all those details really makes a difference to end users.

For system tests, Docker-based tests are definitely a lightweight way to write them! Kafka and Confluent use a small framework called ducktape (https://github.com/confluentinc/ducktape) for writing system tests. It's a simple, python based tool that abstracts out the concept of a cluster and formalizes the idea of services & tests in a way that makes it pretty straightforward to write new tests (albeit with some of the unavoidable challenges you'll always get with system tests, e.g. longer test turn around times).

Our system tests for connectors focus on a reasonable end-to-end exercise of the connector -- once you've got all your services allocated, just make sure data is actually delivered as expected and try to exercise lots of functionality in one test since the overhead of a system test is pretty high. We tend to focus on things like validation of delivery of data, schema migration (and its impact on the external system), and robustness to failures (i.e. exactly once delivery or at least once delivery) since these are things that aren't easily validated in unit tests.

 

-Ewen

Best wishes,

Joris Borgdorff

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

[Ewen Cheslack-Postava]

Updated, thanks for the heads up.

-Ewen

Joris,

Best wishes,

Joris Borgdorff

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/092e0231-b331-42d1-b937-72049eb8a26f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/ec4ea3f5-e122-427b-bdbc-c8f4840e823a%40googlegroups.com.