Feedback: Elasticsearch Connector

108 views
Skip to first unread message

Thomas Schulz

unread,
Nov 17, 2016, 6:48:01 AM11/17/16
to Confluent Platform
Hi guys,

I was evaluating the Elasticsearch (ES) Connector [1] for my client and (sadly) missed two key features:
  • recognition of string-encoded dates in avro objects (e.g. '2016-11-17T12:44:38.353+01:00') and mapping to ES dates
  • creating day-based ES indices via config (e.g., test-topic-2016-11-17, test-topic-2016-11-18, ...)
Another great feature would be:
  • adding custom fields via connector config (e.g., hostname, cluster name, worker name)
  • supporting ES v5.0.0+ :-)
I hope that helps :)

Kind regards,
Thomas

Gwen Shapira

unread,
Nov 17, 2016, 1:19:38 PM11/17/16
to confluent...@googlegroups.com
Hey! Thank you for evaluating!

We have a github project for the connector:
https://github.com/confluentinc/kafka-connect-elasticsearch
Feel free to open issues for your suggestions. This will help us (and
you) track them.

Gwen
> --
> You received this message because you are subscribed to the Google Groups
> "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to confluent-platf...@googlegroups.com.
> To post to this group, send email to confluent...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/c5412918-e467-48c2-bad0-1e88cf51742b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Shikhar Bhushan

unread,
Nov 17, 2016, 4:54:24 PM11/17/16
to confluent...@googlegroups.com
Hi Thomas,

Thanks for the feedback! As Gwen mentioned Github issues would be a good way for us to track requests. A couple of the features you mentioned we would probably end up handling with support for lightweight message transformations.

Regarding timestamp indexing -- I think you can already do this by defining an explicit mapping yourself with the right date format (https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html) and using `schema.ignore=true`.

For time-based routing you may want to follow https://github.com/confluentinc/kafka-connect-elasticsearch/issues/33.

For ES 5.0 - did you try it? While I haven't yet, I don't believe the REST APIs have changed incompatibly.

Best,

Shikhar

Thomas Schulz

unread,
Nov 18, 2016, 5:22:57 AM11/18/16
to Confluent Platform
Hi Shikhar,

thanks for your response.

regarding explicit date type mapping: I tried the reference you mentioned already. As long as the Kafka Connect worker (KCW) runs, ES indices are (will become) time-aware. Once the KCW restarts or another is started concurrently, it will try to override the existing index and dies due to the conflict. In summary, adjusting explicit mapping manually is unfriendly from an administrators POV :(

regarding ES v5.0.0+: The REST API is not the issue. The mapping from avro string to ES string fails because "string" is not a valid ES type anymore (https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_mapping_changes.html)

My whole point was: The ES connector is hardly usable out-of-the-box regarding AVRO due to the 2 points in my 1st post :(

Thanks for the hint, though. I might post an issue on git if time permits.

Regards,
Thomas

> To post to this group, send email to confluent...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/c5412918-e467-48c2-bad0-1e88cf51742b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

Shikhar Bhushan

unread,
Nov 18, 2016, 2:25:56 PM11/18/16
to Confluent Platform
Hi Thomas,

It was intended to not clobber any existing mapping but due to an unfortunate bug (https://github.com/confluentinc/kafka-connect-elasticsearch/pull/53) this can happen. That seems like the issue you are describing.

However, if you set `schema.ignore=true`, the connector will not attempt to create a mapping, even with this bug. That does mean you have to create it yourself, which may not be ideal for your usecase either.

Thanks for the pointer about the 'string' type issue. I have created an issue here for ES 5.0 compatibility: https://github.com/confluentinc/kafka-connect-elasticsearch/issues/55, This can also be worked around for now by using `schema.ignore=true` and creating the mapping yourself.

I am hopeful that all or most of your concerns will be addressed in the next CP release.

Best,

Shikhar

> To post to this group, send email to confluent...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/c5412918-e467-48c2-bad0-1e88cf51742b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

Thomas Schulz

unread,
Nov 21, 2016, 4:25:08 AM11/21/16
to Confluent Platform
Hi Shikhar,

thank you so much for the hint. "schema.ignore=true" was the option I was looking for. I can't believe I missed that.

Elasticsearch is inferring the mapping of the types just fine for version 2.4.1.

That was a very productive conversation :-) Have a nice week

Best regards,
Thomas
> To post to this group, send email to confluent...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/c5412918-e467-48c2-bad0-1e88cf51742b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages