Email streaming using Kafka connect

[Vijaykumar Gundavarapu]

Hi,

I am wondering if we can use kafka connect to stream email messages to the broker. Any idea whether this is possible using kafka connect or any other platform?

Regards,

Vijay

[Shikhar Bhushan]

Hi Vijaykumar,

That's an interesting use-case :-) I guess you want to target a protocol like POP/IMAP?

It is certainly possible to build such a source connector, I'm not aware of an existing one. You can refer to http://docs.confluent.io/3.1.0/connect/devguide.html. The source code for existing open-source connectors https://www.confluent.io/product/connectors/ may be helpful as you develop.

Best,

Shikhar

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/94ba1de6-fc26-44be-bb4d-723c7fa48580%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vijaykumar Gundavarapu]

Thanks Shikhar.

Yes I am trying to connect to exchange server using IMAP.

I guess my problem with writing own source producer is that we already have existing logic in python where we connect to the server and try to manually parse the message, but that is error prone and as we are migrating to kafka, I was thinking on the lines of out of the box solution for this.

On Mon, Nov 14, 2016 at 3:57 PM, Shikhar Bhushan <shi...@confluent.io> wrote:

Hi Vijaykumar,

That's an interesting use-case :-) I guess you want to target a protocol like POP/IMAP?

It is certainly possible to build such a source connector, I'm not aware of an existing one. You can refer to http://docs.confluent.io/3.1.0/connect/devguide.html. The source code for existing open-source connectors https://www.confluent.io/product/connectors/ may be helpful as you develop.

Best,

Shikhar

On Mon, Nov 14, 2016 at 12:52 PM Vijaykumar Gundavarapu <vijay.gu...@gmail.com> wrote:

Hi,

I am wondering if we can use kafka connect to stream email messages to the broker. Any idea whether this is possible using kafka connect or any other platform?

Regards,

Vijay

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/94ba1de6-fc26-44be-bb4d-723c7fa48580%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/50ieoCNoTy8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAPXNJrV_gCX19aA3a-7MtF-C-4X%2BqZc%2BJoO5%2Bzh1R7W7HOXd0A%40mail.gmail.com.

[shilp...@gmail.com]

I need the same. I need to stream emails to Kafka from inbox for being processed by my application.

Could you tell us how you finally did it? Does kafka connectors give us such an option?

[Vijay]

I ended up using python imap library to connect to my mailbox and then started streaming them by using infinite while loop . Whenever it receives any messages , it will read them and then my application will take care of enriching and then publish it to Kafka 

Sent from my iPhone

--

You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/50ieoCNoTy8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/e360454d-6487-4217-9022-eede77a5d23d%40googlegroups.com.

[shilp...@gmail.com]

Okay. I am already doing that. We wanted to use a better approach by using something like Kafka connectors. But I couldn't find much details about the same. Thanks for your reply.

On Thursday, March 9, 2017 at 9:21:09 PM UTC+5:30, Vijaykumar Gundavarapu wrote:

I ended up using python imap library to connect to my mailbox and then started streaming them by using infinite while loop . Whenever it receives any messages , it will read them and then my application will take care of enriching and then publish it to Kafka 

Sent from my iPhone

On Mar 9, 2017, at 10:16 AM, shilp...@gmail.com wrote:

I need the same. I need to stream emails to Kafka from inbox for being processed by my application.

Could you tell us how you finally did it? Does kafka connectors give us such an option?

On Tuesday, November 15, 2016 at 2:22:28 AM UTC+5:30, Vijaykumar Gundavarapu wrote:

Hi,

I am wondering if we can use kafka connect to stream email messages to the broker. Any idea whether this is possible using kafka connect or any other platform?

Regards,

Vijay

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/50ieoCNoTy8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.