How to write remote files into kafka

495 views
Skip to first unread message

Singh N

unread,
Aug 24, 2017, 5:08:17 PM8/24/17
to Confluent Platform
Hi,

I am trying to get files from remote server into kafka. Could any one please help me out on how to proceed forward with it. I am trying for something which can be implemented in java.

I heard about kafka connect and would like to know hoe to proceed with it.

Thanks

Roger Hoover

unread,
Aug 24, 2017, 9:24:12 PM8/24/17
to confluent...@googlegroups.com
Hi Singh,

You could use the spooldir connector:  https://github.com/jcustenborder/kafka-connect-spooldir

Cheers,

Roger

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/83833abe-76ca-4f44-96f4-e831df74edec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Singh N

unread,
Aug 25, 2017, 11:52:00 AM8/25/17
to Confluent Platform
Hi Roger,

Thank you, but kafka-connect-spooldir does not allow me connect to a remote server and download the files. Currently i'm using ftp to download the file to my local machine. Now i want to do the process with kafka.

Thanks
Singh


On Thursday, August 24, 2017 at 6:24:12 PM UTC-7, Roger Hoover wrote:
Hi Singh,

You could use the spooldir connector:  https://github.com/jcustenborder/kafka-connect-spooldir

Cheers,

Roger
On Thu, Aug 24, 2017 at 2:08 PM, Singh N <snaksh...@gmail.com> wrote:
Hi,

I am trying to get files from remote server into kafka. Could any one please help me out on how to proceed forward with it. I am trying for something which can be implemented in java.

I heard about kafka connect and would like to know hoe to proceed with it.

Thanks

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Roger Hoover

unread,
Aug 25, 2017, 1:24:15 PM8/25/17
to confluent...@googlegroups.com
Sorry, I didn't read carefully that you wanted to pull remote files.  I found this ftp connector that might help.


On Fri, Aug 25, 2017 at 8:52 AM, Singh N <snaksh...@gmail.com> wrote:
Hi Roger,

Thank you, but kafka-connect-spooldir does not allow me connect to a remote server and download the files. Currently i'm using ftp to download the file to my local machine. Now i want to do the process with kafka.

Thanks
Singh

On Thursday, August 24, 2017 at 6:24:12 PM UTC-7, Roger Hoover wrote:
Hi Singh,

You could use the spooldir connector:  https://github.com/jcustenborder/kafka-connect-spooldir

Cheers,

Roger
On Thu, Aug 24, 2017 at 2:08 PM, Singh N <snaksh...@gmail.com> wrote:
Hi,

I am trying to get files from remote server into kafka. Could any one please help me out on how to proceed forward with it. I am trying for something which can be implemented in java.

I heard about kafka connect and would like to know hoe to proceed with it.

Thanks

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4dcc1acb-118c-4df8-8626-717c8b168d66%40googlegroups.com.

ha...@confluent.io

unread,
Aug 25, 2017, 8:47:57 PM8/25/17
to Confluent Platform
General rule of thumb when looking for a Kafka Connector is to check the Connect Hub page at https://www.confluent.io/product/connectors/ and if it's not there then try searching github for kafka-connect-system where "system" is the name of the protocol or target system you are trying to connect to.

-hans


On Friday, August 25, 2017 at 10:24:15 AM UTC-7, Roger Hoover wrote:
Sorry, I didn't read carefully that you wanted to pull remote files.  I found this ftp connector that might help.

On Fri, Aug 25, 2017 at 8:52 AM, Singh N <snaksh...@gmail.com> wrote:
Hi Roger,

Thank you, but kafka-connect-spooldir does not allow me connect to a remote server and download the files. Currently i'm using ftp to download the file to my local machine. Now i want to do the process with kafka.

Thanks
Singh

On Thursday, August 24, 2017 at 6:24:12 PM UTC-7, Roger Hoover wrote:
Hi Singh,

You could use the spooldir connector:  https://github.com/jcustenborder/kafka-connect-spooldir

Cheers,

Roger
On Thu, Aug 24, 2017 at 2:08 PM, Singh N <snaksh...@gmail.com> wrote:
Hi,

I am trying to get files from remote server into kafka. Could any one please help me out on how to proceed forward with it. I am trying for something which can be implemented in java.

I heard about kafka connect and would like to know hoe to proceed with it.

Thanks

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages