Kafka FileStreamSource sending record with null key

309 views
Skip to first unread message

austin solomon

unread,
Nov 8, 2016, 8:43:48 AM11/8/16
to Confluent Platform
Hi ,

I am using kafka FileStreamSource connector to read a text file and sink it to Ignite's cache. 

I could able to sink the data into ignite's cache through kafka console producer, However when i tried to pull the data from  FileStreamSource connector I got an Error :
"Failed to stream a record with null key!".

Is it possible to send a key FileStreamSource connector, like how we are doing it in console producer by providing --property parse.key=true --property key.separator=,
or parse file lines and insert them into Kafka with keys..?

Thanks,
Austin

Ewen Cheslack-Postava

unread,
Nov 8, 2016, 7:54:57 PM11/8/16
to Confluent Platform
It is not possible with the connector as it stands today -- it is an intentionally simple connector that just takes each line and emits it as a string value (with no key).

You might try https://github.com/jcustenborder/kafka-connect-spooldir, though I think it only supports CSV at the moment. Alternatively, small modifications to the file connector would support your use case.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0e6ace95-e8b1-42b4-9b02-d5703b88b7d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen

austin solomon

unread,
Nov 12, 2016, 12:38:57 AM11/12/16
to Confluent Platform
Hi Ewen,

Thanks for your reply.

If I want to modify file source connector, how to set values to key Object.

I am getting the following result when i printed the Record :
SinkRecord{kafkaOffset=7, timestampType=CreateTime} ConnectRecord{topic='ignitetest', kafkaPartition=1, key=null, value=k2,v2, timestamp=1478928520407}

Can you please help me on this.

Thanks,
Austin


On Wednesday, November 9, 2016 at 6:24:57 AM UTC+5:30, Ewen Cheslack-Postava wrote:
It is not possible with the connector as it stands today -- it is an intentionally simple connector that just takes each line and emits it as a string value (with no key).

You might try https://github.com/jcustenborder/kafka-connect-spooldir, though I think it only supports CSV at the moment. Alternatively, small modifications to the file connector would support your use case.

-Ewen
On Tue, Nov 8, 2016 at 5:43 AM, austin solomon <austin.s...@gmail.com> wrote:
Hi ,

I am using kafka FileStreamSource connector to read a text file and sink it to Ignite's cache. 

I could able to sink the data into ignite's cache through kafka console producer, However when i tried to pull the data from  FileStreamSource connector I got an Error :
"Failed to stream a record with null key!".

Is it possible to send a key FileStreamSource connector, like how we are doing it in console producer by providing --property parse.key=true --property key.separator=,
or parse file lines and insert them into Kafka with keys..?

Thanks,
Austin

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.



--
Thanks,
Ewen

Ewen Cheslack-Postava

unread,
Nov 12, 2016, 12:44:07 AM11/12/16
to Confluent Platform
Austin,

Not sure I understand your last question, but based on your previous error it seems you would need to modify the code to include a key. You'd probably need to modify this line:


which creates the SourceRecord, which gets passed to the framework, converted to byte[] data for both the key and value, and then passed to a KafkaProducer to be written into Kafka. If you included a key and key schema, it would have a non-null key and (presumably) not cause an error downstream.

-Ewen

On Fri, Nov 11, 2016 at 9:38 PM, austin solomon <austin.s...@gmail.com> wrote:
Hi Ewen,

Thanks for your reply.

If I want to modify file source connector, how to set values to key Object.

I am getting the following result when i printed the Record :
SinkRecord{kafkaOffset=7, timestampType=CreateTime} ConnectRecord{topic='ignitetest', kafkaPartition=1, key=null, value=k2,v2, timestamp=1478928520407}

Can you please help me on this.

Thanks,
Austin

On Wednesday, November 9, 2016 at 6:24:57 AM UTC+5:30, Ewen Cheslack-Postava wrote:
It is not possible with the connector as it stands today -- it is an intentionally simple connector that just takes each line and emits it as a string value (with no key).

You might try https://github.com/jcustenborder/kafka-connect-spooldir, though I think it only supports CSV at the moment. Alternatively, small modifications to the file connector would support your use case.

-Ewen
On Tue, Nov 8, 2016 at 5:43 AM, austin solomon <austin.s...@gmail.com> wrote:
Hi ,

I am using kafka FileStreamSource connector to read a text file and sink it to Ignite's cache. 

I could able to sink the data into ignite's cache through kafka console producer, However when i tried to pull the data from  FileStreamSource connector I got an Error :
"Failed to stream a record with null key!".

Is it possible to send a key FileStreamSource connector, like how we are doing it in console producer by providing --property parse.key=true --property key.separator=,
or parse file lines and insert them into Kafka with keys..?

Thanks,
Austin

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/50f7704e-e1f3-470e-ad22-72d4426a04fb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen
Reply all
Reply to author
Forward
0 new messages