--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b9fa362-e44d-4581-ad34-5b72e0221f53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Does this converter meet your needs for input data? key.converter=org.apache.kafka.connect.json.JsonConverterYou can find more information on the worker configurations and what they do specifically here. You'll want to see the descriptions of key.converter and value.converter.http://docs.confluent.io/3.0.0/connect/userguide.html#common-worker-configsThen to write parquet formatted data to HDFS, you can use the configuration outlined here as an example http://docs.confluent.io/3.0.0/connect/connect-hdfs/docs/hdfs_connector.html#format-and-partitioner
On Thu, Aug 4, 2016 at 11:20 PM, <zhangxin...@gmail.com> wrote:
Hi all,I am a newer using Confluent Platform, but I used kafka for ETL pipeline before.Our scenario is below: We put data into Kafka in json-format, and want to use Kafka connector to read json-formatted data from topic and transform to parquet-format to upload to HDFS.When I followed the quick-start of HDFS CONNECTOR , I found that data should be written into Kafka using avro-format firstly, and "key.converter" is set by "io.conluent.convert.avro.AvroConvertor" in etc/connect-avro-standalone.properties. so we must modify previous codes to translate json-formatted data into avro-format first , and then using "io.conluent.convert.avro.AvroConvertor" to transform avro-formatted data to parquet-format. But we don't prefer to do that.I want to know how to parse json-formatted data in Kafka and transform to parquet-format using Kafka connector directly, can anyone help me solve this issue?Thanks a lot
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b9fa362-e44d-4581-ad34-5b72e0221f53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Does this converter meet your needs for input data? key.converter=org.apache.kafka.connect.json.JsonConverter
You can find more information on the worker configurations and what they do specifically here. You'll want to see the descriptions of key.converter and value.converter.http://docs.confluent.io/3.0.0/connect/userguide.html#common-worker-configsThen to write parquet formatted data to HDFS, you can use the configuration outlined here as an example http://docs.confluent.io/3.0.0/connect/connect-hdfs/docs/hdfs_connector.html#format-and-partitioner
On Thu, Aug 4, 2016 at 11:20 PM, <zhangxin...@gmail.com> wrote:
Hi all,I am a newer using Confluent Platform, but I used kafka for ETL pipeline before.Our scenario is below: We put data into Kafka in json-format, and want to use Kafka connector to read json-formatted data from topic and transform to parquet-format to upload to HDFS.When I followed the quick-start of HDFS CONNECTOR , I found that data should be written into Kafka using avro-format firstly, and "key.converter" is set by "io.conluent.convert.avro.AvroConvertor" in etc/connect-avro-standalone.properties. so we must modify previous codes to translate json-formatted data into avro-format first , and then using "io.conluent.convert.avro.AvroConvertor" to transform avro-formatted data to parquet-format. But we don't prefer to do that.I want to know how to parse json-formatted data in Kafka and transform to parquet-format using Kafka connector directly, can anyone help me solve this issue?Thanks a lot
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b9fa362-e44d-4581-ad34-5b72e0221f53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b9fa362-e44d-4581-ad34-5b72e0221f53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/613de208-970c-48fb-bf73-35a5b8d751a3%40googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b9fa362-e44d-4581-ad34-5b72e0221f53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/613de208-970c-48fb-bf73-35a5b8d751a3%40googlegroups.com.
--Thanks,
Ewen
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b9fa362-e44d-4581-ad34-5b72e0221f53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/613de208-970c-48fb-bf73-35a5b8d751a3%40googlegroups.com.
----Thanks,
Ewen
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/d1255e91-aa4b-4e6b-b44d-9aa689a40c9a%40googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b9fa362-e44d-4581-ad34-5b72e0221f53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/613de208-970c-48fb-bf73-35a5b8d751a3%40googlegroups.com.
--Thanks,
Ewen
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b9fa362-e44d-4581-ad34-5b72e0221f53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/613de208-970c-48fb-bf73-35a5b8d751a3%40googlegroups.com.
--Thanks,
Ewen
Hello - i'm getting the same error ..
Did you get a fix for this ?
Pls. let me know.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0b9fa362-e44d-4581-ad34-5b72e0221f53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/613de208-970c-48fb-bf73-35a5b8d751a3%40googlegroups.com.
----Thanks,
Ewen
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/eeb42d69-95db-4098-a085-7a02880305a8%40googlegroups.com.
org.apache.kafka.connect.errors.DataException: JsonDeserializer with schemas.enable requires "schema" and "payload"fields and may not contain additional fields
at org.apache.kafka.connect.json.JsonConvertor.toConnectData(JsonConvertor.java:332)'
>>>>>> an email to confluent-platform+unsub...@googlegroups.com.
>>>>>> To post to this group, send email to confluent...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/confluent-platform/0b9fa362-e44d-4581-ad34-5b72e0221f53%40googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dustin Cote
>>>>> confluent.io
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Confluent Platform" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to confluent-platform+unsub...@googlegroups.com.
>>>> To post to this group, send email to confluent...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/confluent-platform/613de208-970c-48fb-bf73-35a5b8d751a3%40googlegroups.com.
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Ewen
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Confluent Platform" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to confluent-platform+unsub...@googlegroups.com.
>> To post to this group, send email to confluent...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/confluent-platform/eeb42d69-95db-4098-a085-7a02880305a8%40googlegroups.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to confluent-platform+unsub...@googlegroups.com.
## stacktrace:
[2017-10-13 12:28:20,055] ERROR Task aggregation-local-file-source-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:141)
org.apache.kafka.connect.errors.DataException: Only Struct objects supported for [field extraction], found: java.lang.String
at org.apache.kafka.connect.transforms.util.Requirements.requireStruct(Requirements.java:45)
at org.apache.kafka.connect.transforms.ExtractField.apply(ExtractField.java:60)
at org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:39)
at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:189)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:167)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:139)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:182)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
# connect-standalone.properties
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
# connect-file-source.properties
name=aggregation-local-file-source
connector.class=FileStreamSource
tasks.max=1
file=testData/test.txt
topic=connect-test
transforms=ExtractKey,SetKey
transforms.ExtractKey.type=org.apache.kafka.connect.transforms.ExtractField$Value
transforms.ExtractKey.field=id
transforms.SetKey.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.SetKey.fields=id
# local file as input source - "testData/test.txt"
{id:1230,description:,value:2.003275776E9,timeStamp:1507748059000}
{id:1231,description:,value:3.746452528E9,timeStamp:1507748059000}
{id:1232,description:,value:7.047452615E9,timeStamp:1507748059000}
{id:1233,description:,value:1.002746454E9,timeStamp:1507748059000}
--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/e5e68227-fea4-4d11-8e2d-4149e40481aa%40googlegroups.com.To post to this group, send email to confluent-platform@googlegroups.com.