kafka (Json) to s3 (parquet)

[Arpit Garg]

Hi, 

Is it possible to use secor to read from a kafka topic which has json format messages and convert them to parquet format in S3. We do not have a fixed schema defined for kafka json and we do not have any schema repo/registry, can we use secor to do this dynamic json to parquet conversion and write to s3?

if yes, can you point me to any link or doc.

thanks,

Arpit

[hc...@pinterest.com]

You can raise this question in https://github.com/pinterest/secor/issues

Secor does support parquet writing, there are various fixes on the parquet writer path.  But I am not sure whether people are doing json->parquet.  Json->parquet usually needs the Avro as the intermediate step.