Druid Parser With Protobuf Not working.

[Stelios Philippou]

Hello Everyone,

We have started a new project where we want to ingest various Kafka sources into a Druid on a Docker Image.

Now those sources are all in Protobuf format.

I have attached the Protobuf Format and the Druid Spec that i am using.

The connection seems to be working correctly at this moment with the SSLand i can see the flow of binary data

Running the parser at this moment will result me in the following 

2021-03-19T08:32:48,437 ERROR [MonitorScheduler-0] org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor - [15,823] unparseable events discarded. Turn on debug logging to see exception stack trace.

{
  "ingestionState": "COMPLETED",
  "unparseableEvents": {},
  "rowStats": {
    "buildSegments": {
      "processed": 0,
      "processedWithError": 0,
      "thrownAway": 0,
      "unparseable": 113636
    }
  },
  "errorMsg": null
}

And it seems that i cannot parse any of the data.

Unfortunately the logs do not provide any information on what is happening and why it is failing.

I have been checking various stuff for the past 2 days and i could not get the data to correctly parse.

What might be going wrong :

1 - Protobuf does not like inline records and thus cannot parse them.

2 - Perhaps the docker refused to fine the desc file ?

Thank you for your time,

Stelios