Found null value for non-optional schema

[Chris Retford]

I'm running in to an exception when starting what I think is normal operation.

I'm running all services in standalone mode following the guide: ZK, kafka, schema-registry, kafka-connect. I have a table with ~360k entries, and I have verified that the id and timestamp columns are not nullable and do not contain nulls. Here's my config:

name=postgres

connector.class=io.confluent.connect.jdbc.JdbcSourceConnector

tasks.max=1

connection.url=jdbc:postgresql://localhost:5432/umod?user=***&password=***

mode=timestamp+incrementing

incrementing.column.name=id

timestamp.column.name=last_update_date

topic.prefix=mb-

table.whitelist=board_comment

And the stack trace:

[2016-03-30 17:35:48,356] ERROR Task postgres-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerSourceTask:362)

[2016-03-30 17:35:48,356] ERROR Task is being killed and will not recover until manually restarted: (org.apache.kafka.connect.runtime.WorkerSourceTask:363)

org.apache.kafka.connect.errors.DataException: Found null value for non-optional schema

at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:299)

at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:450)

at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:267)

at io.confluent.connect.avro.AvroConverter.fromConnectData(AvroConverter.java:90)

at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:142)

at org.apache.kafka.connect.runtime.WorkerSourceTask.access$600(WorkerSourceTask.java:50)

at org.apache.kafka.connect.runtime.WorkerSourceTask$WorkerSourceTaskThread.execute(WorkerSourceTask.java:356)

at org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)

[Chris Retford]

I'm also getting this warning when my data includes a column with SQL type "boolean not null default false":

[2016-03-30 18:13:11,494] WARN Ignoring record due to SQL error: (io.confluent.connect.jdbc.JdbcSourceTask:73)

org.postgresql.util.PSQLException: Bad value for type byte : f

at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getByte(AbstractJdbc2ResultSet.java:2100)

at io.confluent.connect.jdbc.DataConverter.convertFieldValue(DataConverter.java:281)

at io.confluent.connect.jdbc.DataConverter.convertRecord(DataConverter.java:68)

at io.confluent.connect.jdbc.TimestampIncrementingTableQuerier.extractRecord(TimestampIncrementingTableQuerier.java:165)

at io.confluent.connect.jdbc.JdbcSourceTask.poll(JdbcSourceTask.java:211)

at org.apache.kafka.connect.runtime.WorkerSourceTask$WorkerSourceTaskThread.execute(WorkerSourceTask.java:353)

at org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)

[C Mcc]

Chris,

Were you able to resolve this?

I can see this issue was created that relates to this problem ... https://github.com/confluentinc/kafka-connect-jdbc/issues/35

But it appears a fix was never implemented.

Thanks,

Colum

[Chris Retford]

IIRC it stems from the fact that Postgres booleans are not JDBC compliant. I wrote my own fork here:

https://github.com/pushingice/kafka-connect-jdbc/commit/38728c8458ae78971c2012220c93eb3dfb692d5b

It's basically a hack to detect postgres and handle the behavior (it also handles their JSON type). I'm not suggesting it's a long term solution.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/4Xi0--nJdK8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/e383722c-a60b-4c3b-af6b-90b40a4bba72%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[C Mcc]

Thanks very much Chris.

Maybe I should initiate a separate thread:

I'm wondering if anyone from Confluent reading this might have thoughts on how these sorts of non-JDBC-compliant problems might be overcome ... in order to make the kafka-connect-jdbc connector more resilient.  As someone not schooled in these matters, I don't suppose there's an alternative postgres driver other than this one (postgresql-9.4-1206-jdbc41.jar) ... that might play better with the connector ... and overcome some of these known incompatabilities with JDBC?