debezium mysql connector is failing with java heap space issue

21 views
Skip to first unread message

NAGARJUNA D.N

unread,
Apr 19, 2021, 12:49:59 PMApr 19
to debezium

Hi,

we have debezium mysql connectors deployed on GKE cluster. But the pod is failing with Caused by: java.lang.OutOfMemoryError: Java heap space . Increased jvm options to 10g, but still its failing with the same issue.

Below are my config properties:

singleTopicMode=true 
databaseManagementSystem=mysql 
debezium.max.request.size=1048 
debezium.max.queue.size=65536 
debezium.max.batch.size=16384 
debezium.HEAP_OPTS='-Xms10g -Xmx10g'

Complete log:

org.apache.kafka.connect.errors.ConnectException: java.lang.OutOfMemoryError: Java heap space at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:275) at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:49) at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:791) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596) at java.lang.StringBuilder.append(StringBuilder.java:190) at java.io.BufferedReader.readLine(BufferedReader.java:358) at java.io.BufferedReader.readLine(BufferedReader.java:389) at java.nio.file.Files.readAllLines(Files.java:3205) at io.debezium.relational.history.FileDatabaseHistory.lambda$recoverRecords$2(FileDatabaseHistory.java:136) at io.debezium.relational.history.FileDatabaseHistory$$Lambda$181/230925968.run(Unknown Source) at io.debezium.util.FunctionalReadWriteLock.write(FunctionalReadWriteLock.java:96) at io.debezium.relational.history.FileDatabaseHistory.recoverRecords(FileDatabaseHistory.java:133) at io.debezium.relational.history.AbstractDatabaseHistory.recover(AbstractDatabaseHistory.java:73) at io.debezium.connector.mysql.MySqlSchema.loadHistory(MySqlSchema.java:252) at io.debezium.connector.mysql.MySqlTaskContext.loadHistory(MySqlTaskContext.java:165) at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:105) ... 7 more

jiri.p...@gmail.com

unread,
Apr 20, 2021, 1:33:51 AMApr 20
to debezium
Hi,

could you please verify if the heap opts were really applied to the JVM?

J.

NAGARJUNA D.N

unread,
Apr 20, 2021, 1:47:28 AMApr 20
to debe...@googlegroups.com
Yes it's.
the log:

, offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore, java.vm.specification.vendor=Oracle Corporation, max.request.size=104857, os.name=Linux, sun.jnu.encoding=UTF-8, HEAP_OPTS='-Xms10g -Xmx10g', java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib, max.queue.size=65536, java.class.version=52.0, java.specification.name=Java Platform API Specification, max.batch.size=16384, sun.management.compiler=HotSpot 64-Bit Tiered Compilers, , os.version=5.4.89+, user.home=/root, user.timezone=Etc/UTC, java.awt.printerjob=sun.print.PSPrinterJob, file.encoding=UTF-8, java.specification.version=1.8, user.name=root, java.class.path=/opt/dataflow/cdc/resources:/opt/dataflow/cdc/classes:/opt/dataflow/cdc/libs/conscrypt-openjdk-uber-1.3.0.jar:/opt/dataflow/cdc/libs/error_prone_annotations-2.1.3.jar:/opt/dataflow/cdc/libs/fastjson-1.2.68.jar:/opt/dataflow/cdc/libs/grpc-protobuf-lite-1.25.0.jar:/opt/dataflow/cdc/libs/grpc-alts-1.28.1.jar:/opt/dataflow/cdc/libs/grpc-google-cloud-pubsub-v1-1.64.0.jar:/opt/dataflow/cdc/libs/grpc-stub-1.25.0.jar:/opt/dataflow/cdc/libs/google-cloud-core-grpc-1.82.0.jar:/opt/dataflow/cdc/libs/commons-csv-1.8.jar:/opt/dataflow/cdc/libs/connect-file-2.2.0.jar:/opt/dataflow/cdc/libs/jetty-continuation-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/proto-google-cloud-datacatalog-v1-0.35.0.jar:/opt/dataflow/cdc/libs/connect-runtime-2.2.0.jar:/opt/dataflow/cdc/libs/grpc-netty-shaded-1.25.0.jar:/opt/dataflow/cdc/libs/proto-google-cloud-pubsub-v1-1.64.0.jar:/opt/dataflow/cdc/libs/jersey-server-2.27.jar:/opt/dataflow/cdc/libs/j2objc-annotations-1.1.jar:/opt/dataflow/cdc/libs/beam-vendor-calcite-1_20_0-0.1.jar:/opt/dataflow/cdc/libs/gax-grpc-1.56.0.jar:/opt/dataflow/cdc/libs/slf4j-api-1.7.25.jar:/opt/dataflow/cdc/libs/beam-sdks-java-core-2.25.0.jar:/opt/dataflow/cdc/libs/mongo-java-driver-3.9.1.jar:/opt/dataflow/cdc/libs/gax-1.53.1.jar:/opt/dataflow/cdc/libs/jetty-client-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/grpc-protobuf-1.25.0.jar:/opt/dataflow/cdc/libs/jackson-jaxrs-base-2.9.8.jar:/opt/dataflow/cdc/libs/jetty-http-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/google-cloud-datacatalog-0.35.0.jar:/opt/dataflow/cdc/libs/annotations-4.1.1.4.jar:/opt/dataflow/cdc/libs/javax.ws.rs-api-2.1.1.jar:/opt/dataflow/cdc/libs/janino-3.0.11.jar:/opt/dataflow/cdc/libs/grpc-api-1.25.0.jar:/opt/dataflow/cdc/libs/connect-api-2.2.0.jar:/opt/dataflow/cdc/libs/jackson-annotations-2.10.2.jar:/opt/dataflow/cdc/libs/snappy-java-1.1.4.jar:/opt/dataflow/cdc/libs/grpc-grpclb-1.28.1.jar:/opt/dataflow/cdc/libs/connect-transforms-2.2.0.jar:/opt/dataflow/cdc/libs/opencensus-api-0.24.0.jar:/opt/dataflow/cdc/libs/kafka-tools-2.2.0.jar:/opt/dataflow/cdc/libs/proto-google-common-protos-1.17.0.jar:/opt/dataflow/cdc/libs/beam-runners-direct-java-2.25.0.jar:/opt/dataflow/cdc/libs/jackson-datatype-jdk8-2.9.8.jar:/opt/dataflow/cdc/libs/javax.inject-1.jar:/opt/dataflow/cdc/libs/log4j-1.2.17.jar:/opt/dataflow/cdc/libs/beam-sdks-java-extensions-sql-2.25.0.jar:/opt/dataflow/cdc/libs/plexus-utils-3.1.0.jar:/opt/dataflow/cdc/libs/google-http-client-1.34.0.jar:/opt/dataflow/cdc/libs/protobuf-java-3.10.0.jar:/opt/dataflow/cdc/libs/commons-collections-3.2.2.jar:/opt/dataflow/cdc/libs/jackson-core-2.10.2.jar:/opt/dataflow/cdc/libs/grpc-google-cloud-datacatalog-v1beta1-0.35.0.jar:/opt/dataflow/cdc/libs/commons-logging-1.2.jar:/opt/dataflow/cdc/libs/beam-model-job-management-2.25.0.jar:/opt/dataflow/cdc/libs/gson-2.7.jar:/opt/dataflow/cdc/libs/kafka-log4j-appender-2.2.0.jar:/opt/dataflow/cdc/libs/args4j-2.33.jar:/opt/dataflow/cdc/libs/proto-google-cloud-datacatalog-v1beta1-0.35.0.jar:/opt/dataflow/cdc/libs/google-auth-library-credentials-0.16.2.jar:/opt/dataflow/cdc/libs/commons-text-1.6.jar:/opt/dataflow/cdc/libs/connect-json-2.2.0.jar:/opt/dataflow/cdc/libs/lz4-java-1.5.0.jar:/opt/dataflow/cdc/libs/commons-compiler-3.0.11.jar:/opt/dataflow/cdc/libs/debezium-connector-mysql-0.9.5.Final.jar:/opt/dataflow/cdc/libs/osgi-resource-locator-1.0.1.jar:/opt/dataflow/cdc/libs/jsr305-3.0.2.jar:/opt/dataflow/cdc/libs/paranamer-2.7.jar:/opt/dataflow/cdc/libs/grpc-auth-1.28.1.jar:/opt/dataflow/cdc/libs/argparse4j-0.7.0.jar:/opt/dataflow/cdc/libs/antlr4-runtime-4.7.jar:/opt/dataflow/cdc/libs/grpc-core-1.25.0.jar:/opt/dataflow/cdc/libs/opencensus-contrib-grpc-metrics-0.21.0.jar:/opt/dataflow/cdc/libs/jetty-security-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/commons-codec-1.14.jar:/opt/dataflow/cdc/libs/javax.inject-2.5.0-b42.jar:/opt/dataflow/cdc/libs/jersey-media-jaxb-2.27.jar:/opt/dataflow/cdc/libs/slf4j-log4j12-1.7.25.jar:/opt/dataflow/cdc/libs/debezium-embedded-0.9.5.Final.jar:/opt/dataflow/cdc/libs/httpclient-4.5.10.jar:/opt/dataflow/cdc/libs/checker-qual-2.0.0.jar:/opt/dataflow/cdc/libs/beam-vendor-guava-26_0-jre-0.1.jar:/opt/dataflow/cdc/libs/hk2-utils-2.5.0-b42.jar:/opt/dataflow/cdc/libs/protobuf-java-util-3.7.1.jar:/opt/dataflow/cdc/libs/xz-1.8.jar:/opt/dataflow/cdc/libs/beam-vendor-grpc-1_26_0-0.3.jar:/opt/dataflow/cdc/libs/animal-sniffer-annotations-1.14.jar:/opt/dataflow/cdc/libs/google-http-client-jackson2-1.34.0.jar:/opt/dataflow/cdc/libs/maven-artifact-3.6.0.jar:/opt/dataflow/cdc/libs/auto-value-annotations-1.7.jar:/opt/dataflow/cdc/libs/joda-time-2.10.5.jar:/opt/dataflow/cdc/libs/reflections-0.9.11.jar:/opt/dataflow/cdc/libs/commons-beanutils-1.9.3.jar:/opt/dataflow/cdc/libs/jersey-hk2-2.27.jar:/opt/dataflow/cdc/libs/mysql-binlog-connector-java-0.19.1.jar:/opt/dataflow/cdc/libs/debezium-ddl-parser-0.9.5.Final.jar:/opt/dataflow/cdc/libs/grpc-context-1.25.0.jar:/opt/dataflow/cdc/libs/jackson-databind-2.10.2.jar:/opt/dataflow/cdc/libs/threetenbp-1.4.3.jar:/opt/dataflow/cdc/libs/activation-1.1.1.jar:/opt/dataflow/cdc/libs/cdc-common-0.1.jar:/opt/dataflow/cdc/libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/dataflow/cdc/libs/avro-1.8.2.jar:/opt/dataflow/cdc/libs/jetty-server-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/jaxb-api-2.3.0.jar:/opt/dataflow/cdc/libs/zstd-jni-1.3.8-1.jar:/opt/dataflow/cdc/libs/beam-model-pipeline-2.25.0.jar:/opt/dataflow/cdc/libs/commons-configuration2-2.5.jar:/opt/dataflow/cdc/libs/jetty-util-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/beam-sdks-java-io-mongodb-2.25.0.jar:/opt/dataflow/cdc/libs/google-cloud-pubsub-1.82.0.jar:/opt/dataflow/cdc/libs/commons-compress-1.8.1.jar:/opt/dataflow/cdc/libs/hk2-api-2.5.0-b42.jar:/opt/dataflow/cdc/libs/jersey-container-servlet-core-2.27.jar:/opt/dataflow/cdc/libs/jetty-servlet-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/jersey-container-servlet-2.27.jar:/opt/dataflow/cdc/libs/google-auth-library-oauth2-http-0.20.0.jar:/opt/dataflow/cdc/libs/log4j-api-2.6.2.jar:/opt/dataflow/cdc/libs/jackson-core-asl-1.9.13.jar:/opt/dataflow/cdc/libs/jersey-common-2.27.jar:/opt/dataflow/cdc/libs/jackson-mapper-asl-1.9.13.jar:/opt/dataflow/cdc/libs/javax.annotation-api-1.3.2.jar:/opt/dataflow/cdc/libs/jetty-servlets-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/opencensus-contrib-http-util-0.24.0.jar:/opt/dataflow/cdc/libs/commons-lang3-3.9.jar:/opt/dataflow/cdc/libs/google-cloud-core-1.82.0.jar:/opt/dataflow/cdc/libs/jackson-module-jaxb-annotations-2.9.8.jar:/opt/dataflow/cdc/libs/httpcore-4.4.12.jar:/opt/dataflow/cdc/libs/validation-api-1.1.0.Final.jar:/opt/dataflow/cdc/libs/debezium-core-0.9.5.Final.jar:/opt/dataflow/cdc/libs/kafka-clients-2.2.0.jar:/opt/dataflow/cdc/libs/guava-25.1-jre.jar:/opt/dataflow/cdc/libs/proto-google-iam-v1-0.13.0.jar:/opt/dataflow/cdc/libs/beam-sdks-java-extensions-join-library-2.25.0.jar:/opt/dataflow/cdc/libs/javax.servlet-api-3.1.0.jar:/opt/dataflow/cdc/libs/jetty-io-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/javassist-3.21.0-GA.jar:/opt/dataflow/cdc/libs/perfmark-api-0.19.0.jar:/opt/dataflow/cdc/libs/api-common-1.9.0.jar:/opt/dataflow/cdc/libs/beam-vendor-bytebuddy-1_10_8-0.1.jar:/opt/dataflow/cdc/libs/hk2-locator-2.5.0-b42.jar:/opt/dataflow/cdc/libs/jersey-client-2.27.jar:/opt/dataflow/cdc/libs/jackson-jaxrs-json-provider-2.9.8.jar:/opt/dataflow/cdc/libs/mysql-connector-java-8.0.13.jar, offset.flush.timeout.ms=60000, java.vm.specification.version=1.8, sun.arch.data.model=64, sun.java.command=com.google.cloud.dataflow.cdc.connector.App, java.home=/usr/local/openjdk-8/jre, user.language=en, java.specification.vendor=Oracle Corporation, awt.toolkit=sun.awt.X11.XToolkit, java.vm.info=mixed mode, database.hostname=35.207.195.53, java.version=1.8.0_282, java.ext.dirs=/usr/local/openjdk-8/jre/lib/ext:/usr/java/packages/lib/ext, sun.boot.class.path=/usr/local/openjdk-8/jre/lib/resources.jar:/usr/local/openjdk-8/jre/lib/rt.jar:/usr/local/openjdk-8/jre/lib/sunrsasign.jar:/usr/local/openjdk-8/jre/lib/jsse.jar:/usr/local/openjdk-8/jre/lib/jce.jar:/usr/local/openjdk-8/jre/lib/charsets.jar:/usr/local/openjdk-8/jre/lib/jfr.jar:/usr/local/openjdk-8/jre/classes, java.vendor=Oracle Corporation, file.separator=/, database.server.name=dtdc, java.vendor.url.bug=http://bugreport.sun.com/bugreport/, offset.storage.file.filename=/opt/dataflow-cdc/offset/offset-tracker, database.user=dtdc, sun.io.unicode.encoding=UnicodeLittle, sun.cpu.endian=little, database.whitelist=dtdc, decimal.handling.mode=string, connector.class=io.debezium.connector.mysql.MySqlConnector, database.history=io.debezium.relational.history.FileDatabaseHistory, sun.cpu.isalist=}

Regards,
Nagarjuna


--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/0d801d45-d179-4179-a121-078f26ebfb47n%40googlegroups.com.

This communication is for informational purposes only.  Any comments or statements made herein do not necessarily reflect those of Blue Pi Consulting India Private Limited, its subsidiaries and affiliates. This transmission may contain information that is privileged, confidential, legally privileged, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Blue Pi Consulting India Private Limited, its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you.

jiri.p...@gmail.com

unread,
Apr 20, 2021, 1:53:54 AMApr 20
to debezium
Yes,

but that does not mean they are really applied. Because for example heap opts cannot be passed as connector parameter (as JVM is already running) but as JVM parameter.

J.

NAGARJUNA D.N

unread,
Apr 20, 2021, 1:55:13 AMApr 20
to debe...@googlegroups.com
Would you please suggest how do we do it?

Regards,
Nagarjuna


jiri.p...@gmail.com

unread,
Apr 20, 2021, 3:19:41 AMApr 20
to debezium
Try setting KAFKA_HEAP_OPTS environment variable of tje COnnect cluster. Unfortunately I am not versed in GKE so I can give you more precise advice in how/where this should be done.

J.

NAGARJUNA D.N

unread,
Apr 20, 2021, 3:34:41 AMApr 20
to debe...@googlegroups.com
I tried the KAFKA_HEAP_OPTS setting also but no effect.

It's nothing to with GKE I guess, am referring to debezium source code https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/master/v2/cdc-parent#deploying-the-connector

Regards,
Nagarjuna


Gunnar Morling

unread,
Apr 20, 2021, 3:46:21 AMApr 20
to debezium
How you need to specify JVM options depends on the specific environment you're running in. As you seem to run GCP Dataflow, please consult the documentation / support channels over in its community for more details.

Best,

--Gunnar
Reply all
Reply to author
Forward
0 new messages