debezium mysql connector is failing with java heap space issue

[NAGARJUNA D.N]

Hi,

we have debezium mysql connectors deployed on GKE cluster. But the pod is failing with Caused by: java.lang.OutOfMemoryError: Java heap space . Increased jvm options to 10g, but still its failing with the same issue.

Below are my config properties:

singleTopicMode=true 

databaseManagementSystem=mysql 

debezium.offset.flush.timeout.ms=60000 

debezium.offset.flush.interval.ms=10000 

debezium.max.request.size=1048 

debezium.max.queue.size=65536 

debezium.max.batch.size=16384 

debezium.HEAP_OPTS='-Xms10g -Xmx10g'

Complete log:

org.apache.kafka.connect.errors.ConnectException: java.lang.OutOfMemoryError: Java heap space at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:275) at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:49) at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:791) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596) at java.lang.StringBuilder.append(StringBuilder.java:190) at java.io.BufferedReader.readLine(BufferedReader.java:358) at java.io.BufferedReader.readLine(BufferedReader.java:389) at java.nio.file.Files.readAllLines(Files.java:3205) at io.debezium.relational.history.FileDatabaseHistory.lambda$recoverRecords$2(FileDatabaseHistory.java:136) at io.debezium.relational.history.FileDatabaseHistory$$Lambda$181/230925968.run(Unknown Source) at io.debezium.util.FunctionalReadWriteLock.write(FunctionalReadWriteLock.java:96) at io.debezium.relational.history.FileDatabaseHistory.recoverRecords(FileDatabaseHistory.java:133) at io.debezium.relational.history.AbstractDatabaseHistory.recover(AbstractDatabaseHistory.java:73) at io.debezium.connector.mysql.MySqlSchema.loadHistory(MySqlSchema.java:252) at io.debezium.connector.mysql.MySqlTaskContext.loadHistory(MySqlTaskContext.java:165) at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:105) ... 7 more

[jiri.p...@gmail.com]

Hi,

could you please verify if the heap opts were really applied to the JVM?

J.

[NAGARJUNA D.N]

Yes it's.

the log:

, offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore, java.vm.specification.vendor=Oracle Corporation, max.request.size=104857, os.name=Linux, sun.jnu.encoding=UTF-8, HEAP_OPTS='-Xms10g -Xmx10g', java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib, max.queue.size=65536, java.class.version=52.0, java.specification.name=Java Platform API Specification, max.batch.size=16384, sun.management.compiler=HotSpot 64-Bit Tiered Compilers, , os.version=5.4.89+, user.home=/root, user.timezone=Etc/UTC, java.awt.printerjob=sun.print.PSPrinterJob, file.encoding=UTF-8, java.specification.version=1.8, user.name=root, java.class.path=/opt/dataflow/cdc/resources:/opt/dataflow/cdc/classes:/opt/dataflow/cdc/libs/conscrypt-openjdk-uber-1.3.0.jar:/opt/dataflow/cdc/libs/error_prone_annotations-2.1.3.jar:/opt/dataflow/cdc/libs/fastjson-1.2.68.jar:/opt/dataflow/cdc/libs/grpc-protobuf-lite-1.25.0.jar:/opt/dataflow/cdc/libs/grpc-alts-1.28.1.jar:/opt/dataflow/cdc/libs/grpc-google-cloud-pubsub-v1-1.64.0.jar:/opt/dataflow/cdc/libs/grpc-stub-1.25.0.jar:/opt/dataflow/cdc/libs/google-cloud-core-grpc-1.82.0.jar:/opt/dataflow/cdc/libs/commons-csv-1.8.jar:/opt/dataflow/cdc/libs/connect-file-2.2.0.jar:/opt/dataflow/cdc/libs/jetty-continuation-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/proto-google-cloud-datacatalog-v1-0.35.0.jar:/opt/dataflow/cdc/libs/connect-runtime-2.2.0.jar:/opt/dataflow/cdc/libs/grpc-netty-shaded-1.25.0.jar:/opt/dataflow/cdc/libs/proto-google-cloud-pubsub-v1-1.64.0.jar:/opt/dataflow/cdc/libs/jersey-server-2.27.jar:/opt/dataflow/cdc/libs/j2objc-annotations-1.1.jar:/opt/dataflow/cdc/libs/beam-vendor-calcite-1_20_0-0.1.jar:/opt/dataflow/cdc/libs/gax-grpc-1.56.0.jar:/opt/dataflow/cdc/libs/slf4j-api-1.7.25.jar:/opt/dataflow/cdc/libs/beam-sdks-java-core-2.25.0.jar:/opt/dataflow/cdc/libs/mongo-java-driver-3.9.1.jar:/opt/dataflow/cdc/libs/gax-1.53.1.jar:/opt/dataflow/cdc/libs/jetty-client-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/grpc-protobuf-1.25.0.jar:/opt/dataflow/cdc/libs/jackson-jaxrs-base-2.9.8.jar:/opt/dataflow/cdc/libs/jetty-http-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/google-cloud-datacatalog-0.35.0.jar:/opt/dataflow/cdc/libs/annotations-4.1.1.4.jar:/opt/dataflow/cdc/libs/javax.ws.rs-api-2.1.1.jar:/opt/dataflow/cdc/libs/janino-3.0.11.jar:/opt/dataflow/cdc/libs/grpc-api-1.25.0.jar:/opt/dataflow/cdc/libs/connect-api-2.2.0.jar:/opt/dataflow/cdc/libs/jackson-annotations-2.10.2.jar:/opt/dataflow/cdc/libs/snappy-java-1.1.4.jar:/opt/dataflow/cdc/libs/grpc-grpclb-1.28.1.jar:/opt/dataflow/cdc/libs/connect-transforms-2.2.0.jar:/opt/dataflow/cdc/libs/opencensus-api-0.24.0.jar:/opt/dataflow/cdc/libs/kafka-tools-2.2.0.jar:/opt/dataflow/cdc/libs/proto-google-common-protos-1.17.0.jar:/opt/dataflow/cdc/libs/beam-runners-direct-java-2.25.0.jar:/opt/dataflow/cdc/libs/jackson-datatype-jdk8-2.9.8.jar:/opt/dataflow/cdc/libs/javax.inject-1.jar:/opt/dataflow/cdc/libs/log4j-1.2.17.jar:/opt/dataflow/cdc/libs/beam-sdks-java-extensions-sql-2.25.0.jar:/opt/dataflow/cdc/libs/plexus-utils-3.1.0.jar:/opt/dataflow/cdc/libs/google-http-client-1.34.0.jar:/opt/dataflow/cdc/libs/protobuf-java-3.10.0.jar:/opt/dataflow/cdc/libs/commons-collections-3.2.2.jar:/opt/dataflow/cdc/libs/jackson-core-2.10.2.jar:/opt/dataflow/cdc/libs/grpc-google-cloud-datacatalog-v1beta1-0.35.0.jar:/opt/dataflow/cdc/libs/commons-logging-1.2.jar:/opt/dataflow/cdc/libs/beam-model-job-management-2.25.0.jar:/opt/dataflow/cdc/libs/gson-2.7.jar:/opt/dataflow/cdc/libs/kafka-log4j-appender-2.2.0.jar:/opt/dataflow/cdc/libs/args4j-2.33.jar:/opt/dataflow/cdc/libs/proto-google-cloud-datacatalog-v1beta1-0.35.0.jar:/opt/dataflow/cdc/libs/google-auth-library-credentials-0.16.2.jar:/opt/dataflow/cdc/libs/commons-text-1.6.jar:/opt/dataflow/cdc/libs/connect-json-2.2.0.jar:/opt/dataflow/cdc/libs/lz4-java-1.5.0.jar:/opt/dataflow/cdc/libs/commons-compiler-3.0.11.jar:/opt/dataflow/cdc/libs/debezium-connector-mysql-0.9.5.Final.jar:/opt/dataflow/cdc/libs/osgi-resource-locator-1.0.1.jar:/opt/dataflow/cdc/libs/jsr305-3.0.2.jar:/opt/dataflow/cdc/libs/paranamer-2.7.jar:/opt/dataflow/cdc/libs/grpc-auth-1.28.1.jar:/opt/dataflow/cdc/libs/argparse4j-0.7.0.jar:/opt/dataflow/cdc/libs/antlr4-runtime-4.7.jar:/opt/dataflow/cdc/libs/grpc-core-1.25.0.jar:/opt/dataflow/cdc/libs/opencensus-contrib-grpc-metrics-0.21.0.jar:/opt/dataflow/cdc/libs/jetty-security-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/commons-codec-1.14.jar:/opt/dataflow/cdc/libs/javax.inject-2.5.0-b42.jar:/opt/dataflow/cdc/libs/jersey-media-jaxb-2.27.jar:/opt/dataflow/cdc/libs/slf4j-log4j12-1.7.25.jar:/opt/dataflow/cdc/libs/debezium-embedded-0.9.5.Final.jar:/opt/dataflow/cdc/libs/httpclient-4.5.10.jar:/opt/dataflow/cdc/libs/checker-qual-2.0.0.jar:/opt/dataflow/cdc/libs/beam-vendor-guava-26_0-jre-0.1.jar:/opt/dataflow/cdc/libs/hk2-utils-2.5.0-b42.jar:/opt/dataflow/cdc/libs/protobuf-java-util-3.7.1.jar:/opt/dataflow/cdc/libs/xz-1.8.jar:/opt/dataflow/cdc/libs/beam-vendor-grpc-1_26_0-0.3.jar:/opt/dataflow/cdc/libs/animal-sniffer-annotations-1.14.jar:/opt/dataflow/cdc/libs/google-http-client-jackson2-1.34.0.jar:/opt/dataflow/cdc/libs/maven-artifact-3.6.0.jar:/opt/dataflow/cdc/libs/auto-value-annotations-1.7.jar:/opt/dataflow/cdc/libs/joda-time-2.10.5.jar:/opt/dataflow/cdc/libs/reflections-0.9.11.jar:/opt/dataflow/cdc/libs/commons-beanutils-1.9.3.jar:/opt/dataflow/cdc/libs/jersey-hk2-2.27.jar:/opt/dataflow/cdc/libs/mysql-binlog-connector-java-0.19.1.jar:/opt/dataflow/cdc/libs/debezium-ddl-parser-0.9.5.Final.jar:/opt/dataflow/cdc/libs/grpc-context-1.25.0.jar:/opt/dataflow/cdc/libs/jackson-databind-2.10.2.jar:/opt/dataflow/cdc/libs/threetenbp-1.4.3.jar:/opt/dataflow/cdc/libs/activation-1.1.1.jar:/opt/dataflow/cdc/libs/cdc-common-0.1.jar:/opt/dataflow/cdc/libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/dataflow/cdc/libs/avro-1.8.2.jar:/opt/dataflow/cdc/libs/jetty-server-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/jaxb-api-2.3.0.jar:/opt/dataflow/cdc/libs/zstd-jni-1.3.8-1.jar:/opt/dataflow/cdc/libs/beam-model-pipeline-2.25.0.jar:/opt/dataflow/cdc/libs/commons-configuration2-2.5.jar:/opt/dataflow/cdc/libs/jetty-util-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/beam-sdks-java-io-mongodb-2.25.0.jar:/opt/dataflow/cdc/libs/google-cloud-pubsub-1.82.0.jar:/opt/dataflow/cdc/libs/commons-compress-1.8.1.jar:/opt/dataflow/cdc/libs/hk2-api-2.5.0-b42.jar:/opt/dataflow/cdc/libs/jersey-container-servlet-core-2.27.jar:/opt/dataflow/cdc/libs/jetty-servlet-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/jersey-container-servlet-2.27.jar:/opt/dataflow/cdc/libs/google-auth-library-oauth2-http-0.20.0.jar:/opt/dataflow/cdc/libs/log4j-api-2.6.2.jar:/opt/dataflow/cdc/libs/jackson-core-asl-1.9.13.jar:/opt/dataflow/cdc/libs/jersey-common-2.27.jar:/opt/dataflow/cdc/libs/jackson-mapper-asl-1.9.13.jar:/opt/dataflow/cdc/libs/javax.annotation-api-1.3.2.jar:/opt/dataflow/cdc/libs/jetty-servlets-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/opencensus-contrib-http-util-0.24.0.jar:/opt/dataflow/cdc/libs/commons-lang3-3.9.jar:/opt/dataflow/cdc/libs/google-cloud-core-1.82.0.jar:/opt/dataflow/cdc/libs/jackson-module-jaxb-annotations-2.9.8.jar:/opt/dataflow/cdc/libs/httpcore-4.4.12.jar:/opt/dataflow/cdc/libs/validation-api-1.1.0.Final.jar:/opt/dataflow/cdc/libs/debezium-core-0.9.5.Final.jar:/opt/dataflow/cdc/libs/kafka-clients-2.2.0.jar:/opt/dataflow/cdc/libs/guava-25.1-jre.jar:/opt/dataflow/cdc/libs/proto-google-iam-v1-0.13.0.jar:/opt/dataflow/cdc/libs/beam-sdks-java-extensions-join-library-2.25.0.jar:/opt/dataflow/cdc/libs/javax.servlet-api-3.1.0.jar:/opt/dataflow/cdc/libs/jetty-io-9.4.14.v20181114.jar:/opt/dataflow/cdc/libs/javassist-3.21.0-GA.jar:/opt/dataflow/cdc/libs/perfmark-api-0.19.0.jar:/opt/dataflow/cdc/libs/api-common-1.9.0.jar:/opt/dataflow/cdc/libs/beam-vendor-bytebuddy-1_10_8-0.1.jar:/opt/dataflow/cdc/libs/hk2-locator-2.5.0-b42.jar:/opt/dataflow/cdc/libs/jersey-client-2.27.jar:/opt/dataflow/cdc/libs/jackson-jaxrs-json-provider-2.9.8.jar:/opt/dataflow/cdc/libs/mysql-connector-java-8.0.13.jar, offset.flush.timeout.ms=60000, java.vm.specification.version=1.8, sun.arch.data.model=64, sun.java.command=com.google.cloud.dataflow.cdc.connector.App, java.home=/usr/local/openjdk-8/jre, user.language=en, java.specification.vendor=Oracle Corporation, awt.toolkit=sun.awt.X11.XToolkit, java.vm.info=mixed mode, database.hostname=35.207.195.53, java.version=1.8.0_282, java.ext.dirs=/usr/local/openjdk-8/jre/lib/ext:/usr/java/packages/lib/ext, sun.boot.class.path=/usr/local/openjdk-8/jre/lib/resources.jar:/usr/local/openjdk-8/jre/lib/rt.jar:/usr/local/openjdk-8/jre/lib/sunrsasign.jar:/usr/local/openjdk-8/jre/lib/jsse.jar:/usr/local/openjdk-8/jre/lib/jce.jar:/usr/local/openjdk-8/jre/lib/charsets.jar:/usr/local/openjdk-8/jre/lib/jfr.jar:/usr/local/openjdk-8/jre/classes, java.vendor=Oracle Corporation, file.separator=/, database.server.name=dtdc, java.vendor.url.bug=http://bugreport.sun.com/bugreport/, offset.storage.file.filename=/opt/dataflow-cdc/offset/offset-tracker, database.user=dtdc, sun.io.unicode.encoding=UnicodeLittle, sun.cpu.endian=little, database.whitelist=dtdc, decimal.handling.mode=string, connector.class=io.debezium.connector.mysql.MySqlConnector, database.history=io.debezium.relational.history.FileDatabaseHistory, sun.cpu.isalist=}

Regards,

Nagarjuna

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/0d801d45-d179-4179-a121-078f26ebfb47n%40googlegroups.com.

This communication is for informational purposes only.  Any comments or statements made herein do not necessarily reflect those of Blue Pi Consulting India Private Limited, its subsidiaries and affiliates. This transmission may contain information that is privileged, confidential, legally privileged, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Blue Pi Consulting India Private Limited, its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you.

[jiri.p...@gmail.com]

Yes,

but that does not mean they are really applied. Because for example heap opts cannot be passed as connector parameter (as JVM is already running) but as JVM parameter.

J.

[NAGARJUNA D.N]

Would you please suggest how do we do it?

Regards,

Nagarjuna

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/291c2d99-dc31-4583-93db-01178ed5bfc6n%40googlegroups.com.

[jiri.p...@gmail.com]

Try setting KAFKA_HEAP_OPTS environment variable of tje COnnect cluster. Unfortunately I am not versed in GKE so I can give you more precise advice in how/where this should be done.

J.

[NAGARJUNA D.N]

I tried the KAFKA_HEAP_OPTS setting also but no effect.

It's nothing to with GKE I guess, am referring to debezium source code https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/master/v2/cdc-parent#deploying-the-connector

Regards,

Nagarjuna

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/4d57e2e2-73e2-4f1b-97ef-335cbd603625n%40googlegroups.com.

[Gunnar Morling]

How you need to specify JVM options depends on the specific environment you're running in. As you seem to run GCP Dataflow, please consult the documentation / support channels over in its community for more details.

Best,

--Gunnar