MySQL extractor got an error 'Unable to retrieve column metadata'

99 views
Skip to first unread message

Yuto Umeda

unread,
Oct 7, 2021, 11:39:15 PM10/7/21
to Tungsten Replicator Discuss
Hi,
Thank you for your answer in previous my question.
Sorry I have another question about replicator.

MySQL extractor got an error 'Unable to retrieve column metadata'.
Extractor works just after execute 'trepctl online', but this error occurs about half a day later.
Could you tell me the cause of this error?

logs are the below.(Full logs are attached)

2021/10/08 01:49:44.690 | mysvc-tungsten-extraction-dev | [mysvc - Thread-262] INFO commands.FileDeleteTask File deleted: /opt/continuent/relay/mysvc/mysql-bin-changelog.189126
2021/10/08 01:54:44.649 | mysvc-tungsten-extraction-dev | [mysvc - Relay Client - mysvc] INFO extractor.mysql.RelayLogClient Closing relay log: name=/opt/continuent/relay/mysvc/mysql-bin-changelog.189135 bytes=568
2021/10/08 01:54:44.649 | mysvc-tungsten-extraction-dev | [mysvc - Relay Client - mysvc] INFO extractor.mysql.RelayLogClient Rotating to new relay log: name=/opt/continuent/relay/mysvc/mysql-bin-changelog.189136
2021/10/08 01:54:44.649 | mysvc-tungsten-extraction-dev | [mysvc - Relay Client - mysvc] INFO extractor.mysql.RelayLogClient Opening relay log: name=/opt/continuent/relay/mysvc/mysql-bin-changelog.189136
2021/10/08 01:54:44.649 | mysvc-tungsten-extraction-dev | [mysvc - Relay Client - mysvc] INFO extractor.mysql.RelayLogClient Adding relay log to binlog index: /opt/continuent/relay/mysvc/mysql-bin-changelog.index
2021/10/08 01:54:44.749 | mysvc-tungsten-extraction-dev | [mysvc - binlog-to-q-0] INFO extractor.mysql.MySQLExtractor Checking for old relay log files...
2021/10/08 01:54:44.749 | mysvc-tungsten-extraction-dev | [mysvc - Thread-263] INFO commands.FileDeleteTask File deleted: /opt/continuent/relay/mysvc/mysql-bin-changelog.189127
2021/10/08 01:59:45.320 | mysvc-tungsten-extraction-dev | [mysvc - Relay Client - mysvc] INFO extractor.mysql.RelayLogClient Closing relay log: name=/opt/continuent/relay/mysvc/mysql-bin-changelog.189136 bytes=568
2021/10/08 01:59:45.320 | mysvc-tungsten-extraction-dev | [mysvc - Relay Client - mysvc] INFO extractor.mysql.RelayLogClient Rotating to new relay log: name=/opt/continuent/relay/mysvc/mysql-bin-changelog.189137
2021/10/08 01:59:45.320 | mysvc-tungsten-extraction-dev | [mysvc - Relay Client - mysvc] INFO extractor.mysql.RelayLogClient Opening relay log: name=/opt/continuent/relay/mysvc/mysql-bin-changelog.189137
2021/10/08 01:59:45.320 | mysvc-tungsten-extraction-dev | [mysvc - Relay Client - mysvc] INFO extractor.mysql.RelayLogClient Adding relay log to binlog index: /opt/continuent/relay/mysvc/mysql-bin-changelog.index
2021/10/08 01:59:45.320 | mysvc-tungsten-extraction-dev | [mysvc - binlog-to-q-0] INFO extractor.mysql.MySQLExtractor Checking for old relay log files...
2021/10/08 01:59:45.320 | mysvc-tungsten-extraction-dev | [mysvc - Thread-264] INFO commands.FileDeleteTask File deleted: /opt/continuent/relay/mysvc/mysql-bin-changelog.189128
2021/10/08 02:01:46.178 | mysvc-tungsten-extraction-dev | [mysvc - binlog-to-q-0] ERROR pipeline.SingleThreadStageTask Stage task failed: stage=binlog-to-q seqno=310 fragno=0
2021/10/08 02:01:46.178 | com.continuent.tungsten.replicator.ReplicatorException: Unable to retrieve column metadata: schema=mydb table=mytable
2021/10/08 02:01:46.178 | at com.continuent.tungsten.replicator.filter.ColumnNameFilter.getColumnInformation(ColumnNameFilter.java:266)
2021/10/08 02:01:46.178 | at com.continuent.tungsten.replicator.filter.ColumnNameFilter.filter(ColumnNameFilter.java:153)
2021/10/08 02:01:46.178 | at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.runTask(SingleThreadStageTask.java:459)
2021/10/08 02:01:46.178 | at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.run(SingleThreadStageTask.java:183)
2021/10/08 02:01:46.178 | at java.lang.Thread.run(Thread.java:748)
2021/10/08 02:01:46.178 | mysvc-tungsten-extraction-dev | [mysvc - binlog-to-q-0] INFO pipeline.SingleThreadStageTask Performing emergency rollback of applied changes
2021/10/08 02:01:46.178 | mysvc-tungsten-extraction-dev | [mysvc - binlog-to-q-0] INFO pipeline.SingleThreadStageTask Dispatching error event: Stage task failed: stage=binlog-to-q seqno=310 fragno=0
2021/10/08 02:01:46.178 | mysvc-tungsten-extraction-dev | [mysvc - binlog-to-q-0] INFO pipeline.SingleThreadStageTask Terminating processing for stage task thread
2021/10/08 02:01:46.178 | mysvc-tungsten-extraction-dev | [mysvc - binlog-to-q-0] INFO pipeline.SingleThreadStageTask Last successfully processed event prior to termination: seqno=310 eventid=mysql-bin-changelog.189137:0000000000000728;-1
2021/10/08 02:01:46.178 | mysvc-tungsten-extraction-dev | [mysvc - binlog-to-q-0] INFO pipeline.SingleThreadStageTask Task event count: 311
2021/10/08 02:01:46.178 | mysvc-tungsten-extraction-dev | [mysvc - pool-2-thread-1] ERROR management.OpenReplicatorManager Received error notification, shutting down services :

The db and table in the error is not the target of replication.

Results of tpm reverse are the below.

$ tpm reverse
# Installed from tungsten@myservice-tungsten-extraction:/opt/continuent/software/tungsten-replicator-for-mysql-extractor-6.1.13-45
# Configuration built from /etc/tungsten/tungsten.ini
# Defaults for all data services and hosts
tools/tpm configure defaults \
--auto-recovery-max-attempts=10 \
--disable-security-controls=true \
--install-directory=/opt/continuent \
--mysql-allow-intensive-checks=true \
'--profile-script=~/.bash_profile' \
--skip-validation-check=InstallerMasterSlaveCheck \
--skip-validation-check=MySQLBinaryLogsEnabledCheck \
--skip-validation-check=MySQLMyISAMCheck \
--skip-validation-check=MySQLPermissionsCheck \
--skip-validation-check=RowBasedBinaryLoggingCheck \
--user=tungsten
# Options for the myservice data service
tools/tpm configure myservice \
--master=myservice-tungsten-extraction \
--members=myservice-tungsten-extraction
# Options for myservice-tungsten-extraction
tools/tpm configure --hosts=myservice-tungsten-extraction \
--java-mem-size=3072 \
--rmi-port=10001
# Options for myservice-tungsten-extraction
tools/tpm configure myservice \
--hosts=myservice-tungsten-extraction \
--datasource-mysql-conf=/dev/null \
--enable-heterogeneous-service=true \
--privileged-master=false \
--property=replicator.filter.pkey.addColumnsToDeletes=true \
--property=replicator.filter.pkey.addPkeyToInserts=true \
--property=replicator.filter.replicate.do=db.table \
--property=replicator.service.comments=true \
--replication-host=xxxxxx.rds.amazonaws.com \
--replication-password=XXXXXXXXX \
--replication-user=myuser \
--svc-extractor-filters=dropcatalogdata,replicate,colnames,pkey \
--thl-log-retention=7d \
--thl-port=2112

Thanks

Chris Parker

unread,
Oct 8, 2021, 5:56:17 AM10/8/21
to tungsten-repl...@googlegroups.com
Hi Yuto,

The error is stating that the ColumnNameFilter is unable to find the table anymore.  Is it possible that after an update/insert/delete the table was then dropped from the source?

Also, I not something in the tpm reverse that I would like to check a little more - could you send me the tungsten.ini file please? Mask out the user/pwd information, and if you wish, send to my direct email instead.

Thanks
Chris

Chris Parker
Customer Success Director, EMEA & APAC
Continuent Ltd., a Delaware Corporation

chris....@continuent.com



--
You received this message because you are subscribed to the Google Groups "Tungsten Replicator Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tungsten-replicator...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tungsten-replicator-discuss/9c59930b-2cc2-48aa-a258-a017c0605c2en%40googlegroups.com.

Message has been deleted

Yuto Umeda

unread,
Oct 11, 2021, 4:28:05 AM10/11/21
to Tungsten Replicator Discuss
Hi Chris
Thank you for your reply.


>  Is it possible that after an update/insert/delete the table was then dropped from the source?
It is not possible.


> could you send me the tungsten.ini file please?
The ini file is the below.
Our other environments of MySQL and TungtenReplicator works with this ini file. 

[defaults]
auto-recovery-max-attempts=10
install-directory=/opt/continuent
user=tungsten
mysql-allow-intensive-checks=true
skip-validation-check=RowBasedBinaryLoggingCheck,InstallerMasterSlaveCheck,MySQLPermissionsCheck,MySQLBinaryLogsEnabledCheck,MySQLMyISAMCheck
profile-script=~/.bash_profile
disable-security-controls=true
auto-recovery-max-interval=5s

[mysvc]
replication-host=xxxxxxx.rds.amazonaws.com
replication-user=xxxxxxx
replication-password=xxxxxxx
repl-java-mem-size=1024
master=xxxxxxx
members=xxxxxxx
svc-extractor-filters=dropcatalogdata,replicate,colnames,pkey
replication-port=3306
thl-port=2112
rmi-port=10001
thl-log-retention=7d
datasource-mysql-conf=/dev/null
enable-heterogeneous-service=true
privileged-master=false
property=replicator.filter.pkey.addColumnsToDeletes=true
property=replicator.filter.pkey.addPkeyToInserts=true
property=replicator.service.comments=true


2021年10月8日金曜日 18:56:17 UTC+9 chris....@continuent.com:

Chris Parker

unread,
Oct 11, 2021, 4:33:15 AM10/11/21
to tungsten-repl...@googlegroups.com
Thanks…

In your ini file could you alter this line:

svc-extractor-filters=dropcatalogdata,replicate,colnames,pkey

and change it to:

svc-extractor-filters=dropcatalogdata,colnames,pkey,replicate

And see if that helps? The error clearly implies that the table can no longer be found which seems really odd, but perhaps the order of filtering might be confusing the replicator.

Let me know if this fixes the issue?

Thanks

Chris Parker
Customer Success Director, EMEA & APAC
Continuent Ltd., a Delaware Corporation

chris....@continuent.com

On 11 Oct 2021, at 09:28, Yuto Umeda <yuto....@dotd-inc.com> wrote:

svc-extractor-filters=dropcatalogdata,replicate,colnames,pkey

Yuto Umeda

unread,
Oct 12, 2021, 4:05:37 AM10/12/21
to Tungsten Replicator Discuss
Hi Chris,
Thank you for your answer.
I'll try that and let you know if this fixes the issue.

Thanks

2021年10月11日月曜日 17:33:15 UTC+9 chris....@continuent.com:

Yuto Umeda

unread,
Oct 12, 2021, 9:54:18 PM10/12/21
to Tungsten Replicator Discuss
Hi,
I'm afraid the solution did not fix the issue and the same error occured.
Can it be effective to shutdown the current replicator and launch a new replicator?

Thanks 

2021年10月12日火曜日 17:05:37 UTC+9 Yuto Umeda:

Chris Parker

unread,
Oct 13, 2021, 3:45:48 AM10/13/21
to tungsten-repl...@googlegroups.com
Hi Yuto,

I don’t think a new build will make a difference.  We believe this maybe a timeout between the extractor and Aurora and a bug in the colnames filter.  Is this a different Aurora source than your other environments where there is no problem?

Can you check what the various MySQL timeouts are and see if any differ?

I see you have auto-recovery enabled on the replicator, maybe we could increase those properties? Firstly revert the change to svc-extractor-filters that we made and change the following two lines in the [defaults] section of the ini:

From:
auto-recovery-max-attempts=10
auto-recovery-delay-interval=5s

To:
auto-recovery-max-attempts=20
auto-recovery-delay-interval=10s

After making the changes to the tungsten.ini, you then need to make sure they are applied by issuing `tpm update` on the command line.

In the meantime we will investigate a better fix within the colnames filter.

Thanks
Chris
 
Chris Parker
Customer Success Director, EMEA & APAC
Continuent Ltd., a Delaware Corporation

chris....@continuent.com
+44 (0)7790 685 269


-- 
You received this message because you are subscribed to the Google Groups "Tungsten Replicator Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tungsten-replicator...@googlegroups.com.

Yuto Umeda

unread,
Oct 14, 2021, 2:58:41 AM10/14/21
to Tungsten Replicator Discuss
Hi Chris,


> Is this a different Aurora source than your other environments where there is no problem?
There is no difference in at least the definition of the table mentioned in the error log.
I'll check whether the parameter group has any difference from other environments.


> Can you check what the various MySQL timeouts are and see if any differ?
Thanks, I'll try that. 


2021年10月13日水曜日 16:45:48 UTC+9 chris....@continuent.com:

Chris Parker

unread,
Oct 14, 2021, 10:58:25 AM10/14/21
to tungsten-repl...@googlegroups.com
Hi Yuto,

Just to keep you updated - we have found what we believe maybe the issue and are currently working on a fix.

As soon as I have something for you I will let you know

Many Thanks
Chris

Chris Parker
Customer Success Director, EMEA & APAC
Continuent Ltd., a Delaware Corporation

chris....@continuent.com


Yuto Umeda

unread,
Oct 18, 2021, 1:53:38 AM10/18/21
to Tungsten Replicator Discuss
Hi Chris,


> we have found what we believe maybe the issue and are currently working on a fix.
That's great news for us. Thank you for your information.


> Can you check what the various MySQL timeouts are and see if any differ?
We changed auto-recovery config but we couldn't find any differ...

Many Thanks
2021年10月14日木曜日 23:58:25 UTC+9 chris....@continuent.com:
Reply all
Reply to author
Forward
0 new messages