V$ARCHIVE_DEST_STATUS (SELECT * FROM V$ARCHIVE_DEST_STATUS WHERE STATUS='VALID' AND TYPE='LOCAL' AND ROWNUM=1)
2. Archived Redo Logs files must not be removed or moved to another directory. Debezium may need these files in a re-snapshot case, and if they don't exist, SCN data will be lost.
3. If it is necessary to move the archived Redo Logs to another directory, due to lack of disk space or other reasons, we recommend setting a time of at least 10 to 15 days for these files to remain to ensure that everything is being captured by Debezium. This removal is not advised because in an incident (such as loss of capture or database outage) we may need older Redo Logs due to the SCN contained in them.
4. We should not keep any Redo Logs with SEQ=0, this causes Redo Logs conflict issues in Debezium.
5. A minimum Redo Log time of at least 24 hours is required. Depending on the size of the table and the seasonality at which the data is updated, this time is necessary to ensure that all data is captured. You can reduce this period as you learn how your data behaves. We are in an age of live data (real time remembers?) so learn from your data.
6. You need to deep dive and understand which tables in your ecosystem go more than a day without receiving any records. Oracle understands that the SCN becomes "old" or outdated and the backup process removes the Redo log, as it has reached a condition where its "age" indicates that it needs to be moved or deleted and this causes a problem for the capture stopping the Debezium connector from working.
7. We use a strategy with Heartbeat to fix this improper "removal" from the SCN. With this strategy, Heartbeat sends data every 10 seconds, updating the SCN automatically so it will never go out of date and avoid undue removal.
8. By default, the Debezium Connector on Oracle LogMiner only considers the existing Redo Logs in the +DGRECO directory, any process that moves the Redo Log to another directory, Debezium ignores. A lot of confusion occurs in this context as it leads to believe that, like magic, Debezium automatically identifies where the files are. What is not true.
9. Another important information is that /orabackup/logminer is not considered by the connector, if the Redo Log is moved to this directory as a backup strategy, it will not be recognized by the connector
What I learned from this is that you are going to configure this data capture with Debezium 100 times in a LogMiner environment and each of those 100 times will be different. Be patient and fight the red demon just like Gandalf did with Balrog
Regards,
Marcelo Costa
https://www.linkedin.com/in/marcelojscosta/
the best and easiest way to perform a new snapshot when you hit this scenario is either to:
OR
Using a new name in the latter approach guarantees that a new offsets key will be used since they're based on the connector's names, so there is no need to remove the prior offsets from the broker when using this approach.
This is because the SCN has been deleted or moved. I faced this same problem on a recent installation of Debezium with Oracle Logminer. (Debezium version 1.93 and Oracle 11g)
You need to talk to a DBA and understand why the SCN is being deleted or moved.
Here, in our case, the backup process erased the redologs. So we changed the backup strategy.
I have some tips for configuring Debezium with LogMiner (use at your own risk):
Here are my 9 tips to Configure Debezium with Oracle LogMiner
1. Archived Redo Logs files must be in a single directory only, as Debeziun always collects data from a single ARCHIVE directory in the table:V$ARCHIVE_DEST_STATUS (SELECT * FROM V$ARCHIVE_DEST_STATUS WHERE STATUS='VALID' AND TYPE='LOCAL' AND ROWNUM=1)
2. Archived Redo Logs files must not be removed or moved to another directory. Debezium may need these files in a re-snapshot case, and if they don't exist, SCN data will be lost.
3. If it is necessary to move the archived Redo Logs to another directory, due to lack of disk space or other reasons, we recommend setting a time of at least 10 to 15 days for these files to remain to ensure that everything is being captured by Debezium. This removal is not advised because in an incident (such as loss of capture or database outage) we may need older Redo Logs due to the SCN contained in them.
4. We should not keep any Redo Logs with SEQ=0, this causes Redo Logs conflict issues in Debezium.
5. A minimum Redo Log time of at least 24 hours is required. Depending on the size of the table and the seasonality at which the data is updated, this time is necessary to ensure that all data is captured. You can reduce this period as you learn how your data behaves. We are in an age of live data (real time remembers?) so learn from your data.
6. You need to deep dive and understand which tables in your ecosystem go more than a day without receiving any records. Oracle understands that the SCN becomes "old" or outdated and the backup process removes the Redo log, as it has reached a condition where its "age" indicates that it needs to be moved or deleted and this causes a problem for the capture stopping the Debezium connector from working.
7. We use a strategy with Heartbeat to fix this improper "removal" from the SCN. With this strategy, Heartbeat sends data every 10 seconds, updating the SCN automatically so it will never go out of date and avoid undue removal.
8. By default, the Debezium Connector on Oracle LogMiner only considers the existing Redo Logs in the +DGRECO directory, any process that moves the Redo Log to another directory, Debezium ignores. A lot of confusion occurs in this context as it leads to believe that, like magic, Debezium automatically identifies where the files are. What is not true.
9. Another important information is that /orabackup/logminer is not considered by the connector, if the Redo Log is moved to this directory as a backup strategy, it will not be recognized by the connector
--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/4c329b18-8e94-4caa-9311-3dea76986fe7n%40googlegroups.com.