Help !! | Debezium not capturing deleted document from Mongo

[Akshay Saini]

Hi Community,

I have a use-case where we are using Debezium to audit any change happening in MongoDB. Unlike MySQL-debezium connector Mongo-debezium don't share "before" event in payload.
Following are the details of my setup:

• Mongo Version : 4.2.19
• 3 Different node are running on docker, out of which 1 is primary and rest 2 are replicas
• Debezium version: debezium-connector-mongodb-1.8.1.Final
  
• Debezium is capturing data in default mode which is i guess, Change-stream mode

Whenever I create some document or update existing document, debezium sends data with op: 'c' and op: 'u' respectively and they do have stringify json in "after" key. But in case of deletion of document, we get op: 'd' and "after" key contains null.

Since we are not getting any information of deleted document, how can we identify which document has been deleted in Mongo ?

Best Regards
Akshay

[Chris Cranford]

Hi Akshay -

What you described is the exact behavior that we describe in the documentation:
https://debezium.io/documentation/reference/2.0/connectors/mongodb.html#mongodb-delete-events

In delete events, there is never an "after" section; because after the delete there is no document/row.  In the above link, you'll notice that when using oplog only mode, there is an attribute in the value called "filter" that describes the information about the attributes used to remove the document from MongoDB.  If you want to identify which document was deleted, you'll need to refer to the Kafka event's key for that information rather than the Kafka event's value (payload).

Thanks,
CC

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/ceb4103d-803f-4e60-b7b6-b0173b2cbf62n%40googlegroups.com.

[Akshay Saini]

Hi Chrish,

Thanks a lot for the response, I was stuck on this problem statement for a while. I did went through the document. In documentation, debezium team and you have also mentioned that deletion event can only be capture in oplog mode.

- So thats mean I cannot capture deletion event in change-stream-mode ?
- "If you want to identify which document was deleted, you'll need to refer to the Kafka event's key", can you please give any reference link for this?

Also, I am getting null value in filter key also , below is my sample json of debezium output
----------------------------------------------------------------------------------------------------------------------------------------------------------------

{
  "schema": {
    "type": "struct",
    "fields": [
      {
        "type": "string",
        "optional": true,
        "name": "io.debezium.data.Json",
        "version": 1,
        "field": "after"
      },
      {
        "type": "string",
        "optional": true,
        "name": "io.debezium.data.Json",
        "version": 1,
        "field": "patch"
      },
      {
        "type": "string",
        "optional": true,
        "name": "io.debezium.data.Json",
        "version": 1,
        "field": "filter"
      },
      {
        "type": "struct",
        "fields": [
          {
            "type": "array",
            "items": {
              "type": "string",
              "optional": false
            },
            "optional": true,
            "field": "removedFields"
          },
          {
            "type": "string",
            "optional": true,
            "name": "io.debezium.data.Json",
            "version": 1,
            "field": "updatedFields"
          },
          {
            "type": "array",
            "items": {
              "type": "struct",
              "fields": [
                {
                  "type": "string",
                  "optional": false,
                  "field": "field"
                },
                {
                  "type": "int32",
                  "optional": false,
                  "field": "size"
                }
              ],
              "optional": false,
              "name": "io.debezium.connector.mongodb.changestream.truncatedarray"
            },
            "optional": true,
            "field": "truncatedArrays"
          }
        ],
        "optional": true,
        "name": "io.debezium.connector.mongodb.changestream.updatedescription",
        "field": "updateDescription"
      },
      {
        "type": "struct",
        "fields": [
          {
            "type": "string",
            "optional": false,
            "field": "version"
          },
          {
            "type": "string",
            "optional": false,
            "field": "connector"
          },
          {
            "type": "string",
            "optional": false,
            "field": "name"
          },
          {
            "type": "int64",
            "optional": false,
            "field": "ts_ms"
          },
          {
            "type": "string",
            "optional": true,
            "name": "io.debezium.data.Enum",
            "version": 1,
            "parameters": {
              "allowed": "true,last,false,incremental"
            },
            "default": "false",
            "field": "snapshot"
          },
          {
            "type": "string",
            "optional": false,
            "field": "db"
          },
          {
            "type": "string",
            "optional": true,
            "field": "sequence"
          },
          {
            "type": "string",
            "optional": false,
            "field": "rs"
          },
          {
            "type": "string",
            "optional": false,
            "field": "collection"
          },
          {
            "type": "int32",
            "optional": false,
            "field": "ord"
          },
          {
            "type": "int64",
            "optional": true,
            "field": "h"
          },
          {
            "type": "int64",
            "optional": true,
            "field": "tord"
          },
          {
            "type": "string",
            "optional": true,
            "field": "stxnid"
          },
          {
            "type": "string",
            "optional": true,
            "field": "lsid"
          },
          {
            "type": "int64",
            "optional": true,
            "field": "txnNumber"
          }
        ],
        "optional": false,
        "name": "io.debezium.connector.mongo.Source",
        "field": "source"
      },
      {
        "type": "string",
        "optional": true,
        "field": "op"
      },
      {
        "type": "int64",
        "optional": true,
        "field": "ts_ms"
      },
      {
        "type": "struct",
        "fields": [
          {
            "type": "string",
            "optional": false,
            "field": "id"
          },
          {
            "type": "int64",
            "optional": false,
            "field": "total_order"
          },
          {
            "type": "int64",
            "optional": false,
            "field": "data_collection_order"
          }
        ],
        "optional": true,
        "field": "transaction"
      }
    ],
    "optional": false,
    "name": "Mongo.test.business.Envelope"
  },
  "payload": {
    "after": null,
    "patch": null,
    "filter": null,
    "updateDescription": null,
    "source": {
      "version": "1.8.1.Final",
      "connector": "mongodb",
      "name": "Mongo",
      "ts_ms": 1651209022000,
      "snapshot": "false",
      "db": "test",
      "sequence": null,
      "rs": "docker-rs",
      "collection": "business",
      "ord": 1,
      "h": null,
      "tord": null,
      "stxnid": null,
      "lsid": null,
      "txnNumber": null
    },
    "op": "d",
    "ts_ms": 1651209022792,
    "transaction": null
  }
}
----------------------------------------------------------------------------------------------------------------------------------------------------------------

Again, thanks a lot for the response, looking forward to hear from you :) 

Regards

Akshay 

[Chris Cranford]

Hi Akshay -

Every event sent to Kafka has a key (with a key's schema) and a value (with a value's schema).  What you've shown below is the value side of the event.  Please refer to the event's key for the primary key information I'm referring to. 
See https://kafka.apache.org/31/javadoc/org/apache/kafka/connect/connector/ConnectRecord.html#key()

Hope that helps,
CC

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/5b897a8f-2813-45a7-a16c-46ce4aa61fc8n%40googlegroups.com.

[Akshay Saini]

Thanks a lot Chris for helping me out. You are the best. Now I am able to get the deletion ID :D

Below is the answer is anyone is looking for the solution:
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Debezium is pushing event with both key and value. When people consume data from Kafka, usually they consumer value part and below is the command which user would have run, to view data from consumer:

kafka-console-consumer --bootstrap-server localhost:9092 --topic <topic name>

The change which is need to be made to view the key part :

kafka-console-consumer --bootstrap-server localhost:9092 --topic <topic name> --property print.key=true

 This will fetch the key part of event and you will able to view the mongo's object ID based on which creation, updation and deletion is happening in Mongo 

----------------------------------------------------------------------------------------------------------------------------------------------------------------

Best Regards

Akshay