Debezium Server Not Showing MongoDB Deleted ID

[Chachi Pereira]

Hi All,

I'm in the process of setting up Debezium Server. So far it works very well, however I'm struggling to capture the MongoDB document ID on delete events.

Typically, I grab the document ID in the 'after' property of the JSON payload. But on delete events the 'after' property is null/none as expected.

Now looking at the documentation, if I enable 'schema' the document should be encapsulated inside a 'payload' property which should contain an ID. Unfortunately, that doesn't seem to work. The payload still outputs as follows:

{'after': None,
    'patch': None,
    'filter': None,
    'updateDescription': None,
    'source':
    {'version': '1.8.1.Final',
                'connector': 'mongodb',
                'name': 'testdb',
                'ts_ms': 1654764135000,
                'snapshot': 'false',
                'db': 'testdb',
                'sequence': None,
                'rs': 'testdb-shard-0',
                'collection': 'contracts',
                'ord': 1,
                'h': None,
                'tord': None,
                'stxnid': None,
                'lsid': None,
                'txnNumber': None
    },
    'op': 'd',
    'ts_ms': 1654764135671,
    'transaction': None
}

Any ideas on how I can emit the document ID on delete events?

[Chachi Pereira]

Having read the documentation again, it's still quite confusing, according to this page https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-change-events-key

Debezium should pick up the ID in a delete event:
"For a document identifier, an event key’s payload.id value is a string that represents the updated document’s original _id field as a MongoDB extended JSON serialization that uses strict mode"

So we should always have the '_id' in the payload and I don't see any config options to enable this behaviour.

For context, I'm using Debezium Server 1.8 and Pub/Sub.

Please help! 😭 

[Chris Cranford]

Hi Chachi -

To obtain the deleted id, you should be able to obtain this information by looking at the event's key.  What you've shown below is the value portion of the event.  I do recall that in the older oplog implementation, you could often obtain the id in the "filter" part of the payload because that is where MongoDB recorded it; however that is no longer the case with Change Streams.

Hope that helps.
Chris

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/dd158a52-0774-47d9-9486-39cb8dca72dfn%40googlegroups.com.

[Chachi Pereira]

Hey Chris,

Thank you for your response, we've been working to that since but:

Sorry to seem a bit silly, we can't seem to get it to work.

The docs do state that "The event key and event payload are in a change event only if you configure a converter to produce it."

However, I can't find a way to explicitly enable the event key when a change event is sent to pub/sub. It seems only the value portion is sent.

[Chachi Pereira]

Figured it out if anyone else has this issue:

Set: debezium.sink.pubsub.ordering.enabled=true in your .properties file.

The deleted document id will be available in the ordering_key property of the message.

e.g

{"payload":{"id":"{\"$oid\": \"5f90326300eb454324be9261\"}"}

[Chris Cranford]

Hi Chachi -

Glad you found the solution and that's most certainly one way to have gotten the value into the payload, among others such as using a single message transformation to accomplish the same.

Thanks,
Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/79a8f8a3-9713-4bdc-8d5a-eef2f568b74bn%40googlegroups.com.